KR101139731B1 - Cdss evaluation toolkit apparatus and evaluation method using the same - Google Patents

Abstract

The present invention relates to a device and method for evaluating conformity to standards and standards that can guarantee the minimum quality of the clinical decision support system (CDSS), medical standard code conversion function, database integrity diagnosis function, by test item CDSS inspection toolkit device including inspection function (handling inspection in case of no required input value, response check for non-clinical value, time stamp related inspection, etc.), CDSS security level check function, and inspection method through it By doing so, it is possible to improve the reliability of technical document screening, safety, and validity test of CDSS-related medical devices.

CDSS, Inspection, Evaluation, Quantification, Reliability, U-Health

Description

CDSS inspection toolkit device and inspection method {CDSS EVALUATION TOOLKIT APPARATUS AND EVALUATION METHOD USING THE SAME}

The present invention relates to a toolkit device for testing CDSS and its test method, and more particularly, to a toolkit device for testing CDSS for evaluating conformity to standards and standards that can ensure the minimum quality of the Clinical Decision Support System (CDSS) and It is about the inspection method.

The reality that chronically ill patients account for 30% of Korea's total population and health care costs exceeding 7% of GDP is expected to become more severe with the aging society. One of the solutions is the home health care based on u-health. Has been continuously highlighted.

Clinical Decision Support System (CDSS), one of the basic technologies that are developing along with the active activation of the home healthcare market, is a system that helps medical decision making by implementing specialized medical knowledge in the form of computer software. CDSS technology has continued to evolve with the development of computers and medicine since the idea was announced in the early 1950s. Early CDSS was simply defined as a computer program (Shortliffe, 1990) that assists medical professionals in making clinical decisions because they did not anticipate the pace and outcome of modern technology, but the computer is easily accessible to the general public. As painters became involved, all computer systems that provided medical knowledge in the form of WWW or e-books had the problem of being included (van Bemmel and Musen, 1997). For this reason, Johnston (1994) 's definition of "computer software using a knowledge base (KB) designed to help healthcare practitioners directly manage their patients' decision making has been primarily used." Has been.

With the beginning of the ubiquitous era, medical technology has been created in the form of CDSS, which can be used even within the reach of professional medical personnel, and the convergence of CDSS and ubiquitous technology has taken medical accessibility to the next level. Although it can be used as an effective means to protect and promote public health, there are also negative aspects of misuse of medical knowledge. Therefore, in order to guarantee the health of the people, standards and standards that can guarantee the minimum quality of the CDSS are needed, and the certification means for this is also urgently developed.

The purpose of the present invention, which is newly proposed to solve the above problems, first defines the CDSS accurately and sets the scope thereof, and then provides the CDSS without the internal disclosure of the CDSS in order to provide certification for the CDSS belonging to the definition and scope. The present invention provides a toolkit for a CDSS inspection apparatus and a method of inspecting the same to provide an objective and reliable result of evaluating the performance of the CDSS.

In addition, another object of the present invention can be effectively performed through the standard toolkit for a variety of directions, including medical standards code compliance for CDSS, database integrity, functional inspection of the inspection unit by inspection item, security level check To provide a CDSS inspection toolkit device and its inspection method.

In order to achieve the above object, the CDSS inspection toolkit device according to an embodiment of the present invention comprises a medical standard code conversion configuration, database integrity diagnostic configuration, inspection item configuration by inspection items quantitative evaluation of the CDSS Get results.

The inspection unit configuration for each inspection item includes a handling inspection in the absence of a required input value, a corresponding inspection for non-clinical numerical values, a time stamp related inspection, and the like.

In addition, the test method using the CDSS toolkit device for testing according to another embodiment of the present invention comprises the steps of confirming the compliance of the medical standards of the CDSS by checking whether the code and the unit, guidelines and protocols of the CDSS; Providing clinical input to the CDSS and confirming that the results meet clinical criteria; In the event of loss or damage of time stamp mixes and required inputs, this may include verifying software stability.

The step of confirming the clinical reliability further includes a step of determining whether to provide a case in which the numerical value that cannot be clinically displayed as an input.

The step of confirming the clinical reliability further includes a step of confirming whether or not to provide a correction function for compensation for the difference in the numerical value of the situation.

On the other hand, the platform independence of the CDSS and whether the implementation of the additional function for the visually impaired user may further comprise the step of checking.

CDSS inspection tool apparatus according to the present invention and its inspection method is the performance of CDSS quantitatively quantified by measuring by using a toolkit that checks whether the function of the CDSS to be certified to meet a predetermined standard by a normalized inspection method There is an effect that can be evaluated, and can provide objectivity and credibility to the evaluation data.

The present invention as described above will be described with drawings and tables.

First of all, the present invention basically starts from the definition of CDSS, which is to filter what can be classified as CDSS among various medical information providing systems, and distinguishes CDSS to be evaluated for performance according to at least these criteria. Afterwards, the toolkit system design of the present invention to perform a performance evaluation on such CDSS should be made.

In particular, the definition of the CDSS corresponds to the guidelines for utilizing the toolkit system according to the embodiment of the present invention for the purpose of CDSS certification by the Food and Drug Administration. That is, the toolkit system according to the embodiment of the present invention does not intend to provide an approximate test result for any medical information providing system, and distinguishes and authenticates a CDSS functioning correctly as a substantial CDSS among strictly classified CDSSs. This is to provide the data for improving the CDSS.

To this end, the CDSS according to the embodiment of the present invention should have at least a consideration according to the legal scope, such as the legal scope of the medical law and the legal scope as the authentication object, and a practical technical scope.

First of all, under the medical law, CDSS can be divided into those for medical personnel and those for non-medical persons. CDSS for non-medical persons should be prevented from making diagnosis or prescription for medical practice under medical law. As it is used, it is possible to make a diagnosis or prescription corresponding to a medical act. In the embodiment of the present invention, the CDSS to be examined may be limited to a medical CDSS.

On the other hand, considering the legal scope of the CDSS as an authentication object, the CDSS is a computer program in which an algorithm is implemented. However, in order to operate the CDSS, it is necessary to consider security. It should be possible to provide comprehensive certification for additional configurations.

Therefore, the CDSS inspection toolkit system according to an embodiment of the present invention should be configured to perform security authentication for such medical CDSS as well as quantitative inspection of its subordinate algorithm.

On the other hand, in order to distinguish the CDSS from the computerized contents of simple folk remedies as a technical scope for the professionalism of the CDSS, numerical inspection from (1) medical devices licensed by the Food and Drug Administration or (2) technology separately recognized by the Food and Drug Administration. Specify the conditions of the computer program using the results. The "recognition technique" in (2) complements the case where the medical device is not used, such as the "diagnosis algorithm for skin diseases using computer vision" or "the algorithm for dementia diagnosis specialist".

As a result, it is preferable that CDSS, which is the inspection target of the present invention, be defined as follows.

CDSS Definition
CDSS as an Object of the FDA Certification Program
(1) Quantified input by KFDA approved medical devices
(2) Quantification technology recognized by KFDA
(3) Enter your medical history or symptom
(4) Signs or opinions by specialist staff.
Computer software that presents medical inference results from one or more of the following:

Therefore, the CDSS test method according to the present invention can be configured as follows.

First, the quality of the CDSS is performed by reviewing the software of the CDSS using the toolkit system configured according to the embodiment of the present invention and determining the validity of the output value for a given input. In addition, when such quality certification passes, the operation certification may be performed by inspecting security matters using the toolkit system.

1 shows an overview of quantitatively evaluating Decision Support System for Healthcare (3: DSSH, a broader concept than CDSS) using the toolkit system 100, which provides a standardized scenario (1) to the CDSS. Comparing the preset results (4) with the results (2) and the actual standardized scenario (1), the performance can be quantitatively identified (5). In order to realistically implement the concept of quantitative evaluation of CDSS performance through the use of the toolkit system 100, design an optimal toolkit system on what to check, how to perform quantification, and how to actually connect with the CDSS. Needs to be.

In the embodiment of the present invention, a design scheme for implementing an optimal toolkit system considering the functional characteristics of the CDSS will be described.

2 is a block diagram of a CDSS test tool according to an embodiment of the present invention, the input and output unit 110 in charge of the input and output to the examinee 10 (test target CDSS), the predetermined clinical input content and the result accordingly Scenario database (140) having a, the reliability check unit 130 to perform a CDSS test for the examinee as a result to check the clinical reliability, the security check to confirm the security status by checking the configuration and input and output status of the CDSS The control unit 150 manages the unit 120, the input / output unit 110, the reliability check unit 130, and the security check unit 120, and provides a physical function for some of the units.

The input / output unit 110 is responsible for input / output with the subject 10. That is, in order to quantitatively evaluate the CDSS to be inspected, a scenario of a preset method is provided to the inspected CDSS 10 and the result is received.

The scenario database 140 is composed of a standard scenario database (Standard Senario DB) having a predetermined clinical input content and results according to the above, the standard scenario database can be managed using a professional DB or a directory structure. Do.

The security checker 120 is configured to classify the type according to the type and operation type of the inspection target CDSS 10 and to confirm whether the security level is set accordingly.

The reliability verification unit 130 performs a CDSS test on the test target CDSS 10 to evaluate the actual performance of the subject 10 to measure the compliance of medical standards, software stability and clinical reliability.

3 is a configuration diagram of the input / output unit 110 according to an embodiment of the present invention, and the protocol conversion unit for converting or using a different type of protocol when the test target CDSS 10 does not comply with a medical standard protocol. (111), the code conversion unit 112 for converting the input and output code between the inspection target 10 and the inspection toolkit system by recognizing the type of input and output code by the preset classification method, converting the measured values of the medical device into a standard inspection unit It includes a conversion unit 113 to.

In order to quantitatively evaluate the CDSS to be inspected, it is necessary to classify the existing CDSS according to the type of input / output for the design of the toolkit system that provides the input in a preset scenario and receives and evaluates the results. In fact, when requesting the CDSS development team, it is almost impossible to request the inside of the CDSS (source code, etc.). Therefore, performance evaluation of the CDSS can only be performed on the input value and the output value, and the CDSS for this input / output relationship should be classified according to the characteristics of the input / output value. The present invention classifies the type of the existing CDSS according to the input and output values as shown in Table 1 based on the performance evaluation method.

The classification of using knowledge base (PK1-PK3) represents the classification of general CDSS, which can be used basically when the development method is published. In addition, it can be divided into PI classification according to the type of input value and PO classification according to the type of output value.

PI1 / PO1 in Table 1 indicates when input / output values are input / output by standard codes and values. Since most standard codes use CUID (Conceptual Unique ID), they can have the most clear evaluation method, and most u-Healthcare-based CDSS will use this method later.

PI2 / PO2 of Table 1 refers to codes that input / output values are used for internal use by CDSS developers rather than non-standard codes, that is, codes that are generally used. For example, when an EEG signal is analyzed outside of CDSS, divided into several types, and the types are self-coded and input, this is called PI2 CDSS.

PI3 / PO3 in Table 1 is an input / output case in which input / output is difficult to express by code such as Computer Vision, EEG signal.

The code conversion unit 112 recognizes the classified code types through the classification process and converts the input / output code between the inspection object 10 and the inspection tool system. Although it is desirable to require compliance with a specific medical standard code for CDSS for smooth authentication and general purpose, a code conversion unit may be required to allow mutual compatibility of codes used in medical standards such as foreign CDSS.

In other words, except when technically unacceptable, externally recognized standard codes such as UMLS / SNOMED CT / KOSTOM are used as input / output to ensure compatibility, and input provided by the CDSS toolkit can be provided as one of them. In addition, in order to manage the output of the CDSS, the code conversion unit 112 may be configured so that one of them may be converted into another code and then converted into another code.

The conversion unit 113 converts the measurement values of the medical device to a standard test unit so that a standard test unit can be used in addition to the medical code. In addition, the conversion unit 113, if the guidelines for a particular disease is clearly presented and used in the professional society, to follow the terms, classification, diagnostic methods, units of measurement, etc. presented in the guidelines.

Meanwhile, the input / output unit 110 of the toolkit is configured to utilize the medical standard transmission / document protocol, and the medical standard document such as CCD and CCR and the medical transmission standard such as HL7 are followed.

If the protocol conversion unit 111 does not follow this standard protocol among the CDSS, it may convert it or use another kind of protocol. The protocol conversion unit 111 is basically configured to support some well-known standards, and other communication schemes are preferably induced to be converted into a standard protocol provided by the corresponding CDSS.

In addition, the input / output unit 110 transmits the information regarding the medical standard compliance of the input and output data obtained in the input and output process to the reliability confirmation unit through the control unit, the reliability check unit performs a test of compliance with the medical standard of the test target CDSS To do it.

4 is a block diagram of a reliability check unit 130 according to an embodiment of the present invention, the reliability check unit 130 is medical standards compliance, software stability and clinical reliability to evaluate the actual performance of the test target CDSS Measure

The scenario provider 131 processes the standard scenario data extracted from the scenario database 140, and then provides the test scenario modified with the extracted scenario to the inspection target CDSS.

The data inspecting unit 132 performs a check on the medical standard compliance of the data received by the input and output unit, the medical standard compliance conditions are divided into the necessary conditions and recommendations according to necessity, the recommended conditions are strong recommendations according to their importance Can be divided into conditions and general recommendations. The conditions of compliance with the medical standards are as follows.

i) Utilization of medical standard codes (strong recommendation): Except in the case of technical difficulties, externally recognized standard codes such as UMLS / SNOMED CT / KOSTOM are used as input / output to ensure compatibility.

ii) Use of standard test units (strong recommendation): When using test values such as measurement values of medical devices, the standard test units generally used should be used.

iii) Use of four elements of inspection (general recommendations): Inspection, ID, measurement time, measurement items, and inspection values should be clear.

iv) Utilization of standard medical guidelines (general recommendations): When the guidelines for diseases, etc. are clearly presented and used in the professional society, the terms, classification, diagnostic method, and unit of measurement suggested in the guidelines should be followed whenever possible.

v) Use of medical standard transmission / document protocol (general recommendation): It is recommended to directly input the format of medical standard document such as CCD or CCR and medical transmission standard such as HL7.

In addition, the data inspecting unit 132 inspects the data obtained by the input / output unit according to data confusion, input error, etc. in terms of software stability, and the authentication condition of the software stability is also required and recommended according to necessity. The recommendation can be divided into strong recommendation and general recommendation according to its importance. The authentication conditions of the software stability are as follows.

i) Measurement time handling (Date / Time Stamp Handling: Strongly recommended condition): If the date / time of measurement such as the check value of a certain CDSS input is mixed, this should be handled properly. In the u-Healthcare area, PHR is used to pursue mutual compatibility of data. In other words, the result of the test stored in the database named A and the result of the test stored in the database named B may be merged. In such a situation, proper adjustment may be necessary in the merge function itself. There is a possibility of this not being possible, so the ability to handle it within the CDSS itself must be added.

ii) Missing Data Handling (mandatory condition): If an input value for a CDSS is deformed or disappeared even though it is a required input value, it should be handled properly.

iii) Unclear Data Handling (mandatory condition): It should be possible to handle the case where the data is not provided in the form of clear data due to communication or damage to some of the data.

The data inspecting unit 132 may additionally configure the platform of the CDSS to check the versatility, and may further configure the additional function for the visually impaired user as an authentication requirement.

After the test subject CDSS performs the test using the scenario provided by the scenario providing unit 131, the clinical reliability check unit 133 checks the following clinical reliability conditions with respect to the result to determine whether the certificate is certified. By determining the actual clinical reliability of the CDSS to be tested can be confirmed.

The conditions for the certification of clinical reliability are as follows, and are divided into essential conditions and recommendations according to their necessity, and the recommendations can be divided into strong recommendations and general recommendations according to their importance.

i) Minimum False Positive Ratio (mandatory condition): If the result of the CDSS is screening for any disease, the result must satisfy the minimum false positive ratio that has been negotiated by the KFDA through a relevant society.

ii) Minimum False Negative Ratio (mandatory condition): If the result of the CDSS is screening of a disease, the result must satisfy the minimum False Negative Ratio negotiated by the KFDA through a related society.

iii) Clinically Improper Data Handling (strong recommendation): There should be a function to properly determine if a value beyond the clinically acceptable value is provided as input to the CDSS.

iv) Test Condition Control (Strong Recommended Condition): In the case of blood pressure measurement, it is known that the value is different between the measurement at the hospital and at home. In addition, the application of the test value is often required according to the psychological state and the physical state such as before meals, after meals, and after exercise, and the CDSS should have a function for correcting it internally. In the absence of such a function, the CDSS user should be able to suggest that the CDSS can be used under certain conditions, regardless of how the manual affects the condition.

The clinical reliability check unit 133 is configured to determine the output of the desired target value range by grasping the output according to the clinical reliability conditions.

The evaluation result providing unit 134 provides the evaluator with the corresponding evaluation result according to the examination of the data inspecting unit 132 and the clinical reliability checking unit 133.

5 is a block diagram of a security check unit 120 according to an embodiment of the present invention. Since the security checker 120 is configured to classify the type according to the type and operation type of the inspection target CDSS, and to check whether the security level is set accordingly, the security checker 120 is operated. CDSS operation type checking unit 121 for classifying the CDSS type according to the configuration, and a security check unit 122 for verifying the security level according to the type of CDSS is configured to include.

The CDSS operation type checking unit 121 may classify the types of CDSSs to be inspected in terms of operation and security as shown in Table 2, and the classification is performed from an external control signal (CDSS using user input or communication) to the toolkit system. Information).

As shown in Table 2, the classification of CDSS according to operational security is classified into portability (P), sharing (S), networking (N), data storage (D), and form factor (F) according to the classification type. The form of each operation is indicated by adding a T. The higher the number that follows, the higher the risk of security and the higher the level of security policy required.

Security requirements in the operational classification shall always comply with Worst Case Senario. In other words, a laptop computer can be portable or can be used in a fixed form like a desktop computer. However, if TP1 and TP2 are available, both TP1 and TP2 criteria must be met.

In addition, if the network system is used to operate CDSS on the server side, store data, and access only from the client side, the security requirements for the server side and client side must be satisfied.

The following is a detailed description of the classification of CDSS according to operational security.

i) TP (Portability): It is a classification of CDSS operation type according to portability. The TP1 is a case where it is impossible to carry because it is more than a certain weight like a desktop computer. The TP2 is portable and is expected to appear mainly in an embedded form.

ii) TS (Sharing): It is classified into 4 steps from TS1 to TS4 according to sharing. TS1 is a CDSS operation that is primarily used for personal purposes and is less likely to be shared with others. TS2 is a form that can be shared in very limited small groups such as family, relatives and friends, such as home health devices. TS3 is a company that shares a CDSS system with certain people, such as a company. TS4 is a case where a large number of people share CDSS, such as the body fat system of Jjimjilbang.

iii) TN (Networking): TN1 is defined as the case where the information cannot be moved to the network. TN2 is the case when forgery of data through the network is not easy due to the limited service operators such as mobile phones. And when data is transmitted through local wireless, it is defined as TN3, and TN4 means general Internet environment or more. TN3 and TN4 can be defined simultaneously.

iv) TD (Data Storage): TD1 is a case where CDSS does not store the information separately in the media such as having only systematic display function. If you keep the data of a specific individual, it is called TD2. If you keep the data of a small group such as family, TD3, if you store data of people who have a certain identity, such as a company or membership gym, TD4 When stored, it is called TD5.

v) TF (Form Factor): When CDSS is implemented in hardware, firmware, and pure software are classified into TF1, TF2, and TF3, respectively.

The operation security check unit 122 is secured in the case of theft according to the type of CDSS operation, the access management for personal information management, whether to secure the confidentiality of the exchanged data, the authentication manager only personal information Verify that the operational security requirements of the subject CDSS are certified by checking the data integrity by allowing modifications and changes, and checking the accessibility of authorized users with all such security maintained.

Figure 6 shows a flow chart of the inspection method using the CDSS inspection toolkit apparatus according to an embodiment of the present invention.

The CDSS toolkit device checks the code, unit, guidelines and protocol utilization of the CDSS to confirm compliance with the medical standard of the test target CDSS (S601), and provides input to the test target CDSS, and the result is based on the clinical standard. Check the suitability to confirm the clinical reliability (S602), time stamp mixed with the loss or damage of the required input to verify the results of the software stability (S603).

7 is a data flow diagram of a test method using a CDSS inspection toolkit device according to an embodiment of the present invention, the situation for the authentication of the CDSS, the classification criteria of the CDSS for each situation and the design method for the CDSS inspection The configured toolkit system can be utilized through the following evaluation methods.

That is, when the evaluator applied for evaluation for inspection (S501), the evaluator selects evaluation items from the toolkit system and generates a test scenario derived from the standard scenario (S502).

At this time, the toolkit generates the test ID together (S503) and stores the test scenario files in a folder named Test ID, and simultaneously generates and stores a file (TSAE) corresponding to the correct answer of the test scenario under the name of the test ID. (S504, S505). The correct answer file is later compared with the answer file generated by the evaluator using the CDSS.

The evaluator downloads the test scenario files from the folder using the test ID notified from the evaluator (S506), generates a result for each item using the CDSS developed by the evaluator (S507), and creates one file in the designated format. The answer (TSAR) is uploaded (S508).

The evaluator inputs the test ID into the toolkit and then presses the evaluation button to generate a report by comparing the correct answer with the answer from the CDSS (S509). This answer comparison and report generation can be performed automatically by the toolkit system.

In addition, the operation of the evaluator and the evaluator during the process can be automatically performed by the toolkit system according to the setting, it is preferable that such setting is configured to provide a consistent result in the average range.

In the above described and illustrated with respect to preferred embodiments according to the present invention. However, the present invention is not limited to the above-described embodiments, and various changes and modifications may be made by those skilled in the art without departing from the scope of the present invention. .

1 is an overview of quantitatively evaluating Decision Support System for Healthcare (DSSH) using a toolkit system.

2 is a block diagram of a CDSS inspection toolkit apparatus according to an embodiment of the present invention.

3 is a block diagram of an input-output unit according to an embodiment of the present invention.

4 is a block diagram of a reliability check unit according to an embodiment of the present invention.

5 is a block diagram of a security check unit according to an embodiment of the present invention.

6 is a flowchart of a test method using a CDSS test toolkit device according to an embodiment of the present invention.

7 is a data flowchart of a test method using a CDSS test toolkit device according to an embodiment of the present invention.

DESCRIPTION OF REFERENCE NUMERALS

10: Evaluator (inspection target CDSS) 100: CDSS inspection toolkit device

110: input and output unit 120: security check unit

130: reliability verification unit 140: scenario database

150: control unit

Claims (13)

An input / output unit having a medical standard code and a unit conversion unit; Using the scenario extracted from the preset scenario database or providing the modified input to the CDSS and receiving the result and comparing the result with the preset result according to the scenario, or the expected result and the result of the CDSS according to the modified input content A reliability verification unit that provides quantitative evaluation information on clinical reliability by comparing the; A security checker configured to check a configuration and input / output state of the CDSS, collect a result of approaching in a preset manner, and check a security state; A scenario database having predetermined clinical input contents and results according thereto; And a controller for managing the input / output unit, the reliability verification unit, and the security verification unit, and providing a physical function for some of the components. The reliability check unit CDSS inspection toolkit device, characterized in that to confirm compliance with the medical standards of the CDSS by checking the code and unit, instructions and protocol utilization of the CDSS. The apparatus of claim 1, wherein the reliability check unit comprises a handling check in the absence of a required input value, a corresponding check on non-clinical values, a time stamp related check, and the like. The apparatus of claim 1, wherein the input / output unit further comprises a code conversion unit for classifying and converting types of input / output codes between the CDSS and the CDSS inspection toolkit device in a predetermined manner. The apparatus of claim 1, wherein the unit converting unit converts the measured value of the CDSS into a standard test unit. The apparatus of claim 1, wherein the input / output unit further comprises a protocol conversion unit for converting the CDSS if the CDSS does not conform to a medical standard protocol or enabling a different type of protocol. The method of claim 1, wherein the reliability check unit uses the medical standard code of the CDSS, uses a standard test unit, uses a standard medical guideline, uses a medical standard transmission / protocol, and checks, IDs, measurement time, measurement items, and test values. Device for testing CDSS tool, characterized in that for testing at least one of the clarity of. The apparatus of claim 1, wherein the reliability check unit checks the platform of the CDSS to check versatility or to check an additional function for an audiovisual impaired user. The method of claim 1, wherein the clinical reliability is based on a minimum false positive ratio, a minimum false negative ratio, a clinically improper data handling, and a test condition correction. CDSS inspection toolkit device, characterized in that at least one. According to claim 1, wherein the security check unit is at least a condition of the portability of the CDSS device, whether the CDSS device is shared, whether information is moved through the network, whether the information is stored, whether hardware or software is implemented as a condition of the CDSS security state CDSS inspection toolkit device characterized in that for checking the security status using one or more. Checking, by the CDSS inspection toolkit device, whether CDSS code, units, guidelines, and protocols are utilized to confirm compliance with the medical standards of the CDSS; Confirming clinical reliability by providing an input to the CDSS by the toolkit for CDSS examination and confirming that the result conforms to clinical criteria; And checking the results of the CDSS inspection toolkit device by checking the result when the time stamp is mixed or the required input is lost or damaged. 11. The method of claim 10, wherein the step of confirming the clinical reliability further comprises the step of determining whether to provide a case for providing a value that can not be clinically displayed using the CDSS test toolkit device, characterized in that method of inspection. The method of claim 10, wherein the determining of the clinical reliability further comprises determining whether to provide a correction function for compensating for the difference in numerical values for each situation. 11. The method of claim 10, wherein the step of verifying the software stability further comprises the step of confirming whether the platform independence of the CDSS and the implementation of additional functions for the visually impaired user, the test using the CDSS inspection toolkit device Way.

Images