KR100190267B1 - Expert system tester - Google Patents

Abstract

The present invention relates to a test system having the ability to perform all combinations of data entry forms using a special data test set in an expert system. The special data test set includes a set of sensor values for each level of the operating system so that all diagnostic levels can be tested without testing all possible sensor values and combinations of sensor values. Test sets can be combined to generate tests of various orders to allow complex relationships between the sensors to be tested and the rules.

Description

Test Systems and Expert System Test Methods for Expert System Testing

1 illustrates the components, inputs, and outputs of the present invention.

2 is a flowchart of the operation of the first test according to the present invention.

3 illustrates how a first test can be changed to perform a second test.

4 shows the data flow when the present invention is used in conjunction with a preferred expert system.

5 shows a preferred expert system and the execution sequence of the present invention.

* Explanation of symbols for main parts of the drawings

10: Regression Tester 12: Expert System

14; Rulebase 16: Inference Engine

20: test system

The present invention relates to an expert system tester, which enables testing based on a wide range of expert system rules without the need for a test program with some knowledge of the particular rulebase being tested. As a test method for analyzing the change of rulebase after the change is made, it is related to the practical regression testing rmethod which effectively improves the quality of rulebase at low cost.

Expert system rulebases usually contain more than 1000 rules, and in these systems, sensor inputs typically in the hundreds. If the sensor is a digital sensor that provides one of the two inputs, the number of possible combinations of input values is huge, so it is practically impossible to test all combinations. In the case of analog sensors with values, the test problem becomes more complicated: in the normal process of developing new rules for large rulebases, standard debug form or production rule testing by experts is carried out. In situations such as when creating a new rule, the expert uses a simulated plant data with simulated plant data whose value exceeds the range of expected sensor values for the sensor inspected by the new rule. By activating, you ensure that the new rules work as expected, that is, generate expected diagnostics when expected. In one situation, the sensor values other than the sensor values used by the new rule remain normal, as a result, the test of the production type does not affect the influence of other sensors on the new rule or the interaction of the new rule with the rest of the rule. The second type of test involves connecting the expert system online with the actual continuous plant data and allowing the expert to examine the diagnosis in detail to determine whether the diagnosis was as expected. It is clear that more effective techniques and tools are required to verify the operating performance of a system with base and input numbers.

The main object of the present invention is to provide a diagnostic rulebase verification to a test system without having to use any knowledge in the tested rulebase to initialize and operate the test tool.

In order to achieve this object, the test system for testing expert systems of the present invention includes test storage means for storing normal test sets and special test sets, and test means for testing expert systems using normal test sets and special test sets. Wherein said testing means comprises an example means for comparing an expert system output responsive to a special test set with an exception condition and reporting a match.

The invention is further clarified from the following description of the preferred embodiment shown by the embodiments only in the accompanying drawings.

Regression testing is a way to analyze the above rulebase changes that have been made. There can be two kinds of changes: (1) an intended change, and (2) an unintended change back to another unrelated aspect of the rulebase. Regression testing is important because changes and error corrections may result in more errors than those occurring in the rulebase's original coding scheme.

Full regression tests thoroughly run the rulebase through all possible combinations of data entry and diagnostic scenarios. In practical terms, such a test is not possible due to the time it takes to perform a complete test. More realistic regression tests perform all combinations of data entry types, and if you want to perform advanced diagnostics, follow a special test case that represents each type (a subset of the full test).

The test system or regression analyzer of the present invention addresses these practical test requirements. The present invention adopts production quality rule base, sensor data set and test plan as input. For each data combination, normal diagnostics are performed using normal data that does not occur any diagnostics indicating abnormal conditions such as alarms. This normal diagnosis is continued by repeating with disturbed data. Normal and disturbed test cycles can continue until all combinations have been tested. The system creates a separate list of exceptions for possible inconsistencies, as defined by the test log file and test plan. Analyzers can be run interactively to perform simple tests, or they can run extensive tests overnight in batch mode. You can determine if regression has occurred by comparing this log file and the exception list between rule changes.

As shown in FIG. 1, the regression tester 10 of the present invention includes a production rulebase 14 and an expert system inference engine 16. Interact with the expert system 12. The expert system inference engine is preferably a Processor Diagnostic System (PDS), available from Westinghouse and described in US Pat. Nos. 4,644,479 and 4,649,515. A description of the operation of the preferred expert system can be found in Westinghouse Electric Corporation, 1987, Kemper and Harper, The PDS Manual, PDS Version 5.1, Diagnostics. The regression analyzer is written using a test plan language, described in detail below, to test system 20 to access normal and test data sets 22 applied to inference engine 18. It includes the test plan 18 used by. The inference engine analyzes (diagnostics) using production rules in the rule base 14 that generate output that is stored in the log file 24 by the test system and compared to the results predicted to produce the exception report 26. Do this. The system also allows the output to be displayed on a CRT screen 28. The present invention is preferably implemented on a computer of the VAX 8000 series of Digital Equipment Co., which runs the VMS operating system. In addition, the present invention is preferably implemented in a language such as C suitable for a structure design approach, as described below.

As will be described in more detail below, the present invention may perform a first to sixth special expert system test. Of course, those skilled in the art will recognize that higher order testing is possible as the processor speeds up. 2 illustrates the general operation of the present invention during the first test. The present invention starts the test 40 by reading 42 the test plan 18. The test plan 18 is generated using a special test plan language which will be described in more detail later. The test plan language generates a regression test plan 18 that is stored in a text file and read 42 into the test system 20. The production rule base 14 is then loaded 44 and the test data is subsequently loaded 46.

Next, by setting 48 the pointer to the sensor whose value is to be changed, all the sensors are set to normal values and updated 50. This update operation causes the inference engine 16 of the expert system to recognize that new sensor values are available. In this way, in step 50, all rules are flagged for execution.

The rule with the new (updated) sensor value is then started in the expert system and continues to run until no further changes (firings) occur. If this is the first cycle during the test (54), the results of the normal sensor data expert system diagnostics are stored 56 as the baseline. Next, one of the sensors is set to a test value and updated 58, and then a supported rule, i.e., a rule using a new sensor value, is started (60). At this time, the output of the expert system (preferably malfunctions updated with positive reliability) is written to log file 24 (62). This output is then compared 64 to the test plan exception and, if there is a match, an exception report is written 66. The pointer is then updated 68 and tested 70 to determine whether the last sensor has been reached. If it is determined that the last sensor has not been reached, the cycle is repeated, otherwise the system is stopped. As can be seen from FIG. 2, the system periodically applies normal test data and special test data and repeatedly applies special test data (one sensor value changes on one side) between cycles.

3 shows the general operation of the present invention when a secondary test is being executed. This requires the addition of a new pointer 80 so that the two sensor values can be changed from a normal value to a disturbed value. Before the result is stored 56, both pointers must be tested 82 to determine if this is the first cycle of the test. Before the rule starts to run (60), two sensor values are currently changing, so both sensor values must be updated (84). Since an additional pointer is included, the system must update this pointer (86) to test to see if the updated pointer has reached the last sensor. In order to provide additional test capabilities, as can be seen by comparing FIGS. 2 and 3, the present invention only needs to provide additional sensor pointers, appropriate sensor update steps and pointer increments and test loops. Annex IV exemplifies how the sixth order test capability is rolled out by structural design.

The sensor data or data set 22 input to the regression analyzer or system 20 of the present invention is preferably in standard PDS format. Seven data sets, or timesets in PDS terminology, are required for sixth order test capability. These timesets can be generated using PDS. The sensor value may be a logical or numerical value as needed.

The first timeset should be a set of normal readings. This set should generate confidence factors between 0 and -1 in the PDS system. This data is used as a benchmark or baseline to compare with all other conditions.

The next six timesets represent data from abnormal equipment operation. In general, an abnormal timeset or dataset should be a value taken at both sides and at the boundary of different condition levels of the expert system. That is, the test set must provide test values for each diagnostic level of the system to be tested, eg, predictive level, diagnostic level, warning level, shutdown level, and sensor failure level at the power plant. In the case of a power plant, these values can be obtained from the alarm level of the power plant. Other types of data sets, such as economical data, may be used, but the examples used herein will be for power plants.

The second timeset has an indicator value expected when each sensor is below range or fails to a lower value. These indicator values generate at least one sensor diagnostic for each fault sensor, and the confidence coefficient for the sensor rule in the PDS system must be positive while all others are negative. The third timeset has an indicator value expected when each sensor is out of range or has failed to a high value. These indicator values give at least one sensor diagnostic for each fault sensor, one confidence coefficient in the PDS system is positive and all others are negative. The fourth timeset is used to define an indicator value having a predicted value in the rulebase design. In other words, the predictive level sensor value initiates the rule for predicting possible outcomes of continued operation under these conditions. This category provides a level of test data before a diagnostic alarm is triggered. The fifth lime set has an indicator value that triggers a diagnostic PDS alert. These indicator values should generate a low level of positive confidence coefficient in at least one malfunction diagnosis. The sixth timeset has an indicator value that triggers a Protection I PDS alert. These indicator values should generate intermediate levels of positive confidence coefficients in at least one malfunction diagnosis. The seventh time set has an indicator value that triggers a ProtectionII PDS alert. These indicator values should generate a high level of positive confidence coefficient in at least one malfunction diagnosis.

In the present invention, the regression test plan 18 is defined in a text file and read into the test system 20. The plan consists of a test that runs the rulebase with a combination of sensor values from 1 to 6 that change with each cycle. As described above, for example, the primary test cycle executes the normal index value through the rulebase, and then executes using the same set of normal index values in which one sensor value is changed to the trip index value. The overall primary test repeats normal / disturbed execution for all sensors. The plan definition may also include an exception report specification for each test. An exception is a requirement to record an example of a condition. For example, a useful exception to include in the primary test described above is to report that there was no malfunction resulting from a confidence factor greater than 0.5 from one of the test cycles.

The exception should be designed in such a way as to signal a regression test failure due to the presence of an example report.

The first line of the test plan file preferably contains a directory specification where the rulebase file resides. Each subsequent line preferably includes a test definition or an exception definition. Exceptions following the test definition apply only to that test.

The single regression test is preferably specified by a set of sensor categories enclosed in parentheses. For example, the first test of the trip data is followed by the second test of the warning and trip data given as follows.

(Trip)

Warn Trip

The plan with this test sequence specifies the primary test that will cause the rulebase to run for all sensor trip values, and the log file will record all updated malfunctions and procedure results. This recording is followed by a secondary test performed by running the rulebase for all trip values, each sensor being then set to a warning value. Since no exception is defined, no entry is made in the example report 26.

The exception is divided into three categories that provide three basic filters: 1. Update exception: 2. Level exception: 3. Change exception. Update exceptions are tested to see if appropriate malfunctions and procedures have been updated by the diagnostic cycle. For example, a courtesy to report any updated malfunction

Exception: Malf updated.

to be. Level exceptions compare the resulting malfunction or procedural parameters with a fixed value. For example, an exception to report when the malfunction confidence factor is greater than 0.5

Exception: Malf CF 0.5.

to be. The change exception compares the diagnostic percent change with a constant. Percent change is calculated using the conventional formula (new-old value) / old value. The old value is a malfunction or procedural parameter of the normal diagnosis stored in step 56 of FIG. The new value is a malfunction or procedural parameter obtained from the second diagnosis of the cycle (steps 58 and 60). As a result, the positive change is far from zero, and the negative change is close to zero. For example, an exception to report a change of at least + 10% in the procedural confidence coefficient is

Exception:% Proc 10.0

to be.

Three exception classes that provide three variants on three basic filters: 1. Simple. Exceptions: 2. Set exceptions: 3. There are met exceptions. Simple exceptions are similar to the category example shown above, in which case the exception consists of parameters, operators, and constants. Each time a simple exception tests true, the test definition, disturbed sensor, malfunction or triggered exception or procedure and parameter values are recorded in the report (step 66). Set exceptions are simple exceptions prefixed with the qualifier ALL, SOME, or NO. In this case, a report is generated if the set of exception parameters matches the exception definition. If this exception tests true during the entire test, the test definition and disturbed sensor are recorded in the exception report (66).

The number of elements in the set with the SOME set modifiers is also recorded in the exception report (66). For example, an example to report when some malfunction confidence coefficients are greater than zero in a regression test is Exception: Some Malf CF> 0.0. The chum exception is a simple exception preceded by the vertical LTN (less) or GTN (greater). In this case, reporting occurs if the number of exception matches is less than (greater than) the order of the regression test. If this exception tests true during the entire test, the test definition, the number of disturbed sensors and the match are recorded (66). For example, an exception to report a malfunction of less than 1 and a severity greater than 3.0 in a trip regression test

(Trip)

Exception: LTN Malf severity> 3.0

to be.

Each exception

Exception: modifier parameter operator constant

Has the form The parameter indicates which diagnostic result should be examined. The parameter is

Object attribute

Is defined. Valid subjects are malfunctions or procedures. Omitting the target part of the parameter indicates that both malfunction and procedure should be checked. When the preferred expert system is used, the attributes used by all categories except update exceptions are confidence factor (CF), severity (SEV), importance (IMP) or priority (PR).

The operator used in the exception is

Updated,! Updated, ==,! =, ＞, ＞ =, <, <=

to be. The first two operators are used only with update exceptions and indicate when any rule supported by the updated value has been triggered. The remainder operator defines the comparison between the parameter and the floating point number, and the third and fourth operators compare equivalent and inequality.

The test plan requirements may be implemented in test plan language as defined in Appendix I. These language definitions are available to those of ordinary skill in the art, and are available in YACC and LEX Unix third generation languages, which adopt a test plan textfile to output the appropriate internal representation of the plan. It is possible to generate parsers generated by development tools. An example of a test plan is

Rulebay Directory: PS: [harper. pds. code. regress. vb1]

TEST 1. (Normal)

Exception 1:! Updated

TEST 2. (Trip)

Exception 1: CF ＞ 0.800000

Exception 2: LTN CF ＞ 0.800000

As illustrated.

It is preferable that two resultant files are generated by the regression analyzer of the present invention. When the analyzer is run in batch mode, there is also an output file from the session. The output file indicates the progress of the test. The log file 24 is typically very long because it lists all sensors, their descriptions and test data, and all malfunctions and procedures, their descriptions and test plans and the individual test results. The exception result or report file 20 is easier to manage and lists all exception reports from the test.

Screen output 28 from the present invention preferably represents information and error messages generated during the execution of the PDS. Each test is identified and timestamped with the diagnostic cycle executed.

This output is similar to the file generated by the diagnostic version of the PDS. Examples of screen output can be found in Appendix II.

Log file 24 is a record of the entire test. This record is written to a conventional source code library, which can then be used to perform a regression comparison by comparing the complete test result with the complete test result of execution at the new rulebase change. A typical difference comparison between full log days shows exactly how changes have been made between the points at which the tests were run. In this way, regression analysis can be performed between rulebase changes. In addition, the log file clearly defines the input and output to the rulebase, so that it can be used later as a training example for other types of expert systems, such as, for example, neural network expert system implementations. As illustrated in Appendix II, it is preferable that the log file consists of three sections. The first section details the sensors and sensor data, malfunctions and procedures, and all their descriptions as test parameters. The second section is a list of test plans. The third section contains test cases. The malfunctions and procedures that were updated for each case are listed in alphabetical order along with their confidence coefficients, severity, importance and priority.

The example file 26 is a report on each case that matches the exceptions defined in the test plan. An exception file is essentially a filtered log file, in which case the filter characteristics are defined by the exception specified by the user. As illustrated in Appendix II, each report is

(<Time set> / <sensor> ...) <object> <message>

Takes the form of. As mentioned above, the timeset is a sensor data category, for example a trip or warning. Sensor is the name of the sensor that has been assigned a value other than normal during this diagnostic cycle. Subject is a malfunction or procedure name. The message is text that identifies the form of the example report.

The present invention is preferably implemented using a structural design method as described in the following document. `` Structured Design: Fundamentals of a Discipline of Computer Program Design, Yourdon and Constantine, Yourdan Press, 1979 '': `` Structural Analysis And System Specification, De Marco, Yourdan Press, 1979 '': 『Software Engineering: A Practitioner's Approach, Pressman , McGraw Hill Book Company, 1982: Tutorial on Software Design Techiques, Freemen and Wasserman, 4th Ed, IEEE Computer Society Press, 1983. More specifically, the present invention provides a CASE Analyst / RT Users Manual for VAX station / VMS Hosts VO. Which implements the methodology described in 'Structured Analysis And System Specification, De Marco, Yourdon Press, 1979'. It is preferably implemented using system development tools such as Mentor Graphics, 1988. This method or tool allows designers to create and maintain control and data flow diagrams that can be implemented quickly and effectively by standard and custom C routines.

4 and 5 illustrate the present invention using data flow and control flow diagrams in this methodology using the tools described above. The data definitions used when the present invention is implemented using this methodology are appended to Appendix III. Those skilled in the art will be able to implement the invention from the drawings and the information and appendices described above. 4 shows the input and output data flow for the test system 20 together with the information described above with respect to FIG. 4 shows that the production whiskers 100, i.e. plant information (unit information), should be included in the system.

As shown in Figures 4 and 5, the first step in the present invention is to read the test inputs in the test plan language by the user (200), and to parse these test plan inputs for internal testing. Create a plan representation 18. The next step is to initialize 202 all global variables. This step is part of the PDS expert system inference engine 16, which creates and initializes true, false, and other contexts, and creates and initializes PDS variables that manage sensor time steps. Next, the system is recovered 204, which is also part of the PDS inference engine 16. This routine takes a text file knowledge base definition and loads the corresponding scheme into memory 14. The next step is also to read 206 the unit information text file as part of the PDS inference engine. This file is part of the PDS production environment 100 associating the knowledge base with a particular orderer application, and as a side effect, the present invention uses unit names in all output file headers. Next, the system reads the sensor value (208), which is also part of the PDS inference engine 16. This step reads the sensor data text file in a standard format and loads the sensor values and time stamps into the internal reading list. PDS inference engine 16 then recovers 218 the history. To perform this step, the inference engine reads the history text file and loads the text history list and the event record into the corresponding scheme. The historical text file is part of the PDS production environment 100 and maintains all of the time-based analysis results that the inference engine restarts. Next, initialization of the data timeset is performed by the test system 20 (212). This step merely reads the sensor reading step 208 to fill a seven-column timeset array with the sensor's normal value, the low value causing the failure, the high value causing the failure, the predicted value, the diagnostic value, the warning value, and the trip value. Use the reading list created in Using arrays rather than text lists speeds up the test system. The next step, which is also part of the present invention, is to generate a regression log (214). This step summarizes and records the complete specification of the analysis in a log file by writing the date and time ordering application for this test plan, all the sensors and their seven data value descriptions, and a description of each and all possible malfunctions.

This step 214 opens and initializes the exception report and log files by filling in the file 26 the date and time and orderer application for this test plan. The next step 216 is also part of the test system 20, which writes the test plan read in the test plan reading step 200 to a log file.

This step attaches a copy of the test plan to a log file. The final step 218 is an execution step of repeatedly calling the PDS inference engine 16 to execute the diagnosis specified by the test plan. A detailed algorithm for this step is described in Appendix IV.

As mentioned above, the present invention provides an efficient and standard test tool for verifying a diagnostic rulebase even after changes and updates are made during development and commercial use. The present invention provides a language for describing these amenities along with log files that can be used to enable programmable exceptions, to train and test all types of expert systems, including neural networks. Provide a programmable test agenda.

Many features and advantages of the invention are apparent from the foregoing detailed description, and therefore are intended to include all features of the invention that fall within the true spirit and scope of the invention by the appended claims. In addition, since various excuses and modifications will be readily made by those skilled in the art, the present invention is not limited to the configurations and operations illustrated and described herein, so that all suitable modifications and equivalents may be classified or fall within the scope of the present invention. have. For example, some expert systems have the function of non-enforcement rules and PDS is one such system. In order to effectively reset the system to baseline, it may be possible to disable all rules in operation without starting all rules to normal values in steps 50 and 52.

Claims (7)

A test system for testing an expert system, comprising: test storage means for storing a normal test set and a speciallzec test set (22) and testing means (20) for testing the expert system using the normal test set and the special test set, the test means outputting an expert system in response to the special test set. Test system for questionnaire system testing, including exception means 64 for comparing the result to the example conditions and reporting when they match. The method of claim 1, wherein the test means 20 generates a logarithmic logic 24 that includes a test parameters, a test plan, and a test case. The test is run in response to a test plan 18 naming a test order, a test type, and exceptions, and the test plan input syntax is entered into the test plan 18. A test system for testing expert systems provided with a test language means (Appendix I) for conversion to. The test for expert system testing according to claim 1, wherein the test means (20) periodically applies the normal test set, i.e., the normal test set value, to the expert system, and performs the nth test when n is a toll. system. 4. The difference according to claim 1, 2 or 3, wherein the result storage means 24 for storing the test results before and after the change of the rule is compared with the results before and after the change of the rule. A test system for expert system testing, further comprising a comparison means (20) indicating. A test system for testing an expert system for a power plant, comprising: a test set comprising a normal sensor value set and a special set of sensor values with sensor fault, predicted, and diagnostic values Test storage means (22) for storing an array of and test means for periodically testing the expert system using the normal and special sensor values and repeatedly applying the special values ( 20) wherein the test means reads a test input in a test plan language and converts the test input into a test plan specifying the order and exception of the test to be tested. Reading means (200), exception means (64) for reporting when the expert system output is matched against said exception, and Logging means 24 for logging test cases including strands, sensors, sensor data, displayed malfunctions, malfunction descriptions, test plans, and special sensor value sets, and logs before and after changes to test plan language for testing. Comparing means for comparing to an expert system comprising: wherein the expert system allows the user to define a test plan that includes a sequence of tests, a test set to be used, expert system rules to be used, and test exceptions. A test system for expert system testing, comprising a language definition for enabling and a routine (200) for converting user input into the test plan available by the test system. A method of testing an expert system, the method comprising: (a) applying a normal test set to the expert system, wherein all input values are set to normal values; and (b) applying a special test set to the expert system. Wherein the one of the input values is set to a test value and the rules of the expert system are started to be executed, and (C) recording an exception generated during step (b), wherein the output of the expert system is output. Is compared with the exception and the step is generated when an exception is generated, and the step (a-c) is repeatedly selected while selecting a different one of the input values in step (b) to set the test value. Expert system testing method comprising the steps of repeating). 7. The method of claim 6, further comprising the step of reading a test requirement, reading an input by a user, and generating a test plan, wherein in step (b) the pair of input values is set to a pair of test values; Step (ac) is performed at rule base change in the expert system, and test output is compared before and after the rule base change.