Knowledge needed for expert system explanation

Download Knowledge needed for expert system explanation

Post on 14-Feb-2017




3 download


  • Knowledge needed for expert system explanation

    by WILLIAM R. SWARTOUT University of Southern California Marina del Rey, California


    Adequate explanations of the steps followed by an expert system can sometimes be produced automatically by paraphrasing the rules or methods of the system in English. However, this approach will not work for other kinds of explanations, such as justifications of the system's actions. The problem is that the basis for justifica-tions lies in knowledge used to create the expert system, which is not represented in the system's code and hence is unavailable for explanation. This paper outlines an approach to capturing the knowledge required to provide justifications and illustrates both the problem and the solution with examples of machine-generated English text.


    From the collection of the Computer History Museum (

  • From the collection of the Computer History Museum (


    Documentation of computer programs is a chronic problem. It is usually created, if at all, by a process (and an individual) not directly coupled to the development and maintenance of the program itself. Thus, the documentation is often unavail-able or out of date, uninformative, and not necessarily reflec-tive of the true content of the system. It is difficult to provide material responsive to all the many different kinds of ques-tions that lead users and would-be maintainers to consult the documentation. Automatic text generation raises the intrigu-ing possibility of creating systems that could provide their own documentation. In particular, it would be a significant im-provement if a computer system could describe its own inner workings-both how and why it does what it does.


    In the area of expert systems, some progress has been made toward achieving this goal. Some first-generation expert sys- . tems were able to provide explanations of how they per-formed various tasks by paraphrasing the rules or methods they used in English. 1, 2 For this to be successful, the rules or methods of the system had to be written in a stylized fashion, and the techniques that the system employed had to be rela-tively close to those that users would be familiar with, since the system's paraphrases mirrored the code directly. If that could be done, there was a major gain: Since the explanations were provided by translation of the code itself, any changes to that code were immediately reflected in the explanations; therefore the system's documentation was always consistent with the system itself.

    Unfortunately, the technique of paraphrasing the code is limited to describing what a system does or did. It cannot provide good explanations of why the system did what it did-that is, justifications of the system's actions. The problem is that the knowledge needed to support such explanations, the "rationale" behind the code, is not represented in the code itself and is hence unavailable to the paraphrasing routines.

    Let us illustrate some of these problems with an example from a first-generation version of the Digitalis Therapy Ad-visor,2,3 an expert system designed to help physicians in pre-scribing digitalis, a drug commonly given for certain cardiac problems. One aspect of digitalis therapy is that there are certain physiological abnormalities such as elevated serum calcium that may make some patients more sensitive to the drug. In Figure 1 we see the sorts of limitations that the translate-the-code approach suffers from in providing justifi-cations. Here, the system has asked the user to enter the level of serum calcium. The user, rather than entering an answer,

    Knowledge Needed for Expert System Explanation 95

    has asked "why?" indicating that he wants to know why the question is being asked. The advisor produced an explanation by translating its program stack. This explanation does sug-gest that serum calcium is a sensitizing factor for digitalis; but if the user wants to know why serum calcium makes a patient more sensitive to digitalis, he's out of luck, because that knowledge is not represented in the system. It is not repre-sented because it does not have to be for the program to perform correctly. Just as one can follow a recipe and bake a cake without ever knowing why the flour or baking powder is there, so too an expert system can deliver impressive per-formance without any representation of the reasoning under-lying its rules or methods. For the digitalis advisor to mimic expert behavior, all it needs to know is that it must ask for the value of serum calcium and reduce the digitalis dose if the serum calcium is above a certain threshold. But if we want to justify that expert behavior and explain why it is important to check serum calcium, much more is needed: The system must know that digitalis administration and elevated serum calcium have the similar effect of making the patient more likely to develop a dangerous arrhythmia and that therefore a danger-ous overdose must be avoided by adjusting the digitalis dose. This gap between the knowledge needed for performance and that needed for explanation is a recurring problem whenever one tries to retrofit an explanation capability to an existing performance expert system. (See also Reference 4 for a re-lated discussion.) This leads to the perhaps obvious conclu-sion that knowledge representation is a concern of text gener-ation because the quality of the text produced depends in part on the underlying knowledge representation. To put it more simply: A system has to know what it's talking about. The question to be addressed is: How can the knowledge needed to justify the system be represented?


    One way to capture the knowledge needed for justifications would be to require the system builder to annotate the system during creation with text strings justifying the system's meth-


    ENTER NUMERICAL VALUE = = = =) why? The user types "why?' indicating that he questions why he is being asked to supply the value of serum calcium.


    Figure I-A limited justification for a question

    From the collection of the Computer History Museum (

  • 96 National Computer Conference, 1985

    ods. These text strings would then be displayed during exe-cution in response to queries. 5 Clearly, the problem with this approach is that there is no way to ensure that the justifica- . tions provided accurately reflect what the code actually does. This problem is exacerbated by program maintenance and evolution. The approach we adopted in the XPLAIN system6

    depended on the observation that the programmer of an ex-pert system usually had the knowledge needed to provide good justifications of its actions, because the knowledge needed for justifications was also needed to create the pro-gram in the first place. The XPLAIN system replaced the human programmer with an automatic programmer that cre-ated an expert system (in this case, portions of the Digitalis. Therapy Advisor) from abstract knowledge about how the domain worked and about how problem solving was per-formed in that domain. As it created the expert system, its reasoning was recorded in a machine-readable form and was used to provide machine-produced justifications of the expert system's performance. The reason for using an automatic pro-grammer is that it assured us that the resulting expert system was consistent with the underlying domain knowledge that it was based upon, and hence that the explanations the system offered actually reflected the behavior of the expert system. Although, in general, automatic programming is hard, it was more tractable for us because algorithmically expert systems are relatively simple.

    The automatic programmer started with a single abstract goal. For example, in creating the digitalis advisor the goal was "administer digitalis." This goal was refined into more specific goals, such as "anticipate digitalis toxicity" and "as-sess toxicity," finally reaching the level of system primitives. (The system primitives were low-level operations that could not be further refined. They were similar to Lisp functions such as SETQ and COND.)

    The automatic programmer relied on two distinct bodies of knowledge in creating the program. The domain model con-sisted of descriptive facts about the domain of the expert system, such as causal relations, associations between diseases and symptoms, and a disease hierarchy. A simplified illustra-tion of the causal information contained in the domain model for digitalis administration is shown in Figure 2. The impor-tant thing to note is that though the domain model described important characteristics of the domain, it did not indicate how problem solving should be done: There was no knowl-

    Sinus Bradycardia Change to V. Fibrillation

    Figure 2-8implified domain model for digitalis therapy

    Goal: Anticipate Drug Toxicity

    Domain Rationale:


    Prototype Method:

    If the Finding exists then: reduce the drug dose else: maintain the d rug dose

    Figure 3-Principle for anticipating drug sensitivity

    edge about how to diagnose or treat a patient. That body of knowledge was contained in the domain principles. * The do-main principles were used to drive the refinement process forward.

    Domain principles had three major parts: a goal, a proto-type method,and a domain rationale. The goal of a domain principle stated what its prototype method could accomplish. For example, the domain principle whose goal was "anticipate digitalis toxicity" had a prototype method that stated in lower-level terms how to do that. When the automatic programmer tried to refine a goal, it examined the domain principles to find one whose goal most closely matched the goal to be refined. In this case, the principle that matched' was "antici-pate drug toxicity." The match succeeded, because digitalis is a kind of drug. A simplified version of this principle appears in Figure 3. It captures the commonsense notion that if one is considering administering a drug, and there is some factor that enhances the deleterious effects of that drug, then if that factor is present in the patient, less drug should be given. Notice that this principle is not specific to digitalis therapy but could be applied to any sort of drug therapy.

    Next, the domain rationale associated with the principle was matched against the domain model. In this case, the auto-matic programmer determined which particular findings should be checked by matching the domain rationale against the domain model. The domain rationale is the major fea-ture that distinguished the XPLAIN approach from other refinement-based systems.7 Its purpose is to integrate knowl-edge from the descriptive domain model into the refinement process by defining terms at one level of refinement with terms at the next level down.

    To elaborate, refinement-based systems move through dif-ferent levels of language: They refine a high-level description of a goal or problem into a low-level specific one. Yet the steps between levels of language are often quite implicit. In partic-ular, the correspondence between terms used at one level of language and their realization at the next level down is usually implicit. The domain rationale allowed us to indicate that

    * The term domain principles is a bit misleading. Though some of the domain principles were quite domain-specific, others (such as principles that set up backward chaining control) were largely domain-independent.

    From the collection of the Computer History Museum (

  • Please enter the value of serum potassium: why?

    The system is anticipating digitalis toxicity. Decreased serum potassium causes increased automaticity, which may cause a change to ventricular fibrillation. Increased digitalis also causes increased automaticity. Thus, If the system observes decreased serum potaSSium, it reduces the dose of digitalis due to decreased serum potassium.

    Please enter the value of serum potassium: 3.7

    Please enter the value of serum calcium: why?

    (The system produces a shortened explanation, reflecting the fact that it has already explained several of the causal relationships in the previous explanation. Also, since the system remembers that it has already told the user abovt serum potassivm, it svggests the analogy between the two here.)

    The system is anticipating digitalis toxiCity. Increased serum calcium also causes increased automaticity. Thus, Cas with decreased serum potassium) if the system observes increased serum calcium, it reduces the dose of digitalis due to increased serum calcium.

    Please ente r the value of serum calcium: 9

    Figure 4-An explanation of why serum potassium and calcium are checked; produced by XPLAIN

    correspondence. Here, for example, the high-level descrip-tion of the problem is "anticipate toxicity," whereas the low-level method deals with "findings." The domain rationale defines which findings should be checked in anticipating tox-icity, namely, those findings that cause something dangerous to happen that is also caused by an increased level of the drug.

    In this case, there were two matches for the domain ra-tionale: one for increased serum calcium and another for de-creased serum potassium. The prototype method, which was an abstract method for accomplishing the goal, was instan-tiated twice, once for each match of the domain rationale. In turn, the instantiations were themselves refined; and methods were found for reducing the dose, determining whether in-creased serum calcium existed, and so forth. This process continued until the level of system primitives was reached.

    The entire refinement process was recorded so that at run-time the system's explanation generator could produce justifi-cations such as those in Figure 4, which shows not only what the system does, but also why serum calcium and potassium should be considered sensitivities.

    The XPLAIN system has been used to implement major portions of the Digitalis Therapy Advisor, and its applicability to MYCIN has been examined.8 It appears that MYCIN could be reimplemented by using XPLAIN and that the resulting explanations would be clearer than those produced by MY-CIN, but we have not actually performed the reimplementa-tion.


    Although we were primarily concerned with providing justifi-cations, several observations emerged during the construction of XPLAIN that relate to making it easier to modify and extend expert systems.

    The separation of descriptive and problem-solving knowl-edge is crucial. Ii was necessary to separate descriptive knowl-edge (Le., the domain "facts" such as causal relations and associations between symptoms and disease) and problem-solving knowledge and then record how the two were brought together to be able to provide justifications. But the sepa-

    Knowledge Needed for Expert System Explanation 97

    ration of knowledge also has important benefits for system maintenance. In most expert system frameworks, descriptive knowledge and problem-solving knowledge are inextricably intertwined. A general principle may never be explicitly rep-resented, but instead may be encoded as many individual rules tailored to specific situations by the particulars of the domain. (See References 4 and 8 for good discussions of this point.) The XPLAIN approach permits us to represent gen-eral principles, and we rely on the automatic programmer to use descriptive knowledge to instantiate the general principles in specific situations. This can make maintenance of expert systems substantially easier. In a conventional framework, if we want to change a general principle, we must carefully modify all the rules encoding that principle, presenting us with many opportunities for making a mistake. In XPLAIN, we just have to modify the general principle itself and rerun the automatic programmer. Additionally, since principles are represented at a more abstract level, it will be possible to export some of them to new problem areas--something that would be quite difficult in a conventional framework, due to the mixing of descriptive and problem-solving knowledge. For example, the principle for anticipating toxicity is applicable to most, if not all, drugs with side effects and could be used directly in creating an advisor for a new drug.

    The creation of an expert system is more principled. In build-ing conventional expert systems, system builders reason about the deep knowledge underlying their systems in their heads. In the XPLAIN approach, the system builder is encouraged to represent deep knowledge explicitly so that the automatic programmer can reason about it mechanically. This method makes the system builder think more closely about the do-main knowledge and problem-solving strategies adopted. The result is that flaws are found that might otherwise be over-looked. For example, an early version of the digitalis advisor, written before our work on XPLAIN, treated abnormal thy-roid function (myxedema) as a sensitizing factor for digitalis. When we reimplemented the advisor, using XPLAIN, we found that myxedema did not seem to meet the requirements imposed by the domain principles for something to be consid-ered a sensitivity. A careful examination of the medical lit-erature revealed that in fact, myxedema is not a sensitivity; instead, it impedes digitalis excretion. (See Reference 6, Sec-tion 7.1.) Thus, by imposing some additional structure on the wayan expert system is constructed, the XPLAIN approach can result in a sounder, more principled system.


    The Explainable Expert Systems (EES) project at ISI9 is ex-tending the XPLAIN paradigm. In particular, we are trying to represent additional kinds of knowledge needed to support additional kinds of questions. A broad view of the EES system is shown in Figure 5.

    We provide system builders with a more expressive knowl-edge base that encourages them to represent knowledge more abstractly and explicitly separate out the different kinds of knowledge that go into an expert system. Like XPLAIN, EES will make use of a domain model and a set of domain prin-ciples. In addition, we will model additional kinds of knowl-

    From the collection of the Computer History Museum (

  • 9-8 National Computer Conference, 1985

    Knowledge Base

    fDes~~f~f~~~~ainll--___________ --' L_~~_,,-~_a_i~_~_"-~=~) __ J :-Pr~-bl""e-m:;~ivi~g--: : Knowledge :

    l~~~-~~~~~!~~~~I::~J : Tradeoff.1 : ~ .. __ ~r_e_f~!~~_c_e_s ____ J .-------------------, , , : Terminology :

    l. ___ ~~~~~~t~~~~ ____ J ,-------------------, , , : Integration :

    ~ ____ ~~~~~~~:~ ____ J ,..------------------, 1 Optimization 1

    t ____ ~~~_~~~~~~ ____ j 1--------... 1 '---:-----'

    Figure 5-Global view of the EES framework

    edge that will allow the system to answer additional kinds of questions. Tradeoffs are associated with domain principles to indicate the beneficial and harmful effects of selecting a par-ticular strategy to achieve a goal. Preferences are associated with goals and are used to set priorities based on tradeoffs. Together, tradeoffs and preferences will provide the system with the knowledge it needs to explain the choices it made among alternative problem-solving strategies. Mappings be-tween abstract terms and the concepts that realize them (which had been included as part of domain principles in XPLAIN) are broken out as a separate type of knowledge to allow terminology to be shared across domain principles. Inte-gration knowledge is used to resolve potential conflicts among knowledge sources. Optimization knowledge represents ways of efficiently controlling the execution of the derived expert system and must be explicit to explain possibly unusual order-ings of actions that may result from efficiency considerations.

    Similar to XPLAIN, the EES framework will generate a runnable expert system by applying the program writer com-ponent to the knowledge base. The steps taken in producing code are recorded in a development history, a lattice structure whose leaf nodes represent system implementation code and whose interior nodes represent goals and decisions made on the way to generating the implementation. The interpreter executes the leaf nodes of the development history. It pro-duces an execution trace and manages the system's normal interaction with users. User questions, however, are sent to the explanation generator, which accesses the knowledge base, development history, interpreter, and execution trace in the course of constructing answers to queries.


    We have argued that one of the failings of current expla-nations of expert systems is that they cannot justify what the

    system does because the knowledge needed to support such explanations is missing. The XPLAIN system captured that reasoning by using an automatic programmer to create an expert' system and recording the decisions it went through while creating the system so that they would be available later to provide justifications. We observed that the organization this approach imposes can also be beneficial to the mainte-nance, evolvability, and consistency of an expert system. Our current work is extending the XPLAIN paradigm to represent additional kinds of knowledge needed to support additional explanations.


    The research described here was supported under DARPA Grant #MDA 903-81-C-0335 and National Institutes of Health Grant #1 POl LM 03374-01 from the National Library of Medicine.

    I would like to thank Bob Balzer, Jeff Barnett, Jack Mostow, Bob Neches, Ramesh Patil, and Peter Szolovits for many enlightening discussions that led to the ideas discussed here.


    1. Buchanan, Bruce G., and Edward H. Shortliffe. Rule-Based Expert Systems: The MYCIN Experiments of the Stanford Heuristic Programming Project. Reading, Mass.: Addison-Wesley, 1984.

    2. Swartout, W. R. A Digitalis Therapy Advisor with Explanations. Technical Report TR-176, Laboratory for Computer Science, Massachusetts Institute of Technology, February 1977.

    3. Gorry, G. A., H. Silverman, and S. G. Pauker. "Capturing Clinical Ex-pertise: A Computer Program that Considers Clinical Responses to Digi-talis." American Journal of Medicine, 64 (1978), pp. 452-460.

    4. Clancey, W. "The Epistemology of a Rule-Based Expert System-A Frame-work for Explanation." Artificial Intelligence, 20 (1983), pp. 215-251.

    5. Chandrasekaran, B., and S. Mittal. "Deep versus Compiled Knowledge Approaches to Diagnostic Problem-Solving." In AAAI Proceedings of the National Conference on Artificial Intelligence. William Kaufman, Inc., Menlo Park, CA, 1982.

    6. Swartout, W. "XPLAIN: A System for Creating and Explaining Expert Consulting Systems." Artificial Intelligence, 21 (1983), pp. 285-325. (Also available as Information Sciences Institute publication RS-83-4.)

    7. Sacerdoti, E. A Structure for Plans and Behavior. Technical Report TN-1OO, Stanford Research Institute, Menlo Park, CA, 1975.

    8. Szolovits, P. "Toward More Perspicuous Expert System Organization." In "Report on Workshop on Automated Explanation Production," W. Swartout (ed.), SIGART Newsletter, ACM, July, 1983.

    9. Neches, R., W. Swartout, and J. Moore. "Enhanced Maintenance and Ex-planation of Expert Systems through Explicit Models of Their Develop-ment." In Proceedings of the IEEE Workshop on Principles of Knowledge-Based Systems. IEEE, Silver Spring, MD, December 1984.

    From the collection of the Computer History Museum (


View more >