Machine Learning for Data Mining in Medicine

Nada Lavrac

J. Stefan Institute, Jamova 39, 1000 Ljubljana, Slovenianada.lavrac@ijs.si

Abstract. Large collections of medical data are a valuable resourcefrom which potentially new and useful knowledge can be discoveredthrough data mining. This paper gives an overview of machine learn-ing approaches used in mining of medical data, distinguishing betweensymbolic and sub-symbolic data mining methods, and giving referencesto applications of these methods in medicine. In addition, the paperpresents selected measures for performance evaluation used in medicalprediction and classification problems, proposing also some alternativemeasures for rule evaluation that can be used in ranking and filtering ofinduced rule sets.

1 Introduction

Large collections of medical data are a valuable resource from which potentiallynew and useful knowledge can be discovered through data mining. Data miningis an increasingly popular field including statistical, visualization, machine learn-ing, and other data manipulation and knowledge extraction techniques aimed atgaining an insight into the relationships and patterns hidden in the data.

Machine learning methods [51] described in this paper can be classified intosymbolic and sub-symbolic methods. Examples of symbolic methods are ruleinduction methods such as learning of if-then rules [13], decision and regressiontrees [57] and logic programs [37], and case-based reasoning. Examples of sub-symbolic methods are instance-based learning methods [17,3], artificial neuralnetworks [60] and Bayesian classification [33,34]. These induction methods aremostly concerned with the analysis of classificatory properties of data tables.Data represented in a tables may be collected from measurements or acquiredfrom experts. Rows in the table correspond to objects (training examples) tobe analyzed in terms of their properties (attributes) and the class (concept)to which they belong. In a medical setting, a concept of interest could be aset of patients with a certain disease or outcome. Supervised learning assumesthat training examples are classified whereas unsupervised learning concerns theanalysis of unclassified examples.

In medical problem solving it is important that a decision support systemis able to explain and justify its decisions. Especially when faced with an un-expected solution of a new problem, the user requires substantial justificationand explanation. Hence the interpretability of induced knowledge is an impor-tant property of systems that induce solutions from medical data about past

W. Horn et al. (Eds.): AIMDM’99, LNAI 1620, pp. 47–62, 1999.c© Springer-Verlag Berlin Heidelberg 1999

48 N. Lavrac

solved cases. Symbolic data mining methods have this property since they in-duce symbolic representations (such as decision trees) from data. On the otherhand, sub-symbolic data mining methods typically lack this property which hin-ders their use in situations for which explanations are required. Nevertheless,when the classification accuracy is the main applicability criterion, sub-symbolicmethods may turn out to be very appropriate since they typically achieve accu-racies that are at least as good as (or frequently better than) those of symbolicclassifiers.

The first part of this paper gives an overview of machine learning approachesused in mining of medical data, distinguishing between symbolic and sub-symbolicdata mining methods. Due to space restrictions this overview does not containillustrative examples; for a subset of described methods, these can be found in[39]. It also gives references to applications of these methods in medicine, someof which are described in detail in an edited volume on intelligent data analy-sis in medicine and pharmacology [38]. The second part of this paper describesselected measures for performance evaluation used in medical prediction andclassification problems, proposing also some alternative measures for rule evalu-ation that can be used in ranking and filtering of induced rule sets. Some othermeasures not discussed here are described in [39,41].

2 Symbolic Data Mining Techniques

2.1 Rule Induction

If-Then Rules Given a set of classified examples, a rule induction system con-structs a set of rules of the form IF Conditions THEN Conclusion, or, if usinga different notation, Conditions→ Conclusion. The condition part of a rule isa conjunction of attribute tests of the form Ai = value for discrete attributes,and Ai < value or Ai > value for continuous attributes. The conclusion part isclass assignment C = ci.

An example is covered by a rule if the attribute values of the example fulfillthe conditions of the rule. An example rule induced in the domain of earlydiagnosis of rheumatic diseases [36,22] assigning the diagnosis crystal-inducedsynovitis to male patients older than 46 who have more than three painful jointsand psoriasis as a skin manifestation, is represented as follows: “IF Sex = maleAND Age > 46 AND Number-of-painful-joints > 3 AND Skin-manifestations =psoriasis THEN Diagnosis = Crystal-induced-synovitis”.

The well-known algorithms of the AQ falimy of rule learners [48] use thecovering approach to construct a set of rules for each possible class ci in turn:when rules for class ci are being constructed, examples of this class are positive,all other examples are negative. The covering approach works as follows: AQconstructs a rule that correctly classifies some examples, removes the positiveexamples covered by the rule from the training set and repeats the process untilno more examples remain. To construct a single rule that classifies examplesinto class ci, AQ starts with a rule with an empty antecedent (IF part) and the

Machine Learning for Data Mining in Medicine 49

selected class ci as a consequent (THEN part). The antecedent of this rule issatisfied by all examples in the training set, and not only those of the selectedclass. AQ then progressively refines the antecedent by adding conditions to it,until only examples of class ci satisfy the antecedent. To allow for the handlingof imperfect data, some if-then rule learning algorithms may construct a set ofrules which is imprecise, i.e., does not classify all examples in the training setcorrectly.

If-then rule induction algorithms, such as AQ15 [49] and CN2 [13,12] havebeen frequently applied to the analysis of medical data. Examples of medicalapplications include [49,22,35].

Rough Sets If-then rules can be also induced by using algorithms based on thetheory of rough sets introduced by Pawlak [54,55]. Rough sets (RS) are concernedwith the analysis of classificatory properties of data aimed at approximations ofconcepts. RS can be used both for supervised and unsupervised learning.

The main goal of RS analysis is the synthesis of approximations of conceptsci. The basic concept of RS is an indiscernibility relation. Two objects x and yare indiscernible if their object descriptions have the same values of attributes.A main task of RS analysis is to find minimal subsets of attributes that preservethe indiscernibility relation. This is called reduct computation. Decision rulesare generated from reducts by reading off the values of the attributes in eachreduct. The main challenge in inducing rules lies in determining which attributesshould be included in the conditional part of the rule. Rules synthesized fromthe (standard) reducts will usually result in large sets of rules and are likely toover-fit the patterns of interest. Instead of standard reducts, attribute sets that“almost” preserve the indiscernibility relation can be generated.

The list of applications of RS in medicine is significant. It includes extractingdiagnostic rules, image analysis and classification of histological pictures, model-ing set residuals, EEG signal analysis, etc. Examples of RS analysis in medicineinclude [26,32,63].

Association Rules The problem of discovering association rules [2] has recentlyreceived much attention in the data mining community. The problem is definedas follows: Given a set of transactions, where each transaction is a set of itemsof the form Attribute = value, an association rule is an expression of the formB → H where B and H are sets of items. The intuitive meaning of such a ruleis that transactions in a database which contain B tend to contain H .

An example of such a rule is: Diagnosis = pneumonia → Fever = high[C=80,S=10]. The meaning of this rule is: “80% of patients with pneumonia alsohave high fever. 10% of all transactions contain both these items.” Here 80% iscalled confidence of the rule, and 10% support of the rule. Confidence of therule is calculated as the ratio of the number of records having true values for allitems in B and H to the number of records having true values for all items in B.Support of the rule is the ratio of the number of records having true values for allitems in B and H to the number of all records in the database. The problem is to

50 N. Lavrac

find all association rules that satisfy minimum support and minimum confidenceconstraints. To do so, all itemsets that satisfy the minimum support level arefirst generated, and then all frequent itemsets are combined with each other toproduce all possible rules satisfying the minimum confidence constraint.

Association rule learning was applied in medicine, for example, to identifynew and interesting patterns in surveillance data, in particular in the analysis ofthe Pseudomonas aeruginosa infection control data [8]. An algorithm for findinga more expressive variant of association rules, where data and patterns are rep-resented in first-order logic, was successfully applied to the problem of predictingcarcinogenicity of chemical compounds [18].

Ripple Down Rules The knowledge representation of the form of ripple downrules allows incremental rule learning by including exceptions to the current ruleset. Ripple down rules (RDR) [14,15] have the following form:IF Conditions THEN Conclusion BECAUSE Case EXCEPT IF... ELSE IF...

There have been many successful medical applications of the RDR approach,including the system PEIRS [23] which is an RDR reconstruction of the hand-built GARVAN expert system knowledge base on thyroid function tests [28].

2.2 Learning of Classification and Regression Trees

Systems for Top-Down Induction of Decision Trees (TDITD) [57] generate adecision tree from a given set of examples. Each of the interior nodes of the treeis labeled by an attribute, while branches that lead from the node are labeledby the values of the attribute.

The tree construction process is heuristically guided by choosing the ‘mostinformative’ attribute at each step, aimed at minimizing the expected number oftests needed for classification. A decision tree is constructed by repeatedly callinga tree construction algorithm in each generated node of the tree. In the currentnode, the current training set is split into subsets according to the values of themost informative attribute, and recursively, a subtree is built for each subset.Tree construction stops when all examples in a node are of the same class. Thisnode, called a leaf, is labeled by a class name. However, leaves can also be empty,if there are no training examples having attribute values that lead to a leaf, orcan be labeled by more than one class name (if there are training examples withsame attribute values and different class names), together with the probabilityassigned to each class.

The best known decision tree learner is C4.5 [59] (C5.0 is its recent up-grade) which has also been incorporated into commercial data mining tools (e.g.,Clementine and Kepler). The system is is widely used since it is well maintainedand documented, reliable, efficient and capable of dealing with large numbers oftraining examples.

There have been numerous applications of decision tree learning in medicine,e.g., [35,40,38].

Learning of regression trees is similar to decision tree learning: it also usesa top-down greedy approach to tree construction. The main difference is that

Machine Learning for Data Mining in Medicine 51

decision tree construction involves the classification into a finite set of discreteclasses whereas in regression tree learning the decision variable is continuousand the leaves of the tree either consist of a prediction into a numeric value ora linear combination of variables (attributes). An early learning system CART[7] featured both classification and regression tree learning.

2.3 Inductive Logic Programming

Inductive logic programming (ILP) systems learn relational concept descriptionsfrom relational data. In ILP, induced rules typically have the form of Prologclauses. Compared to rules induced by a rule learning algorithm of the formIF Conditions THEN Conclusion, Prolog rules have the form Conclusion :-Conditions, denoting Conclusion← Conditions. Conclusion is a target pred-icate to be learned, and Conditions a conjunction of literals.

The best known ILP systems include FOIL [58] and Progol [52] and Claudien[19]. LINUS is an environment for inductive logic programming [37], enablinglearning of relational descriptions by transforming the training examples andbackground knowledge into the form appropriate for attribute-value learners.

ILP has been successfully applied to medical data analysis, including early di-agnosis of rheumatic diseases [37] and carcinogenesis prediction in the predictivetoxicology evaluation challenge [62].

2.4 Case-Based Reasoning

Case-based reasoning (CBR) uses the knowledge of past experience when dealingwith new cases [1,43]. A “case” refers to a problem situation — although, as withinstance-based learning [3], cases may be described with a simple attribute-valuevector, CBR most often uses a richer, often hierarchical data structure. CBRrelies on a database of past cases that has to be designed in the way to facilitatethe retrieval of similar cases.

Several CBR systems were used, adapted for, or implemented to supportreasoning and data analysis in medicine. Some are described in the Special Issueof Artificial Intelligence in Medicine [44] and include CBR systems for reasoningin cardiology, learning of plans and goal states in medical diagnosis, detectionof coronary heart disease from myocardial scintigrams, and treatment advicein nursing. Other include a system that uses CBR to assist in the prognosis ofbreast cancer [45], case classification in the domain of ultrasonography and bodycomputed tomography [29], and a CBR-based expert system that advises on theidentification of nursing diagnoses in a new client [6].

3 Sub-symbolic Data Mining Methods

3.1 Instance-Based Learning

Instance-based learning (IBL) algorithms [3] use specific instances to performclassification tasks, rather than generalizations such as induced if-then rules.

52 N. Lavrac

IBL algorithms assume that similar instances have similar classifications: novelinstances are classified according to the classifications of their most similar neigh-bors.

IBL algorithms are derived from the nearest neighbor pattern classifier [25,16].The nearest neighbor (NN) algorithm is one of the best known classification al-gorithms and an enormous body of research exists on the subject [17]. In essence,the NN algorithm treats attributes as dimensions of an Euclidean space and ex-amples as points in this space. In the training phase, the classified examples arestored without any processing. When classifying a new example, the Euclideandistance between that example and all training examples is calculated and theclass of the closest training example is assigned to the new example.

The more general k-NN method takes the k nearest training examples anddetermines the class of the new example by majority vote. In improved ver-sions of k-NN, the votes of each of the k nearest neighbors are weighted bythe respective proximity to the new example [21]. An optimal value of k maybe determined automatically from the training set by using leave-one-out cross-validation [64]. In our experiments in early diagnosis of rheumatic diseases [22],using the Wettschereck’s implementation of k-NN [65], the best k was chosen inthis manner.

3.2 Artificial Neural Networks

Artificial neural networks may be used for both supervised and unsupervisedlearning.

For unsupervised learning — learning which is presented with unclassifiedinstances and aims to identify groups of instances with similar attribute values— the most frequently used neural network approach is that of Kohonen’s selforganizing maps (SOM) [31]. Typically, SOM consist of a single layer of outputnodes. An output node is fully connected with nodes at the input layer. Eachsuch link has an associated weight. There are no explicit connections betweennodes of the output layer.

For supervised learning and among different neural network paradigm, feed-forward multi-layered neural networks [60,24] are most frequently used for mod-eling medical data. They are computational structures consisting of a intercon-nected processing elements (PE) or nodes arranged on a multi-layered hierar-chical architecture. In general, PE computes the weighted sum of its inputs andfilters it through some sigmoid function to obtain the output. Outputs of PEs ofone layer serve as inputs to PEs of the next layer. To obtain the output value forselected instance, its attribute values are stored in input nodes of the network(the network’s lowest layer). Next, in each step, the outputs of the higher-levelprocessing elements are computed (hence the name feed-forward), until the resultis obtained and stored in PEs at the output layer.

Multi-layered neural networks have been extensively used to model med-ical data. Example applications areas include survival analysis [42], clinicalmedicine [5], pathology and laboratory medicine [4], molecular sequence analy-sis [66], pneumonia risk assessment [10], and prostate cancer survival [30]. There

Machine Learning for Data Mining in Medicine 53

are fewer applications where rules were extracted from neural networks: an ex-ample of such data analysis is finding rules for breast cancer diagnosis [61].

Other types of neural networks for supervised learning include Hopfield recur-rent network and neural networks based on adaptive resonance theory mapping(ARTMAP). For the first, an example application is tumor boundary detec-tion [67]. Example studies of application of ARTMAP in medicine include clas-sification of cardiac arrhythmias [27] and treatment selection for schizophrenicand unipolar depressed in-patients [50]. Learned ARTMAP networks can alsobe used to extract symbolic rules [9,20].

3.3 Bayesian Classifier

The Bayesian classifier uses the naive Bayesian formula to calculate the proba-bility of each class cj given the values of all the attributes for a given instanceto be classified [33,34]. For simplicity, let (v1..vn) denote the n-tuple of valuesof example ek to be classified. Assuming the conditional independence of theattributes for the given class, it can be shown that p(cj |v1..vn) is proportional top(cj)

∏i

p(cj |vi)p(cj)

; these probabilities can be in turn estimated from the trainingset, using the relative frequency, the Laplace estimate [53] or the m-estimate[11]. Given the above formula for p(cj |v1..vn), a new instance is classified intothe class with maximal probability.

The Naive Bayesian formula can also be used to support decisions in differentstages of a diagnostic process [46,47] in which doctors use hypothetico-deductivereasoning gathering evidence which may help to confirm a diagnostic hypothe-sis, eliminate an alternative hypothesis, or discriminate between two alternativehypotheses. As shown by [46], Bayesian computation can help in identifying andselecting the most useful tests, aimed at confirming the target hypothesis, elim-inating the likeliest alternative hypothesis, increase the probability of the targethypothesis, decrease the probability of the likeliest alternative hypothesis or in-crease the probability of the target hypothesis relative to the likeliest alternativehypothesis.

4 Evaluation Measures for Predictive Data Mining inMedicine

Predictive induction deals with supervised learning for prediction and/or classi-fication tasks.

4.1 Performance Evaluation

Given a set of training examples, quality of classifications can be tested in twoways.

1. Testing on a training set. This case applies in the construction of rules whereheuristic measures are used for rule evaluation and/or feature selection. This

54 N. Lavrac

approach is used also when we want all the available information to be usedfor learning. Moreover, assigning a quality measure to a rule allows for theirranking, and consequently, for rule filtering.

2. Testing on a separate test set. In this case standard approaches to qualityevaluation can be applied like leave-one-out, cross-validation, etc.

The evaluation measures developed below are appropriate for both evaluationframeworks.

4.2 Confusion Matrix

Consider a binary classification problem (given only two classes: positive andnegative). In this case, a confusion matrix is used as a basis for performanceevaluation.

predicted predictedpositive negative

actual positive TP FN Pa

actual negative FP TN Na

Pp Np N

Table 1. A confusion matrix.

In the confusion matrix shown in Table 1 the following notation is used. Pa

denotes the number of positive examples, Na the number of negative examples,Pp the examples predicted as positive by a classifier, and Np the examples pre-dicted as negative. The fields of the confusion matrix contain the numbers ofexamples of the following four subsets (between brackets the symbol denotingthe number of examples in each subset is indicated):

True positives (TP ): True positive answers denoting correct classifications ofpositive cases.

True negatives (TN): True negative answers denoting correct classificationsof negative cases.

False positives (FP ): False positive answers denoting incorrect classificationsof negative cases into class positive.

False negatives (FN): False negative answers denoting incorrect classifica-tions of positive cases into class negative.

In the fields of the confusion matrix, for the convenience of computation, theabsolute numbers may be replaced by the relative frequencies, e.g., TP by TP

N ,and Pa by Pa

N . This may be more convenient when relative frequencies are usedas probability estimates.

Machine Learning for Data Mining in Medicine 55

4.3 Standard Performance Evaluation Measures

The classification accuracy is the most popular performance evaluation measureused in predictive knowledge discovery where the goal of learning is predictionor classification. The classification accuracy measures the proportion of correctlyclassified cases.

In binary classification problems using the confusion matrix notation, theaccuracy is computed as follows:

Acc =TP + TN

TP + TN + FP + FN=

TP + TN

N(1)

Notice that this performance evaluation measure is symmetric w.r.t. the ac-curacy of predictions of the positive and negative examples.1

If in binary classification problems we were only interested in the correctpredictions of the target class, accuracy could also be defined as the fraction ofpredicted positives that are true positives. Let R be the induced classifier, andAcc(R) the accuracy of correct predictions.

Acc(R) =TP

TP + FP=

TP

Pp(2)

This measure, if applied to single rules, is called confidence in associationrule learning, and precision in information retrieval. Accuracy error, derivedfrom accuracy, is defined as Err(R) = 1 −Acc(R) = FP

Pp. Accuracy can also be

used to measure the reliability of the classifier in the prediction of positive casessince it measures the correctness of returned results.

The reliability of negative predictions is defined as follows:

NegRel(R) =TN

TN + FN=

TN

Np

.Sensitivity is a measure frequently used in medical applications. It measures

the fraction of actual positives that are correctly classified. In medical terms,maximizing sensitivity means detecting as many ill patients as possible.

Sens(R) =TP

TP + FN=

TP

Pa

This measure is identical to recall known from information retrieval (recall ofpositive cases).

Specificity is also a measure frequently used in medical applications. Speci-ficity can be interpreted as recall of negative cases:

Spec(R) =TN

TN + FP=

TN

Na

1 For multi-class problems (k classes), let TPj denote the fraction of correctly classifiedinstances of class cj ; then the classification accuracy can be computed as follows:

Acc = 1N

∑k

1TPj .

56 N. Lavrac

Maximizing specificity is equivalent to minimizing the false alarm rate, whereFalseAlarm(R) = 1 − Spec(R) = FP

TN+FP . In medicine, this measure is aimedat minimizing the fraction of healthy patients declared as ill.

4.4 Rule Set versus Single Rule Evaluation

In this section we consider only symbolic learning, where the outputs are pre-diction or classification rules, induced from classified examples.

A learner usually induces a set of rules and not a single rule. Consider a setof rules of the form Hi ← Bi, where Hi is class assignment C = ci and Bi isa conjunction of attribute tests. Notice, however, that the form Hi ← Bi alsocovers association rules, in which Hi does not represent class assignment buttypically a conjunction of attribute tests.

1. If all the Hi are identical, i.e., ∀i: Hi = H, this set of rules can be consideredas a hypothesis about H. In this case, a hypothesis can be written in theform of a single rule H ← B, where B is a disjunction of all Bi. Underthese circumstances, the evaluation of a hypothesis (a set of rules) can beelaborated in the same framework as the evaluation of single rules.

2. If Hi are not identical, one can not form a hypothesis as a single rule. Inthis case the evaluation of a set of rules can not be elaborated in the sameframework as the evaluation of single rules, and evaluation measures for aset of rules have to be defined.

4.5 Non-standard Measures for Rule Evaluation

The measures outlined in Section 4.3 can be used for the evaluation of classifiersperformance (e.g., performance of a set of rules), as well as the evaluation ofsingle rules. In addition to the above standard measures, other measures thatevaluate the quality of single rules can be defined [41]. In data mining thesemeasures may turn out to be important when trying to extract individual rulesrepresenting meaningful regularities hidden in the data.

Let us use the notation in which n(X) denotes the cardinality of X , e.g.,n(HiBi) is the number of instances for which Hi is true and Bi is true (i.e., thenumber of instances correctly covered by the rule), n(Bi) is the set of instancesfor which Bi is false (instances not covered by the rule, etc. N denotes the totalnumber of instances in the sample. The relative frequency n(X)

N associated withX is denoted by p(X).

Accuracy of rule Ri = Hi ← Bi is here defined as the conditional probabilitythat Hi is true given that Bi is true:

Acc(Hi ← Bi) = p(Hi|Bi)

.

Machine Learning for Data Mining in Medicine 57

Acc(H ← B) defined as p(H |B) indeed measures the fraction of predictedpositives that are true positives in the case of binary classification problems, asdefined in Equation 2:

Acc(R)=TP

TP + FP=

n(HB)n(HB) + n(HB)

=n(HB)n(B)

=n(HB)

Nn(B)

N

=p(HB)p(B)

=p(H |B).

Relative accuracy, defined as

RAcc(Hi ← Bi) = p(Hi|Bi)− p(Hi)

is the accuracy gain of a rule Hi ← Bi relative to a rule Hi ← true that wouldclassify every instance into Hi (i.e., class ci). This measure indicates that a ruleis only interesting if it improves upon the ‘default’ accuracy p(Hi).

The point about relative measures is that they give more information aboutthe utility of a rule than absolute measures. For instance, if in a prediction taskthe accuracy of a rule is lower than the relative frequency of the class it predicts,then the rule actually performs badly, regardless of its absolute accuracy.

It is now useful to provide another view on performance evaluation. In pre-dictive induction it is well understood that a rule may be considered useful onlyif its predictive accuracy is higher than the accuracy of the classifier that clas-sifies all examples into the majority class. This understanding is incorporatedalso into building an ordered list of rules by a rule learner CN2 [13] which stopsbuilding new rules once their accuracy drops below the accuracy defined by themajority class threshold. If H0 denotes the majority class in predictive induc-tion dealing with multi-class problems, the majority class threshold is defined asp(H0) = n(H0)

N .Relative threshold accuracy is defined as follows:

TRAcc(R) = p(H |B)− p(H0)

where H0 denotes the majority rule head.There is however a problem with relative accuracy as such: it is easy to obtain

high relative accuracy with highly specific rules, i.e., rules with low generalityp(B) (low proportion of examples covered by the body of a rule). To overcomethis problem, a weighted variant of the relative accuracy measure is introduced.

Weighted relative accuracy trades off generality and relative accuracy. It isdefined as follows:

WRAcc(H ← B) = p(B)(p(H |B) − p(H)).

It was shown in [41] that rules with high weighted relative accuracy also havehigh novelty. High novelty is achieved by trading off generality and rule accuracygained in comparison with a trivial rule H ← true. This also means that havinghigh relative accuracy is not enough for considering a rule to be interesting, sincethe rule needs to be general enough as well.

58 N. Lavrac

4.6 Other Rule Evaluation Measures

With new tasks being addressed in knowledge discovery, the development ofnew evaluation measures is important. Additional measures are needed as manyknowledge discovery tasks involve the induction of a large set of redundant rulesand the problem is the ranking and filtering of the induced rule set. It was shownin previous work [41] that relative and weighted relative measures can be intro-duced not only for accuracy but also for the reliability of negative predictions,as well as for sensitivity and specificity. The utility of these measures for datamining in medicine still needs to be empirically evaluated in further work.

The paper [39] presents also some other evaluation measures, including post-test probability (which is appropriate for stepwise diagnostic processes), infor-mation score (which is similar in spirit to the above discussed relative accuracyfor classifiers giving probabilistic answers), and misclassification costs.

5 Conclusion

Traditionally, data analysis was the final phase of experimental design that, typ-ically, included a careful selection of patients, their features and the hypothesisto test. With the introduction of data warehouses, such a selective approach todata collection is altered and data may be gathered with no specific purpose inmind. Yet, medical data stored in warehouses may provide a useful resource forpotential discovery of new knowledge.

The process of hypothesis generation and knowledge discovery is supportedby data mining tools, among which the use of machine learning tools turns out tobe advantageous; their use may namely result in logical rules that can be easilyinterpreted by medical experts. The aim of this paper is to present a variety ofdata mining methods and to discuss some of the evaluation criteria appropriatefor supporting medical problem solving.

Acknowledgments

I am grateful to Blaz Zupan and Peter Flach for their contribution to thisoverview; Blaz Zupan provided the information on CBR and neural networks,and Peter Flach contributed to the development of non-standard measures forrule evaluation described in this paper. Thanks to Jan Komorowski and ClaudeSammut for their help in describing rough set and ripple-down rules approaches,respectively.

The work on this paper was financially supported by the Slovenian Ministryof Science and Technology. In part it was also supported by a Joint Project withCentral/Eastern Europe funded by the Royal Society.

Machine Learning for Data Mining in Medicine 59

References

1. Aamodt, A. and Plaza, E.: Case-based reasoning: Foundational issues, methodolog-ical variations, and system approaches, AI Communications, 7(1) 39–59 (1994).

2. Agrawal, R., Mannila, H., Srikant, R., Toivonen, H. and Verkamo, A.I.: Fast dis-covery of association rules. In U.M. Fayyad, G. Piatetski-Shapiro, P. Smyth andR. Uthurusamy (Eds.) Advances in Knowledge Discovery and Data Mining, AAAIPress, 1996, pp. 307–328.

3. Aha, D., Kibler, D. and Albert, M.: Instance-based learning algorithms, MachineLearning, 6: 37–66 (1991).

4. Astion, M.L. and Wielding, P.: The application of backpropagation neural networksto problems in pathology and laboratory medicine, Arch Pathol Lab Med, 116: 995–1001 (1992).

5. Baxt, W.G.: Application of artificial neural networks to clinical medicine, Lancet,364(8983) 1135–1138 (1995).

6. Bradburn, C., Zeleznikow, J. and Adams, A.: Florence: synthesis of case-based andmodel-based reasoning in a nursing care planning system, Computers in Nursing,11(1): 20–24 (1993).

7. Breiman, L., Friedman, J.H., Olshen, R.A. and Stone, C.J.: Classification andRegression Trees. Wadsworth, Belmont, 1984.

8. Brossette, S.E., Sprague, A.P., Hardin, J.M., Waites, K.B., Jones, W.T. and Moser,S.A.: Association rules and data mining in hospital infection control and publichealth surveillance. Journal of the American Medical Inform. Assoc. 5(4): 373–81(1998).

9. Carpenter, G.A. and Tan, A.H.: Rule extraction, fuzzy ARTMAP and medicaldatabases. In: Proc. World Cong. Neural Networks, 1993, pp. 501–506.

10. Caruana, R., Baluja, S., and Mitchell, T.: Using the Future to Sort Out the Present:Rankprop and Multitask Learning for Medical Risk Analysis, Neural InformationProcessing 7 (1995).

11. Cestnik, B.: Estimating Probabilities: A Crucial Task in Machine Learning, In:Proc. European Conf. on Artificial Intelligence, Stockholm, 1990, pp. 147-149.

12. Clark, P. and Boswell, R.: Rule induction with CN2: Some recent improvements. In:Proc. Fifth European Working Session on Learning, Springer, 1991, pp. 151–163.

13. Clark, P. and Niblett, T.: The CN2 induction algorithm. Machine Learning,3(4):261–283 (1989).

14. Compton, P. and Jansen, R.: Knowledge in context: A strategy for expert sys-tem maintenance. In: Proc. 2nd Australian Joint Artificial Intelligence Conference,Springer LNAI 406, 1988, pp. 292–306.

15. Compton, P., Horn, R., Quinlan, R. and Lazarus, L.: Maintaining an expert system.In: Applications of Expert Systems (Quinlan, R., ed.), Addison Wesley, 1989, pp.366–385.

16. Cover, T.M., Hart, P.E.: Nearest neighbor pattern classification, IEEE Transac-tions on Information Theory, 13: 21–27 (1968).

17. Dasarathy, B.V., ed.: Nearest Neighbor (NN) Norms: NN Pattern ClassificationTechniques. IEEE Computer Society Press, Los Alamitos, CA, 1990.

18. Dehaspe, L, Toivonen, H. and King, R.D.: Finding frequent substructures in chem-ical compounds. In: Proc. 4th International Conference on Knowledge Discoveryand Data Mining, (KDD-98) (Agrawal, R., Stolorz, P. and Piatetsky-Shapiro, G.,eds.), AAAI Press, 1998, pp. 30–37.

60 N. Lavrac

19. De Raedt, L. and Dehaspe, L.: Clausal discovery. Machine Learning, 26:99–146(1997).

20. Downs, J., Harrison, R.F., Kennedy, R.L., and Cross, S.C.: Application of the fuzzyARTMAP neural network model to medical pattern classification tasks, ArtificialIntelligence in Medicine, 8(4): 403–428 (1996).

21. Dudani, S.A.: The distance-weighted k-nearest neighbor rule, IEEE Transactionson Systems, Man and Cybernetics, 6(4): 325–327 (1975).

22. Dzeroski, S. and Lavrac, N.: Rule induction and instance-based learning appliedin medical diagnosis, Technology and Health Care, 4(2): 203–221 (1996).

23. Edwards, G., Compton, P., Malor, R., Srinivasan, A. and Lazarus, L.: PEIRS: Apathologist maintained expert system for the interpretation of chemical pathologyreports, Pathology 25: 27–34 (1993).

24. Fausett, L.V.: Fundamentals of neural networks: Architectures, algorithms and ap-plications, Prentice Hall, Upper Saddle River, NJ, 1994.

25. Fix, E., Hodges, J.L.: Discriminatory analysis. Nonparametric discrimination. Con-sistency properties. Technical Report 4, US Air Force School of Aviation Medicine.Randolph Field, TX, 1957.

26. Grzyma la-Busse, J.: Applications of the rule induction systems LERS, In: [56],1998, pp. 366–375.

27. Ham, F.M. and Han, S., Classification of cardiac using fuzzy ARTMAP, IEEETransactions on Biomedical Engineering, 43(4): 425–430 (1996).

28. Horn, K., Compton, P.J., Lazarus, L. and Quinlan, J.R.: An expert system for theinterpretation of thyroid assays in a clinical laboratory, Austr. Comput. Journal17(1): 7–11 (1985).

29. Kahn, C.E., and Anderson, G.M.: Case-based reasoning and imaging procedureselection, Investigative Radiology, 29(6): 643–647 (1994).

30. Kattan, M.W., Ishida, H., Scardino, P.T. and Beck, J.R.: Applying a neural net-work to prostate cancer survival data. In: Intelligent data analysis in medicine andpharmacology (Lavrac, N. Keravnou, E. and Zupan, B., eds.), Kluwer, 1997, pp.295–306.

31. Kohonen, T.: Self-organization and associative memory, Springer-Verlag, NewYork, 1988.

32. Komorowski, J. and Øhrn, A.: Modelling prognostic power of cardiac tests usingrough sets, Artificial Intelligence in Medicine, 1998 (in press).

33. Kononenko, I.: Semi-naive Bayesian classifier. In: Proc. European Working Sessionon Learning-91 (Kodratoff, Y., ed.), Porto, Springer, 1991, pp. 206-219.

34. Kononenko, I.: Inductive and Bayesian learning in medical diagnosis, Applied Ar-tificial Intelligence, 7: 317–337 (1993).

35. Kononenko, I., Bratko, I., and Kukar, M.: Application of machine learning to med-ical diagnosis. In Machine Learning and Data Mining: Methods and Applications,R. S. Michalski, I. Bratko, and M. Kubat (Eds.), John Willey and Sons, 1998, pp.389–408.

36. Lavrac, N., Dzeroski, S., Pirnat, V. and Krizman, V.: The utility of backgroundknowledge in learning medical diagnostic rules, Applied Artificial Intelligence, 7:273–293 (1993).

37. Lavrac, N. and Dzeroski, S.: Inductive Logic Programming: Techniques and Appli-cations. Ellis Horwood, Chichester, 1994.

38. Lavrac, N., Keravnou, E. and Zupan, B., eds.: Intelligent Data Analysis in Medicineand Pharmacology, 1997, Kluwer.

Machine Learning for Data Mining in Medicine 61

39. Lavrac, N.: Selected techniques for data mining in medicine. Artificial Intelli-gence in Medicine, Special Issue on Data Mining Techniques and Applicationsin Medicine, 1999 (in press).

40. Lavrac, N., Kononenko, I., Keravnou, E., Kukar, M. and Zupan, B.: Intelligent dataanalysis for medical diagnosis: Using machine learning and temporal abstraction.AI Communications, 1999 (in press).

41. Lavrac, N., Flach, P.A. and Zupan, B.: Rule evaluation measures: A unifying view,1999 (submitted to Int. Workshop on Inductive Logic Programming, ILP-99).

42. Liestøl, K., Andersen, P.K. and Andersen, U.: Survival analysis and neural nets,Statist. Med., 13: 1189–1200 (1994).

43. Macura, R.T. and Macura, K., eds.: Case-based reasoning: opportunities and ap-plications in health care, Artificial Intelligence in Medicine, 9(1): 1–4 (1997).

44. Macura, R.T. and Macura, K., eds.: Artificial Intelligence in Medicine: SpecialIssue on Case-Based Reasoning, 9(1), 1997.

45. Mariuzzi, G., Mombello, A., Mariuzzi, L., Hamilton, P.W., Weber, J.E., ThompsonD. and Bartels, P.H.: Quantitative study of ductal breast cancer–patient targetedprognosis: an exploration of case base reasoning, Pathology, Research & Practice,193(8): 535–542 (1997).

46. McSherry, D.: Hypothesist: A development environment for intelligent diagnos-tic systems. In: Proc. Sixth Conference on Artificial Intelligence in Medicine(AIME’97), Springer, 1997, pp. 223–234.

47. McSherry, D.: Avoiding premature closure in sequential diagnosis, Artificial Intel-ligence in Medicine, 10(3): 269–283 (1997).

48. Michalski, R.S.: A theory and methodology of inductive learning. In: MachineLearning: An Artificial Intelligence Approach ( Michalski, R., Carbonell, J. andMitchell, T.M., eds.), volume I, Palo Alto, CA, Tioga, 1983, pp. 83–134.

49. Michalski, R., Mozetic, I., Hong, J. and Lavrac, N.: The multi-purpose incremen-tal learning system AQ15 and its testing application on three medical domains.In Proc. Fifth National Conference on Artificial Intelligence, Morgan Kaufmann,1986, pp. 1041–1045.

50. Modai, I., Israel, A., Mendel, S., Hines, E.L. and Weizman, R.: Neural networkbased on adaptive resonance theory as compared to experts in suggesting treatmentfor schizophrenic and unipolar depressed in-patients, Journal of Medical Systems,20(6): 403–412 (1996).

51. Michie, D., Spiegelhalter, D.J. and Taylor, C.C., eds.: Machine learning, neuraland statistical classification, Ellis Horwood, 1994.

52. Muggleton, S.: Inverse entailment and Progol, New Generation Computing, SpecialIssue on Inductive Logic Programming, 13(3–4): 245–286 (1995).

53. Niblett, T. and Bratko, I.: Learning decision rules in noisy domains. In: Researchand Development in Expert Systems III (Bramer, M., ed.), Cambridge UniversityPress, 1986, pp. 24–25.

54. Pawlak, Z.: Information systems – theoretical foundations. Information Systems,6:205–218 (1981).

55. Pawlak, Z.: Rough Sets: Theoretical Aspects of Reasoning about Data, volume 9 ofSeries D: System Theory, Knowledge Engineering and Problem Solving. Kluwer,1991.

56. Polkowski, L. and Skowron, A., eds.: Rough Sets in Knowledge Discovery 1:Methodology and Applications, volume 18 of Studies in Fuzziness and Soft Com-puting. Physica-Verlag, 1998.

57. Quinlan, J.R.: Induction of decision trees. Machine Learning 1(1): 81–106 (1986).

62 N. Lavrac

58. Quinlan, J.R.: Learning logical definitions from relations, Machine Learning 5(3):239–266 (1990).

59. Quinlan, J.R.: C4.5: Programs for Machine Learning, San Mateo, CA, MorganKaufmann, 1993.

60. Rumelhart, D.E. and McClelland, J.L., eds.: Parallel Distributed Processing, Vol.1: Foundations. MIT Press, Cambridge, MA, 1986.

61. Setiono, R.: Extracting rules from pruned networks for breast cancer diagnosis,Artificial Intelligence in Medicine, 8(1): 37–51 (1996).

62. Srinivasan, A., King, R.D., Muggleton, S.H. and Sternberg, M.J.E.: Carcinogen-esis predictions using inductive logic programming. In Intelligent Data Analysisin Medicine and Pharmacology (Lavrac, N. Keravnou, E. and Zupan, B., eds.),Kluwer, 1997, pp. 243–260.

63. Tsumoto, S.: Modelling medical diagnostic rules based on rough sets, In: Proc.First International Conference on Rough Sets and Soft Computing – RSCTC’98(Polkowski, L. and Skowron, A., eds.), volume 1424 of Lecture Notes in ArtificialIntelligence, Springer Verlag. Springer, 1998, pp. 475–482.

64. Weiss, S.M., Kulikowski, C.A.: Computer Systems that Learn. Morgan Kaufmann,San Mateo, CA, 1991.

65. Wettschereck, D.: A study of distance-based machine learning algorithms, PhDThesis, Department of Computer Science, Oregon State University, Corvallis, OR,1994.

66. Wu, C.H.: Artificial neural networks for molecular sequence analysis, Computers& Chemistry, 21(4): 237–56 (1997).

67. Zhu, Y. and Yan, H.: Computerized tumor boundary detection using a Hopfieldneural network, IEEE Transactions on Medical Imaging, 16(1): 55–67 (1997).