G. Paliouras, V. Karkaletsis, and C.D. Spyropoulos (Eds.): ACAI ’ 99, LNAI 2049, pp. 258-266, 2001.© Springer-Verlag Berlin Heidelberg 2001

Pre- and Post-processing in Machine Learning andData Mining

Ivan Bruha

McMaster University, Dept. Computing & Software, Hamilton, Ont., Canada L8S4L7bruha@mcmaster.ca

1 Introduction: Knowledge Discovery and Machine Learning

Knowledge discovery in databases (KDD) has become a very attractive discipline bothfor research and industry within the last few years. Its goal is to extract "pieces" ofknowledge or "patterns" from usually very large databases. It portrays a robust sequenceof procedures or steps that have to be carried out so as to derive reasonable andunderstandable results. One of its components is the process which induces the above"pieces" of knowledge; usually this is a machine learning (ML) algorithm. However, mostof the machine learning algorithms require perfect data in a specific format. The data thatare to be processed by a knowledge acquisition (inductive) algorithm are usually noisyand often inconsistent. Many steps are involved before the actual data analysis starts.Moreover, many ML systems do not easily allow processing of numerical attributes aswell as numerical (continuous) classes. Therefore, certain procedures have to precede theactual data analysis process. Next, a result of an ML algorithm, such as a decision tree,a set of decision rules, or weights and topology of a neural net, may not be appropriatefrom the view of custom or commercial applications. As a result, a concept description(model, knowledge base) produced by an inductive process has to be usuallypostprocessed. Postprocessing procedures usually include various pruning routines, rulequality processing, rule filtering, rule combination, model combination, or evenknowledge integration. All these procedures provide a kind of symbolic filter for noisy,imprecise, or non-user-friendly knowledge derived by an inductive algorithm. Therefore,some preprocessing routines as well as postprocessing ones should fill up the entire chainof data processing. The pre- and post-processing tools always help to investigatedatabases as well as to refine the acquired knowledge. Usually, these tools exploittechniques that are not genuinely symbolic/logical, e.g., statistics, neural nets, and others.

Historically, different names have been used for the process of extracting usefulknowledge from databases, e.g., knowledge extraction, data analysis, informationdiscovery, knowledge discovery in databases, data mining. Especially the last term is usedquite often and its connection to knowledge discovery is discussed at many papers andconferences. Some researchers claim that data mining is a subset of KDD (see e.g. [12]),others declare that it is the opposite way [20]. In this survey we use the firstinterpretation.

Research in knowledge discovery is supposed to develop methods and techniques toprocess large databases in order to acquire knowledge (which is "hidden" in thesedatabases) that is compact, more or less abstract, but understandable, and useful for

Pre- and Post-processing in Machine Learning and Data Mining 259

further applications. Agrawal and Psaila [1] define knowledge discovery as a nontrivialprocess of identifying valid, novel, and ultimately understandable knowledge in data.

In our understanding, knowledge discovery refers to the overall process of determininguseful knowledge from databases, i.e. extracting high-level knowledge from low-leveldata in the context of large databases. Knowledge discovery can be viewed as amultidisciplinary activity because it exploits several research disciplines of artificialintelligence such as machine learning, pattern recognition, expert systems, knowledgeacquisition, as well as mathematical disciplines such as statistics, theory of information,uncertainty processing.

The input to a knowledge discovery process is a database, i.e. a collection (a set) ofobjects. An object (also called case, event, evidence, fact, instance, manifestation, record,observation, statement) is a unit of the given problem. A formal description of an objectis then formed by a suitable collection of so-called elementary descriptions. They can beof either quantitative (numerical) or qualitative (symbolic) character. Their collectionscan be of various forms, too, e.g. numerical vectors, lists of attribute values, strings,graphs, ground facts.

The output of the knowledge discovery process is a collection of "pieces" ofknowledge "dug" from a database. They should exhibit a high-level description in aparticular language. Such a knowledge base (model, concept description) is usuallyrepresented by a set of production (decision) rules, decision trees, collection of prototypes(representative exemplars), etc.

The entire chain of knowledge discovery consists of the following steps:

1. Selecting the problem area. Prior to any processing, we first have to find and specifyan application domain, and to identify the goal of the knowledge discovery processfrom the customer’s viewpoint. Also, we need to choose a suitable representation forthis goal.

2. Collecting the data. Next, we have to choose the object representation and collect dataas formally represented objects. If a domain expert is available, then he/she couldsuggest what fields (attributes, features) are the most informative. If not, then thesimplest method is a ’brute-force’ which indicates that we measure everything availableand only hope that the right (informative, relevant) attributes are among them.

3. Preprocessing of the data. A data set collected by the ’brute-force’ method or byfollowing advices of a domain expert is not directly suitable for induction (knowledgeacquisition); it comprises in most cases noise and missing values, the data are notconsistent, the data set is too large, there are too many attributes that describe eachobject of the database and so on. Therefore, we need to minimize the noise in the data,and choose a strategy for handling missing (unknown) attribute values. Also, weshould use any suitable method for selecting and ordering attributes (features)according to their informativity. In this step, we also discretize and/or fuzzifynumerical (continuous) attributes, and eventually, process continuous classes.

4. Data mining: Extracting pieces of knowledge. Now, we reach the stage of selecting aparadigm for extracting pieces of knowledge (e.g., statistical methods, neural netapproach, symbolic/logical learning, genetic algorithms). First, we have to realize thatthere is no optimal algorithm which would be able to process correctly any database.

260 I. Bruha

Second, we are to follow the criteria of the end-user; e.g., he/she might be moreinterested in understanding the model extracted rather than its predictive capabilities. Afterwards, we apply the selected algorithm and derive (extract) new knowledge.

5. Postprocessing of the knowledge derived. The pieces of knowledge extracted in theprevious step could be further processed. One option is to simplify the extractedknowledge. Also, we can evaluate the extracted knowledge, visualize it, or merelydocument it for the end user. They are various techniques to do that. Next, we mayinterpret the knowledge and incorporate it into an existing system, and check forpotential conflicts with previously induced knowledge.

It is worth mentioning here that we may return from any step of the knowledgediscovery process to any previous step and change our decisions. Thus, this processinvolves significant iteration and represents a time-consuming mechanism with manyloops. Most previous work has been done in step 4. However, the other steps are alsoimportant for the successful application of knowledge discovery in practice.

2 Description of Workshop Papers

This section presents an overview of the papers presented during the workshop on "Pre-and Post-Processing in Machine Learning and Data Mining" organized as part of theAdvanced Course on Artificial Intelligence (ACAI ’99) [23].

The survey paper by Bruha [5] focuses on collecting and preprocessing the data, andpostprocessing the extracted knowledge. It describes in detail all the essential steps foreach of the three processes. The paper was motivated by several other papers, amongthem [1], [3], [12]. Here we will only introduce in brief the steps of collecting andpreprocessing the data, and postprocessing extracted knowledge.

Collecting and Preprocessing of Data

1. Choosing the object representation. The input to a knowledge discovery process isa database, i.e. a set of objects. An object as a unit of the given problem must beformally described by a collection of elementary descriptions. Therefore, we haveto choose the appropriate object representation. The most common choice is theattribute representation of objects. Elementary properties, called attributes, areselected on actual objects. An object is thus represented by a list of attributes andtheir values. Each attribute has its domain of possible values. Depending on theorganization of the attribute's domain, we distinguish three basic types of attributes:symbolic (discrete, nominal, categorial), continuous (numerical), and structured.

2. Mapping and collecting data. After selecting a proper representation, we choose theattributes to be measured on objects (either following a suggestion of a domainexpert or using the 'brute-force' method). Also, we have to determine the attributenames and the names of their values. The data collected are thus mapped into asingle naming convention and uniformly represented.

Pre- and Post-processing in Machine Learning and Data Mining 261

3. Scaling large datasets. Practically all learning algorithms assume that data are in themain memory and pay no attention to how the algorithm could deal with extremelylarge databases when only a limited number of data can be viewed. There are severalpossibilities of solving this problem, e.g. windowing, batch-incremental mode.

4. Handling noise and errors. There are generally two sources of error. External errorsare introduced from the world outside the system itself (random errors and noise, animperfect teacher, etc.). Internal errors are caused by poor properties of the learning(data mining) system itself, e.g. poor search heuristics or preference criteria.

5. Processing unknown attribute values. When processing real-world data, oneimportant aspect in particular is the processing of unknown (missing) attribute values.This topic has been discussed and analyzed by several researchers in the field ofmachine learning; see e.g. [18], [8], [6]. There are a few important factors to betaken into account when processing unknown attribute values. One of the mostimportant ones is the source of ’unknownness’: (i) a value is missing because it wasforgotten or lost; (ii) a certain attribute is not applicable for a given object, e.g., itdoes not exist for a given object; (iii) an attribute value is irrelevant in a givencontext; (iv) for a given observation, the designer of a training database does not careabout the value of a certain attribute (so-called dont-care value).

6. Discretization/fuzzification of numerical attributes. The symbolic, logical learningalgorithms are able to process symbolic, categorial data only. However, real-worldproblems involve both symbolic and numerical attributes. Therefore, there is animportant issue to discretize numerical (continuous) attributes. It could be performedeither off-line (preprocessor) or on-line (dynamic discretization). A natural extensionof this procedure is fuzzification of numerical attributes.

7. Processing of continuous classes. Similar problem to the above issue is processingof continuous classes. Most inductive symbolic algorithms require discretized(symbolic) classes. In some applications, however, we are faced by continuous(numerical) classes. Similarly to the problem above, there are two approaches: eitheroff-line or on-line splitting.

8. Grouping of values of symbolic attributes. It is a known problem that attributes withtoo many values are overestimated in the process of selecting the most informativeattributes, both for inducing decision trees and for deriving decision rules. Toovercome this overestimation problem, the values of multivalued attributes can begrouped into two or more subsets.

9. Attribute selection and ordering. We cannot be definitely sure that all attributes areinformative for the given target. In real-world data, the representation of data oftenuses too many attributes, but only a few of them may be related to the target concept.Attribute selection and ordering procedures help to solve the above problem. Theyorder the set of input attributes according to their informativity, and then select arelatively small subset of the most informative attributes.

10. Attribute construction and transformation. As we stated earlier, finding suitableattributes for problem representation could be a difficult and time-consuming task.If the attributes are inappropriate for the target concept, data mining (learning) couldbe difficult or impossible. To overcome this problem a system needs to be capable

262 I. Bruha

of generating (constructing) new appropriate attributes. There are two differentmethods for doing that: attribute construction and attribute transformation.

11. Consistency checking. As for inconsistency in a database, it may not be eliminatedby the previous steps of preprocessing. There are two general approaches to handleinconsistency in the data. The first one is done ’off-line’, i.e. by a preprocessor orwithin the data mining process itself. Another possibility is to utilize the loop facilityof the knowledge discovery process, i.e. to return to one of the previous steps andperform it again for different parameters.

Postprocessing1. Knowledge filtering: Rule truncation and postpruning. If the training data is noisy then

the learning algorithm generates leaves of a decision tree or decision rules that covera very small number of training objects. This happens because the learning algorithmtries to split subsets of training objects to even smaller subsets that would be genuinelyconsistent. To overcome this problem a tree or a decision rule set must be shrunk, byeither postpruning (decision trees) or truncation (decision rules).

2. Interpretation and explanation. Now, we may use the acquired knowledge directly forprediction or in an expert system shell as a knowledge base. If the knowledgediscovery process is performed for an end-user, we usually document the derivedresults. Another possibility is to visualize the knowledge, or to transform it to anunderstandable form for the user-end. Also, we may check the new knowledge forpotential conflicts with previously induced knowledge. In this step, we can alsosummarize the rules and combine them with a domain-specific knowledge providedfor the given task.

3. Evaluation. After a learning system induces concept hypotheses (models) from thetraining set, their evaluation (or testing) should take place. There are several widelyused criteria for this purpose: classification accuracy, comprehensibility, computationalcomplexity, and so on.

4. Knowledge integration. The traditional decision-making systems have been dependanton a single technique, strategy, model. New sophisticated decision-supporting systemscombine or refine results obtained from several models, produced usually by differentmethods. This process increases accuracy and successfulness.

The paper by Brijs et al. [4] deals with large sets of decision rules discovered duringthe mining process. Control of a large set of decision rules is extremely costly anddifficult. Moreover, a set of rules is usually incomplete, i.e., some objects (instances) arenot covered by any rule, and the rules are mutually inclusive, i.e., some objects may becovered by more than one rule. Therefore, the authors propose a postprocessing,knowledge-filtering procedure to solve the problem of selecting the most promisingsubset of decision (characteristic) rules. Their method also influences the pruning processin a sense that some overall measure of quality of the reduced decision set can beproduced. In fact, it allows controlling a user-defined level of overall quality of the modelin combination with a maximum reduction of the redundancy in the original decision set.

The paper discusses two integer-programming (IP) techniques developed by theauthors, namely the “Maximum redundancy reduction” technique and the “Incorporating

Pre- and Post-processing in Machine Learning and Data Mining 263

noise and discriminant power” one. The first technique searches for an optimal selectionof rules that is able to maximally reduce redundancy under the constraint of covering allpositive objects (instances) that are covered by the original set of rules. In the lattertechnique, the first technique has been adapted to account for noise in the data and toimpose a quality criterion on the final set of decision rules. Both techniques have beenempirically tested on real-world data and compared with the well-known RuleCoverheuristic [22] as a method for redundancy reduction. It has been found that the integer-programming techniques are able to produce significantly better results than theRuleCover heuristic; in terms of the number of rules that remain in the final set of rules,in terms of the discriminant power of the final set of rules, and also in terms of the totalredundancy.

The paper by Kalles and Papagelis [14] introduces a project on discretization ofnumerical attributes (as a preprocessor) for decision-tree induction. The authors preferdescriptions that make the grouping of objects easier rather than those that allowfuzzification. This bias can be captured by the observation that entropy gain is maximizedat the class boundaries for numerical attributes [11]. The method splits the range of eachnumerical attribute to a small number of non-overlapping intervals; each interval isassigned to a bucket. Then the buckets are merged utilizing the 2 metric, which leads toa simple prepruning technique. The experiments on various datasets have revealed thatthe greater the confidence level that is required, the better the classification occurrenceis achieved. The only drawback is the size of a decision tree constructed using thismethod for handling numerical attributes. On the other hand, an interesting aspect of themethod is that it naturally paves the way for deploying more effective postprocessingschemes. The authors investigated the postprocessing technique for character recognitionsetting.

The paper by Kontkanen et al. [15] utilizes Bayesian networks for visualization as animportant step in postprocessing for visual inspection of the rationality of the learnedmodels (classifiers, knowledge bases). The authors have designed a model-basedvisualization scheme which utilizes a Bayesian network representing the problem domainprobability distribution, and producing two- or three-dimensional (2D or 3D) images ofthe problem domain. To display high-dimensional data on a 2D or 3D device, each datavector has to be provided with the corresponding 2D or 3D coordinates that determine itsvisual location. The traditional statistical data analysis provides so-calledmultidimensional scaling [10]. The authors of the paper have developed a model-basedvisualization scheme with the motto ’Two vectors are considered similar if they lead tosimilar predictions’. This idea follows from a Bayesian distance metric [16].

The authors also discuss methods for validating the goodness of differentvisualizations, and suggest a simple scheme based on estimating the prediction accuracythat can be obtained by using reduced, low-dimensional data. Experiments have beenperformed on UCI benchmark databases. Examples of the 2D visualizations for somedatabases demonstrate that the proposed scheme produces visualizations that are muchbetter (with respect to the validation scheme suggested) than those obtained by classicalEuclidean multidimensional scaling.

264 I. Bruha

The paper by Kralik and Bruha [17] introduces an off-line discretization procedure (asa preprocessor) for genetic learning algorithms. The genetic algorithms are usuallydeveloped in such a way that they are capable of processing symbolic data only, notcontinuous ones. There exist quite a few discretization procedures in the machine learningfield. The paper introduces a version of a new algorithm for discretization of continuousattributes [2], which has been applied to a learning system based on genetic algorithms.

The induction stage of a learning process consists in searching usually a huge spaceof possible concept descriptions. There exist several paradigms on how to control thissearch. One of the promising and efficient paradigms is genetic algorithms. Severalapproaches have been proposed concerning the application of genetic algorithms tolearning. The paper describes an efficient composite of the discretization preprocessorand a genetic algorithm in an attribute-based rule-inducing ML algorithm. Actually, adomain-independent genetic algorithm has been incorporated into the covering machinelearning algorithm CN4 [7], a large extension of the well-known algorithm CN2. Theinduction procedure of CN4 (beam search methodology) has been replaced by the geneticalgorithm.

The paper by Smid et al. [19] deals with the attribute selection problem. Modelling aninput response system is an immense problem of great complexity. A good modeltypically requires insight from an engineer or a scientist. With the World Wide Web(WWW), engineers and scientists can use data mining techniques to understand betterpatterns and relationships among data. The authors discuss the selection of significant(informative) attributes. The set of input attributes sometimes is not sufficient and newattributes have to be derived (constructed). The project designed by the authors employsvisual analysis [9] for discovering new attributes (in addition to the set of the inputattributes). The set of new attributes also includes operators such as standard deviation,difference, delay. The authors utilize WebGraph, a data visualization and analysis tool.WebGraph provides algorithms and functions that allow users to preview patterns inselected data sets. The users need to use only a standard web browser. Two case studiesare presented to demonstrate the use of the system for building an efficient model.

The paper by Talavera [21] presents an unsupervised feature (attribute) selectionmethod aimed to reduce the dimensionality of a symbolic database prior to a clusteringprocess. The author’s method is based on the assumption that features which are notcorrelated with other features in the database are likely to be irrelevant. The author hasapplied a feature dependency measure proposed by [13]. Feature selection is viewed asa heuristic search in a feature space in which several characteristics such as an evaluationfunction, stopping criterion, or a generation procedure has to be determined. The author’smethod represents a greedy search that only performs one step at a time in the featurespace to select the final feature set. In his experiments the author has utilized COBWEB,the well-known symbolic probabilistic clustering algorithm. The method has beenevaluated on UCI databases. Despite a close relationship of the dependency measure usedto rank the importance of features and the clustering system employed in the experiments,the method developed can be applied to any symbolic clustering algorithm. Theexperiments have demonstrated evidence that the dependency measure can identifycompletely irrelevant features. Thus, removal of these features should improve theperformance of various clustering algorithms.

Pre- and Post-processing in Machine Learning and Data Mining 265

3 Conclusion

The discussion that followed the presentation of the papers first focused on thespecification of the fields and methodologies of KDD. The conclusion of the discussionwas that there are two main streams in KDD, namely: methodology and applications. Inboth streams, the following issues arise:

- selection of a problem and its representation;- selection of a description language;- data collection;- data preprocessing (discretization, grouping, attribute selection, construction of new

attributes, etc.);- knowledge extracting;- postprocessing the knowledge base (rule filtering, rule quality, etc.);- evaluation, interpretation, visualization, combination, etc. of the derived knowledge.

The results of the fruitful discussion could be summarized as follows:

� There is work that combines the above mentioned streams in KDD, i.e. methodologyand real-world applications. However, there is a ’gap’ of terminology between thesetwo streams. Methodologists use traditional terms common in KDD and ML. On theother hand, the specialists who apply KDD and ML methods to real problem useterminology common for the given discipline (medicine, financing, insurance, carmanufacturing, etc.) which, however, is not understandable to KDD specialists.

� Preprocessing is a favourite topic in KDD, in contrast to postprocessing. There is muchmore work on preprocessing than on postprocessing. There are many algorithms for’attribute mining’, but only a few for postprocessing. Nevertheless, the visualinterpretation and evaluation is now getting a great spirit in the field of KDD.

� Knowledge integration (i.e., model combination, merging, and modification) shouldnot concentrate solely on boosting and bagging.

� Last but not least, a question arose about the difference between machine learning(ML) and data mining (DM). The general consensus was that ML and DM overlap andone is to observe ML as a scientific discipline, whereas DM as an engineering one.

References

1. R. Agrawal and G. Psaila: Active data mining. 1st International Conference on KnowledgeDiscovery and Data Mining, Menlo Park, Calif. (1995), 3-8

2. P. Berka, I. Bruha: Empirical comparison of various discretization procedures. InternationalJournal of Pattern Recognition and Artificial Intelligence, 12, 7 (1998), 1017-1032

3. P. Brazdil and P. Clark: Learning from imperfect data. In: P. Brazdil, K. Konolige (eds.):Machine Learning, Meta-Reasoning, and Logics, Kluwer (1990), 207-232

4. T. Brijs, K. Vanhoof, G. Wets: Reducing redundancy in characteristic rule discovery by usingIP-techniques. In [23]

266 I. Bruha

5. Bruha: From machine learning to knowledge discovery: Preprocessing and postprocessing.In [23]

6. Bruha and F. Franek: Comparison of various routines for unknown attribute value processing:covering paradigm. International Journal of Pattern Recognition and Artificial Intelligence,10, 8 (1996), 939-955

7. Bruha, S. Kockova: A support for decision making: cost-sensitive learning system, ArtificialIntelligence in Medicine, 6 (1994), 67-82

8. B. Cestnik, I. Kononenko, I. Bratko: ASSISTANT 86: a knowledge-elicitation tool for sophist-icated users. In: I. Bratko and N. Lavrac (eds.): Progress in Machine Learning: EWSL’87,Sigma Press (1987)

9. K. Cox, S. Eick, G. Wills: Visual data mining: recognizing telephone calling fraud. DataMining and Knowledge Discovery, 1, 2 (1997)

10. R. Duda, P. Hart: Pattern classification and scene analysis. John Wiley (1973)11. U. Fayyad, K.B. Irani: On the handling of continuous-valued attributes on decision tree

generation. Machine Learning, 8 (1992), 87-10212. U. Fayyad, G. Piatetsky-Shapiro, P. Smyth: From data mining to knowledge discovery in

databases. Artificial Intelligence Magazine (1996), 37-5313. D.H. Fisher: Knowledge acquisition via incremental conceptual clustering. Machine Learning,

2 (1987), 139-17214. D. Kalles, A. Papagelis: Induction of decision trees in numeric domains using set-valued

attributes. In [23]15. P. Kontkanen, J. Lahtinen, P. Myllymaki, T. Silander, H. Tirri: Using Bayesian networks for

visualizing high-dimensional data. In [23]16. P. Kontkanen et al.: On Bayesian case matching. In: B. Smyth, P. Cunningham (eds.):

Advances in Case-based Reasoning, EWCBR-98, Springer-Verlag (1998), 13-2417. P. Kralik, I. Bruha: Discretizing numerical attributes in a genetic attribute-based learning

algorithm. In [23]18. J.R. Quinlan: Unknown attribute values in ID3. International Conference ML (1989), 164-16819. J. Smid, P. Svacek, P. Volf, E. Levine, L. Kurz, I. Glucksmann, M. Smid: Web based data

analysis tools. In [23]20. P. Stolorz et al.: Fast spatio-temporal data mining of large geophysical datasets. 1st

International Conference on Knowledge Discovery and Data Mining, Menlo Park, Calif.(1995), 300-305

21. L. Talavera: Dependency-based dimensionality reduction for clustering symbolic data. In [23]22. H. Toivonen et al.: Pruning and grouping of discovered association rules. ECML-99,

Workshop Statistics, Machine Learning, and Discovery in Databases, Heraclion, Greece(1995)

23. Proceedings of the Workshop on Pre- and Post-Processing in Machine Learning and DataMining, Advanced Course on Artificial Intelligence (ACAI ’99), Chania, Greece, 1999(http://www.iit.demokritos.gr/skel/eetn/acai99/Workshops.htm).