ensembles of α-trees for imbalanced classification problems

JOURNAL OF LATEX CLASS FILES, VOL. 6, NO. 1, JANUARY 2007 1

Ensembles of α-Trees for ImbalancedClassification Problems

Yubin Park, Student Member, IEEE, and Joydeep Ghosh, Fellow, IEEE

Abstract—This paper introduces two kinds of decision tree ensembles for imbalanced classification problems, extensivelyutilizing properties of α-divergence. First, a novel splitting criterion based on α-divergence is shown to generalize several well-known splitting criteria such as those used in C4.5 and CART. When the α-divergence splitting criterion is applied to imbalanceddata, one can obtain decision trees that tend to be less correlated (α-diversification) by varying the value of α. This increaseddiversity in an ensemble of such trees improves AUROC values across a range of minority class priors. The second ensembleuses the same alpha trees as base classifiers, but uses a lift-aware stopping criterion during tree growth. The resultant ensembleproduces a set of interpretable rules that provide higher lift values for a given coverage, a property that is much desirablein applications such as direct marketing. Experimental results across many class-imbalanced datasets, including BRFSS, andMIMIC datasets from the medical community and several sets from UCI and KEEL, are provided to highlight the effectiveness ofthe proposed ensembles over a wide range of data distributions and of class imbalance.

Index Terms—Data mining, Decision trees, Imbalanced datasets, lift, ensemble classification.

F

1 INTRODUCTION

IMBALANCED class distributions are frequently en-countered in real-world classification problems,

arising from fraud detection, risk management, textclassification, medical diagnosis, and many other do-mains. Such imbalanced class datasets differ frombalanced class datasets not only in the skewness ofclass distributions, but also in the increased impor-tance of the minority class. Despite their frequentoccurrence and huge impact in day-to-day applica-tions, the imbalance issue is not properly addressedby many standard machine-learning algorithms, sincethey assume either balanced class distributions orequal misclassification costs [1].

Various approaches have been proposed to dealwith imbalanced classes in order to cope with theseissues, including over/undersampling [2], [3], SMOTE(Synthetic Minority Oversampling TEchnique), cost-sensitive [4], modified kernel-based, and active learn-ing methods [5], [6]. Although these methods dosomewhat alleviate the problem, they are often basedon heuristics rather than on clear guidance. Forinstance, in the oversampling technique, the opti-mal oversampling ratio varies largely across datasets,and is usually determined through multiple cross-validations or other heuristics.

Recent findings suggest that the class imbalanceproblem should be approached from multiple anglesas well as using different evaluation metrics [7], [8],

• Y. Park and J. Ghosh are with the Department of Electrical andComputer Engineering, The University of Texas at Austin, Austin,TX, 78712.E-mail: [email protected] and [email protected]

[9], [10]. These findings show that the degree ofimbalance is not the only factor hindering the learningprocess. Rather, the difficulties reside with variousother factors such as overlapping classes, lack ofrepresentative data, and small disjuncts. Even worse,the effect of such factors becomes amplified when thedistribution of classes is highly imbalanced [1].

Chawla et al. [11] show that classical evaluationmetrics such as misclassification rates and accuracyturn out to be inappropriate measures for the classimbalance problem, as such metrics are easily drivenby the majority class. The authors alternatively sug-gest that an Area Under the Receiver Operating Char-acteristic curve (AUROC) and a Lift chart tend toconvey more meaningful performance metrics for theclass imbalance problem, and these evaluation metricsare becoming widely adopted and tested in recentyears. Therefore, to cater a general framework forthe problem, we aim to simultaneously handle bothmultiple objectives and evaluation metrics.

In this paper, we seek to achieve both an inter-pretable classification output and outstanding per-formance in (1) AUROC and (2) Lift chart regimes,through a novel “α-diversification” technique. First,we propose a novel splitting criterion parameter-ized by a scalar α. When applied to imbalanceddata, different values of α induce different splittingvariables in a decision tree (α-diversification), re-sulting in diverse base classifiers in the ensemble.Our α-diversification technique can be easily com-bined with other data preprocessing technique such asunder/over-baggings. In Ensemble of α-Trees (EAT)framework, we show that these new approaches resultin ensembles that are reasonably simple, yet accurate,over a range of class imbalances. Second, we modify


and improve (i) traditional stopping rules of decisiontrees, and adopt (ii) rule ensemble techniques [12],in order to directly optimize the lift objective, andprovide a wider coverage for a fixed lift value. Al-though lift is typically not considered in the machinelearning community, it is a prevalent evaluation met-ric in commercial applications involving class imbal-ance problems, for example in direct marketing [13].When combined with our α-diversification technique,Lift-boosting Ensemble of α-Trees (LEAT) frameworkprovides nearly the optimal achievable lift-coveragepairs for imbalanced data.

The rest of the paper is organized as follows:We begin by reviewing previous approaches deal-ing with imbalanced classification problems, high-lighting those algorithms applicable to decision treeensembles. In Section 3, we introduce the definitionof α-divergence and its basic properties, and thenwe propose a new decision tree algorithm using α-divergence, and analyze its properties on imbalanceddata, “α-diversification” in Section 4. Using this α-diversification technique, we design our first ensem-ble framework using the traditional additive tech-nique (EAT), and the second ensemble framework us-ing logical disjunction technique (LEAT) in Section 5and 6. In Section 7, we validate our findings using20 imbalanced datasets from UCI, KEEL, BRFSS, andMIMIC datasets, comparing our ensembles with thestate of the art ensemble techniques for imbalancedclassification problems. Finally, we summarize ourcontribution, and discuss the possible constraints ofour framework and future work in Section 8.

2 RELATED WORK

Imbalanced classification problems have become oneof the challenges in data mining community, and havebeen widely studied in recent years, due to their com-plexities and huge impacts in real-world applications[1]. Although it is almost impossible to list all the ex-plosive number of research dealing with class imbal-ance so far, Galar et al. has showed that data miningalgorithms for imbalanced data can be categorizedinto three groups: (i) data-level, (ii) algorithm-level,and (iii) cost-sensitive methods [14]. In this section, westart by addressing these three techniques, specificallyfocusing on the techniques applicable to decision treealgorithms. Then, we illustrate ensemble methods thatspecialize in learning imbalanced datasets.

Data-level: Over-sampling, under-sampling ap-proaches, and SMOTE, a variant form of over-sampling methods, are among the popular ap-proaches when tackling the imbalanced learning prob-lem at the data level [15], [16], [17]. In random over-sampling, the minority class data points are randomlyreplicated using the existing minority class members.SMOTE [18] is an alternative oversampling methodthat attempts to add information to the training set by

creating synthetic minority data points, whereas ran-dom sampling approaches do not add additional in-formation to the training data. On the other hand, ran-dom undersampling decreases the number of majorityclass data points by randomly eliminating majorityclass data points currently in the training set [19].Although this technique removes potentially usefulinformation from the training set, it is recently shownthat decision trees perform better with undersamplingthan with oversampling [20], when applied to class-imbalance problems.

Accordingly, these three sampling approaches cre-ate an artificial dataset that has a different class dis-tribution from the original dataset. In theory, as thenumber of the minority class examples is relativelyamplified, traditional machine learning algorithmsand evaluation metrics can be applied without fur-ther modification. However, future data points or testpoints are typically sampled from the original distri-bution, which causes discrepancy between learningand testing unless the algorithms are adjusted for theoriginal distribution [21, Chapter 1.7].

Algorithm-level: When decision trees are used asthe base classifiers, the splitting criterion used largelydetermines the overall characteristics and perfor-mance with respect to data imbalance. Researcherssuggested many types of splitting criteria; for ex-ample, Breiman [22] investigated and summarizedvarious splitting criteria - Gini impurity, Shannonentropy and twoing in detail. Later, Dietterich etal. [23] showed that the performance of a tree canbe influenced by its splitting criteria and proposeda criterion named after its authors DKM (Dietterich,Kearns, and Mansour), which results in lower errorbounds based on the Weak Hypothesis Assumption.The DKM splitting criterion has its root in com-putational learning theory, and is a valuable by-product while understanding C4.5 in the AdaBoostframework [24]. Drummond et al. [25] and Flach [26]subsequently found that the DKM splitting criterionis actually class imbalance-resistant, and the finding issupported by many imbalanced dataset experiments.

Although not directly related to imbalanced data,as a similar approach to our work, Karakos et al. [27]proposed Jensen-Renyi divergence parameterized bya scalar α as a splitting criterion in document clas-sification tasks. Jensen-Renyi divergence is anothergeneralization of Shannon Entropy, but has a differentstructure from the α-divergence introduced in thispaper. Moreover, their algorithm is based on a singleclassifier, and the determination of the “best” α wasbased on heuristics. We will discuss the DKM splittingcriterion in Section 3.1, in order to show that it is aspecial cases of our generalized splitting criterion.

Cost-sensitive learning: Imbalanced learning prob-lems can alternatively be approached by adopt-ing cost-sensitive learning methods [28], [29]. Cost-sensitive methods put more weights on the minority


class examples than the majority class examples dur-ing training and validation phases. These methods areeffective when the data has explicit costs associatedwith its classes; however, the cost definition is usuallytask-specific.

In regards to decision trees, cost-sensitive methodshave been applied to either (i) decision thresholds or(ii) cost-sensitive pruning schemes. In the first cost-sensitive scheme, a decision threshold is adjusted toreduce the expected cost given the predefined costdefinition [30], [31]. However, this threshold adjust-ment provides the optimal solution only in the costevaluation regime, as the resultant AUROC remainsthe same. In the latter case, a pruning formula indecision trees incorporates the predefined costs ratherthan classification error rates; nevertheless, in thepresence of imbalanced data, pruning procedures tendto remove leaves describing the minority concept [7].Even though cost-sensitive methods have shown em-pirically improved results against unmodified algo-rithms, their performance depends on heuristics thatneed to be tuned to the degree of imbalance andcharacteristics of data.

Ensemble methods generally outperform singleclassifiers in class imbalance problems [32], and de-cision trees are popular choices for the base classifiersin an ensemble [33]. In recent years, the RandomForest has been modified to incorporate (i) samplingtechniques (Balanced Random Forest) and (ii) costmatrices (Weighted Random Forest) to handle class-imbalance [34]. Both Random Forests methods showsuperior performance over other imbalance-resistantclassifiers for many imbalanced datasets. The per-formance of Balanced Random Forest and WeightedRandom Forest are comparable to each other, butno clear guidance is suggested regarding when tochoose which method. Furthermore, their samplingratios and cost matrices usually depend on heuristics,which increases their learning time and computationalburden.

According to Galar et al. [14], classifier ensemblesfor imbalanced data can be categorized into threeforms: (i) bagging-, (ii) boosting-, and (iii) hybrid-based approaches. In their study, the authors thor-oughly compared the state of the art ensemble al-gorithms from these three categories using 44 imbal-anced datasets, and concluded that these ensembletechniques are actually effective than a single clas-sifier in imbalanced learning problems. It is notablethat under-bagging techniques are shown to be verypowerful when dealing with class imbalance thanother sophisticated ensemble algorithms, in spite oftheir simple implementation. As stated earlier, ourα-diversification technique can be easily combinedwith other data preprocessing techniques, and thusin this paper, we also provide an example of α-diversified under-bagging ensembles as well as nor-mal α-diversified ensembles.

3 PRELIMINARIES

In this section, we start by illustrating three existingsplitting criteria in decision trees, then present thedefinition of α-divergence. α-divergence generalizesthe Information Gain criterion of C4.5, and we showthat it can be related to other splitting criteria suchas Gini and DKM [35] splitting criteria, which will belater highlighted in Section 4.4.

3.1 Splitting Criteria in Decision TreesDecision tree algorithms generate tree-structured clas-sification rules, which are written in a form of con-junctions and disjunctions of feature values (or at-tribute values). These classification rules are con-structed through (i) selecting the best splitting featurebased on a certain criterion, (ii) partitioning input datadepending on the best splitting feature values, then(iii) recursively repeating this process until certainstopping criteria are met. The selected best splittingfeature affects not only the current partition of inputdata, but also the subsequent best splitting featuresas it changes the sample distribution of the resultingpartition. Thus, the best splitting feature selection isarguably the most significant step in decision treebuilding, and different names are given for decisiontrees that use different splitting criteria, e.g. C4.5 andID3 for Shannon entropy based splitting criteria suchas Information Gain and Information Gain Ratio [36],and CART for the Gini impurity measure.

Different splitting criteria use their own impuritymeasures, which are used to calculate “achievable”impurity reduction after a possible split. Consider anominal feature X and target class Y . For example, inC4.5 and ID3, impurity of a data partition (or a nodein a tree) with a feature X = x is measured usingShannon entropy:

H(Y |x) = −∑y

p(y|x) log p(y|x),

where p(y|x) denotes a conditional distribution ofY given X . If we choose to use the Gini impuritymeasure:

Gini(Y |x) =∑y

p(y|x)(1− p(y|x)),

then we obtain the CART splitting attribute. If theDKM impurity measure is used:

DKM(Y |x) =∏y

√p(y|x),

then the DKM tree can be obtained. After calculatingimpurity scores for each possible splitting feature,the feature with the maximum impurity reductionis chosen as a splitting feature. In general, differentimpurity measures result in different decision trees.However, no single splitting criterion is guaranteed tooutperform over the others in decision tree building


because of its greedy construction process [37, p. 161].Furthermore, even for a single split, the best perform-ing splitting criteria depends on problem characteris-tics.

3.2 α-Divergence

The Information Gain (or impurity reduction) crite-rion in C4.5 and ID3 decision trees can be interpretedfrom a different angle: a weighted Kullback-Leibler(KL) divergence criterion between joint and marginaldistributions. Information Gain (IG), in C4.5 and ID3,is defined as:

IG = H(Y )−∑x

p(x)H(Y |x) (1)

where p(x) is a marginal distribution of a feature X .Equation (1) can be expressed using KL-divergence:

IG = H(Y )−H(Y |X) = I(X;Y )

=∑x

∑y

p(x, y) logp(x, y)

p(x)p(y)

= KL(p(x, y)‖p(x)p(y))

where I(X;Y ) denotes a mutual information andp(x, y) is a joint distribution between X and Y .

In fact, KL-divergence is a special case of the α-family of divergences [38] defined by:

Dα(p‖q) =

∫xαp(x) + (1− α)q(x)− p(x)αq(x)1−αdx

α(1− α)(2)

where p, q are any two probability distributions and αis a real number. If both p(x) and q(x) are well definedprobability density functions i.e.

∫xp(x)dx = 1 and∫

xq(x)dx = 1, then Equation (2) can be abbreviated

as:

Dα(p‖q) =1−

∫xp(x)αq(x)1−αdx

α(1− α). (3)

Some special cases are:

D−1(p‖q) =1

2

∫x

(q(x)− p(x))2

p(x)dx (4)

limα→0

Dα(p‖q) = KL(q‖p) (5)

D 12(p‖q) = 2

∫x

(√p(x)−

√q(x))2dx (6)

limα→1

Dα(p‖q) = KL(p‖q) (7)

D2(p‖q) =1

2

∫x

(p(x)− q(x))2

q(x)dx (8)

Equation (6) is Hellinger distance, and Equation (5)and (7) are KL-divergences. α-Divergence is alwayspositive and is zero if and only if p = q. This enablesα-divergence to be used as a (dis)similarity measurebetween two distributions.

The splitting criterion function of C4.5 can be writ-ten using α-divergence as:

I(X;Y ) = limα→1

Dα(p(x, y)‖p(x)p(y))

= KL(p(x, y)‖p(x)p(y)).

Maintaining consistency with the C4.5 algorithm, wepropose a generalized α-divergence splitting formula.

Definition 3.1 (α-Divergence splitting formula). α-Divergence splitting formula is Dα(p(x, y)‖p(x)p(y)).

Note that α = 1 gives Information Gain in C4.5.Using this splitting criterion, a splitting feature isselected, which gives the maximum α-divergencebetween p(x, y) and p(x)p(y). α-Divergence splittingformulae with different α values may yield differentsplitting features. This aspect will be further illus-trated in Section 4.2.

4 α-TREE

In this section, we propose a novel decision treealgorithm, using the proposed decision criterion, andexamine its properties.

4.1 α-Tree algorithm

The decision tree induction presented in algorithm 1results in an α-Tree. Consider a dataset with features{X ′i}

ji=1 and an imbalanced binary target class Y . To

keep the notation simple, we transform the features{X ′i}

ji=1 into binary features {Xi}ni=1, i.e. {Xi}ni=1 =

Binarize({X ′i}ji=1). The inputs to the α-Tree algorithm

are the binarized features {Xi}ni=1, the correspondingclass labels Y , and a pre-specified α. For differentvalues of α, the algorithm can result in different rulesets. The effect of varying α will be discussed inSection 4.2.

Algorithm 1 Grow a single α-Tree

Input: training data (features X1, X2, ..., Xn and la-bel Y ), αSelect the best feature X∗, which gives the maxi-mum α-divergence criterionif (number of data points < cut off size) or(best feature == None) then

Stop growingelse

partition the training data into two subsets, basedon the value of X∗

left child ← Grow a single α-Tree (data with(X∗ = 1), α)right child ← Grow a single α-Tree (data with(X∗ = 0), α)

end if


4.2 Properties of α-DivergenceIf both p(x, y) and p(x)p(y) are properly defined prob-ability distributions, then the α divergence criterionformula becomes:

Dα(p(x, y)‖p(x)p(y)) = EX [Dα(p(y|x)‖p(y))]. (9)

Consider two Bernoulli distributions, p(x) and q(x)having the probability of success θp, θq respectively,where 0 < θp, θq < 1/2. If we view Dα(p(x)‖q(x)) asf(θp), then:

f (1)(θp) =−1

(1− α)(θα−1p θ1−αq − (1− θp)α−1(1− θq)1−α)

f (2)(θp) = θα−2p θ1−αq + (1− θp)α−2(1− θq)1−α

f (3)(θp) = (α− 2)(θα−3p θ1−αq − (1− θp)α−3(1− θq)1−α)

where fn(·) denotes nth derivative of f(·). Note thatf (1)(θq) = 0. Thus, the α-divergence from p(x) to q(x)and its 3rd order Taylor expansion with respect to θp,which is close to θq , becomes:

Dα(p‖q) =1− θpαθq1−α − (1− θp)α(1− θq)1−α

α(1− α)(10)

≈ T1(θq)(θp − θq)2 + T2(θq)(α− 2)(θp − θq)3(11)

where T1(θq) = 12 ( 1θq

+ 11−θq ), T2(θq) = 1

6 ( 1θq2 − 1

(1−θq)2 )

and T1(θq), T2(θq) > 0. Then, given 0 < α < 2and θp > θq , the 3rd order term in equation (11) isnegative. So by increasing α, the divergence from pto q increases. On the other hand if θp < θq , the 3rdorder term in Equation (11) is positive and increasingα decreases the divergence. Figure 1 illustrates thisobservation and motivates Proposition 4.1 below. Itshould be noted that T2(θq) approaches to zero asθq → 1/2, which indicates this property becomesmore distinctive when the distribution is skewed thanwhen the distribution is balanced. Later we describeProposition 4.2 and its Corollary 4.3, which will beused extensively in Section 4.3.

Proposition 4.1. Assume that we are given Bernoullidistributions p(x), q(x) as above and α ∈ (0, 2). Givenθq < 1/2, ∃ ε > 0 s.t. Dα(p‖q) is a monotonic “increas-ing” function of α where θp ∈ (θq, θq+ε), and ∃ ε′ > 0 s.t.Dα(p‖q) is a monotonic “decreasing” function of α whereθp ∈ (θq − ε′, θq).

Proof: This follows from equation (11).

Proposition 4.2. Dα(p‖q) is convex w.r.t. θp.

Proof: Second derivative of equation (10) withrespect to θp is positive.

Corollary 4.3. Given binary distributions, p(x), q(x),r(x), where 0 < θp < θq < θr < 1, Dα(q‖p) < Dα(r‖p)and Dα(q‖r) <α (p‖r).

Proof: Since Dα(s(x)‖t(x)) ≥ 0 and is equal if andonly if s(x) = t(x), using proposition 4.2, corollary 4.3directly follows.

Fig. 1: Dα(p||q) with different α’s, where θq = 0.1.x-axis represents θp, y-axis represents Dα(p||q). Notethat Dα(p||q) becomes more “distinctive” with respectto α when θq � 1/2.

4.3 Effect of varying α

Coming back to our original problem, let us assumethat we have a binary classification problem whosepositive class ratio is θc where 0 < θc � 1/2(imbalanced class). After a split, the training data isdivided into two subsets: one with higher (> θc) andthe other with lower (< θc) positive class ratio. Letus call the subset with higher positive class ratio aspositive subset, and the other as negative subset. Withoutloss of generality, suppose we have binary featuresX1, X2, ..., Xn and p(y = 1|xi = 0) < p(y = 1) < p(y =1|xi = 1) and p(xi) ≈ p(xj) for any i, j. From equation(9) the α-divergence criterion becomes:

Dα(p(xi, y)‖p(xi)p(y))

= p(xi = 1)Dα(p(y|xi = 1)‖p(y))

+ p(xi = 0)Dα(p(y|xi = 0)‖p(y)) (12)

where 1 ≤ i ≤ n. From Proposition 4.1, we observethat increase in α increases Dα(p(y|xi = 1)‖p(y)) anddecreases Dα(p(y|xi = 0)‖p(y)) (lower-bounded by 0).

Dα(p(xi, y)‖p(xi)p(y))

≈ p(xi = 1)Dα(p(y|xi = 1)‖p(y)) + const.

From Corollary 4.3, increasing α shifts our focus tohigh p(y = 1|xi = 1). In other words, increas-ing α results in the splitting feature having higherp(y = 1|xi = 1), positive predictive value (PPV) orprecision. On the other hand reducing α results inlower Dα(p(y|xi = 1)‖p(y)) and higher Dα(p(y|xi =0)‖p(y)). As a result, reducing α gives higher p(y =0|xi = 0), negative predictive value (NPV) for the split-ting features. Note that the above analysis is basedon Taylor expansion of α-divergence that holds truewhen p(y|xi) ≈ p(y), which is the case when datasetsare imbalanced. Note that this property may not holdif p(y|xi) differs a lot from p(y).


Different values of α can induce different splittingvariables xi’s, which in turn result in different de-cision trees. In this paper, we call this property ofα-Tree as “α-diversification”. It is important to notethat degree of α-diversification differs from dataset todataset. As a trivial example, for a dataset with onebinary feature, varying α does not affect the choiceof splitting variable. Thus, we define “α-index” for adataset as follows:

Definition 4.4 (α-index). For a given dataset D, α-index α(D) is a number of distinct α-trees Tα that canbe obtained by varying the value of α.

In other words, α-index for a given dataset Dis a degree of diversity obtained from α-divergencesplitting criteria. In Section 7, we will see that this α-index is closely related to the AUROC performancemetric in an ensemble framework. This agrees withtheoretical results that show the importance of diver-sity in ensemble performance [39], [40]α(D) can be obtained by grid search on α ∈ (0, 2).

However, in practice, this process can be computa-tionally expensive. Thus, we propose an approximateα-index, α(D) using 1-depth trees:

Definition 4.5 (approximate α-index). For a givendataset D, an approximate α-index α(D) is a number ofdistinct 1-depth α-trees Tα that can be obtained by varyingthe value of α.

This approximation is under-estimation, α(D) ≤α(D) w.p. 1, but can be computed very fast even withgrid search.

4.4 Connection to DKM and CART

The family of α-divergence naturally includes C4.5’ssplitting criterion, but the connection to DKM andCART is not that obvious. To see the relation betweenα-divergence and the splitting functions of DKMand CART, we extend the definition of α-divergence,Equation (9), as follows:

Definition 4.6. The extended α-divergence criterion for-mula is defined as EX [Dα(p(y|x)‖q(y))] where q(y) is anyarbitrary probability distribution.

Definition 4.6 is defined by replacing p(y) with anyarbitrary distribution q(y), which serves as a referencedistribution. The connection to DKM and CART issummarized in the following two propositions:

Proposition 4.7. Given a binary classification problem, ifα = 2 and q(y) = ( 1

2 ,12 ) then the extended α-divergence

splitting criterion gives the same splitting feature as theGini impurity criterion in CART.

Proof: Assume a binary classification problem,y ∈ {0, 1} and binary feature x ∈ {0, 1}. Since thedistribution of y is fixed for choosing the best feature

x, we can derive the following equation:

Gini =∑y

p(y)(1− p(y))−∑x

p(x)∑y

p(y|x)(1− p(y|x)).

= EX [1

2−

∑y

p(y|x)(1− p(y|x))] + const

∝ EX [D2(p(y|x)‖q(y))] (13)

where q(y) = ( 12 ,

12 ). Equation (13) follows from

Equation (8). Linear relation between the Gini split-ting formula and the extended α-divergence formulacompletes the proof.

Proposition 4.8. Given a binary classification problem, ifα = 1

2 and q(y) = p(y|x) then the extended α-divergencesplitting criterion gives the same splitting feature as theDKM criterion.

Proof: Assume a binary classification problem, y ∈{0, 1} and binary feature x ∈ {0, 1}, then the splittingcriterion function of DKM is:

DKM =∏y

√p(y)−

∏x

p(x)∏y

√p(y|x)

= EX [1

2−∏y

√p(y|x)] + const

∝ EX [D 12(p(y|x)‖q(y))] (14)

where q(y) = p(y|x). Equation (14) follows from Equa-tion (6). Linear relation between the DKM formulaand α-divergence completes the proof.

CART implicitly assumes a balanced reference dis-tribution while DKM adaptively changes its referencedistribution for each feature. This explains why CARTgenerally performs poorly on imbalanced datasetsand DKM provides a more skew-insensitive decisiontree.

5 ENSEMBLE OF α-TREES

In this section, we present a general framework forcombining bagging-based ensemble methods and α-diversification, named the Ensemble of α-Trees (EAT)framework. Figure 2 shows its basic idea. When K = 1and no bagging is used, this EAT framework simplygenerates multiple α-trees for the given dataset. IfK > 1 and sampling with replacement is used, then α-diversification is built on top of a bagged C4.5 ensem-ble, leading to BEAT (Bagged EAT). Finally, if under-sampling is applied to each bag in the ensemble,we obtain α-diversified under-bagged C4.5 (UBEAT:Under-Bagged EAT). This framework can be appliedto any bagging-based ensemble methods. Algorithm 2illustrates its formal procedure. The inputs to the EATframework are training data and the number of databags (K). For each bag of data, a list of α’s thatresult in distinct α-Trees is obtained by grid search onα ∈ (0, 2). Note that if we fix α = 1 in algorithm 2, thealgorithm is the same as (under/over)-bagged C4.5ensemble classifiers.


D

Sample1 Sample2 ... SampleK

Tα Tα′ Tα ...

TenFig. 2: Diagram of EAT framework. Sampling meth-ods in the diagram can be random sampling, andunder/over-sampling.

Algorithm 2 EAT Framework

Input: training data (features X1, X2, ..., Xn and la-bel Y ), K (Number of Trees)for for i = 1 to K doDS ← Sample(training data)Grid search a list of α’s that increase α(DS)Ti ←

∑α Tα(DS)/α(DS)

end forfor for each test record x doy ← Average(T1(x), T2(x), T3(x), ..., TK(x))

end for

6 LIFT-BOOSTING ENSEMBLE OF α-TREES

In this section, we present a second ensemble frame-work for imbalanced data by focusing on the liftmetric. The lift chart is another widely used graph-ical performance metric in class imbalance problems,when measuring the performance of targeting modelsin marketing and medical applications [41], [42]. Westart by modifying the standard stopping rule in de-cision trees to reflect our different evaluation metric,lift. Integrating this stopping rule into α-Tree resultsin a modified α-Tree algorithm. Then, the algorithmfor Lift-boosting Ensemble of α-Trees (LEAT) is pre-sented.

6.1 Modified Stopping Rule

When partitioning algorithms are used in targetingmodels, the aim is either (i) to extract a fixed sizesubgroup with the highest lift or (ii) to identify thelargest possible subgroups with lift greater than aspecified value. The largest possible subgroups canbe formally written as follows:

Definition 6.1 (Coverage). Suppose a partitioning al-gorithm f separates the total data points DT into Ppartitions: {D1,D2, ...DP } where Di ∩ Dj = ∅, ∀i 6= j.For a given lift value L, the coverage of the partitioningalgorithm f is:

Coverage(L) =| ∪{i:lift(Di)>L} Di|

|DT |(15)

where lift(Di) is a function that calculates the lift ofpartition Di.

Lift and coverage are bijectively related in a liftchart, so both goals are conceptually the same; how-ever, these goals lead to different ways of buildingdecision trees. For example, the first objective doesnot significantly change the traditional decision treebuilding process. Decision trees can be built withoutconsidering the lift constraint, since the goal is to finda top subgroup with non-specific lift values. On theother hand, in the second approach, the constructionprocess needs a fundamental change. In this regime,the objective of decision trees becomes maximizingthe coverage of subgroups that satisfy the requiredlift value.

In this paper, we design a decision tree algorithmthat maximizes the coverage with a fixed minimumlift value by modifying the stopping rules. Typically,decision trees stop further splits when (i) a numberof samples at a node are not enough or (ii) no featurewould result in further impurity reduction. However,these stopping rules do not maximize the coveragewith a fixed lift. Suppose we have a decision treenode that already achieves a higher lift value than apre-specified lift value. If a further split is performed,one child node would have a lower lift value than itsparent; nevertheless, in the worst case, the lift valueof this child might be even lower than the requiredlift value, resulting in decrease in coverage. Therefore,to maximize the coverage, decision trees need to stoptheir growth when their nodes meet a certain level oflift.

A thorough statistical test should be incorporated toconclude that a node has reached a desired lift value.As a decision tree grows, its nodes would have lessexamples, which increases the variance of associatedestimated parameters such as positive class ratio. Thisissue becomes severe especially when we deal withclass imbalance problems, as the number of positiveexamples in a node would be very small in most cases.In this paper, we perform the hypothesis test (uppertail test) using p-value:

H0 : θ = L · θr vs. H1 : θ < L · θr

where L is a pre-specified desired lift value and θr isthe positive class ratio of the entire dataset. Then, thetest statistic (Z) is:

Z =θ − L · θrσ/√M

where M is the total number of examples in a node,θ is the number of positive examples divided by M

and σ is√θ(1− θ). Given a level of significance β and

assuming that the null hypothesis is true, we have therejection region of Z > zβ . So we have:


Procedure 6.2. (Lift Stopping Rule) If θ − zβσ√M

>

L · θr, then a node reached a desired lift.

As a result, each node resulting from this stoppingrule would have a higher lift than L (a pre-specifiedlift, threshold) with a level of significance β.

For a given lift value L, coverage value attain-able by any partitioning algorithm is upper-bounded.Theorem 6.3 states that there exists the maximumachievable coverage for a given lift L.

Theorem 6.3 (Ideal Max Coverage). For a given lift L,Coveragef (L) ≤ 1/L for any partitioning algorithm f .

Proof: Suppose there exists a partition that isbigger than 1/L. Then the number of positive samplesin the partition is strictly bigger than the total numberof positive samples in the data, which contradicts ourassumption. On the other hand, if the partition exactlyachieves the lift L, then the size of the partition isequal to 1/L.

In Section 7, we will compare the performance ofLEAT to this maximum achievable coverage.

6.2 LEAT algorithmIn this section, we illustrate a lift-aware α-Tree algo-rithm using the modified stopping rules and proposea new kind of ensemble framework through thesemodified α-Trees. Algorithm 3 describes the modifiedα-Tree. The modified α-Tree outputs logical rule setsthat achieve at least lift L.

Algorithm 3 Grow a modified α-Tree

Input: training data (features X1, X2, ..., Xn and la-bel Y ), α, LOutput: α-Tree rule setSelect the best feature X∗, which gives the maxi-mum α-divergence criterionif (number of data points < cut off size) or(best feature == None) or (lift L is achieved) then

Stop growing.if lift L is achieved then

The node is positive.else

The node is negative.end if

elsepartition the training data into two subsets, basedon the value of X∗

left child ← Grow a single α-Tree (data with(X∗ = 1), α)right child ← Grow a single α-Tree (data with(X∗ = 0), α)

end if

The objective of LEAT is to identify the maximum-sized subgroups that have higher lift values thana threshold L. To achieve this goal, we utilize two

properties from our analysis. First, we recall thatthe modified α-tree outputs a set of rules, whichpartition the input data into two: (i) a higher liftgroup, which belongs to positive nodes in α-tree and(ii) a lower lift group, which corresponds to negativenodes. From the modified stopping rules, the lift ofthe higher lift group is larger than the threshold witha level of significance β. Also, note that different αvalues would result in different rule sets that maycover distinct subgroups. Therefore, if we combinethe positive rules from multiple α-Trees using logi-cal disjunction (union), we obtain a larger coverageof subgroups with lifts higher than the thresholdgiven a level of significance β. This rule ensembletechnique is firstly coined by Friedman et al. [12]to increase interpretability in ensemble frameworks.Their proposed algorithm “RuleFit” exhibits compa-rable performance with other ensemble techniques,while maintaining the simple rule-based interpretabil-ity. In this paper, we adopt such technique to themodified α-Trees, not only to keep interpretability,but also to increase coverage of positive nodes. Withthe modified stopping rule, this rule ensemble tech-nique yields monotonically increasing coverage withlift higher that a pre-specified threshold L. Utilizingthis lift-boosting mechanism, the LEAT (Lift-BoostingEnsemble of Alpha Trees) algorithm for an ensembleof K trees is illustrated in Algorithm 4.

Algorithm 4 Lift-Boosting Ensemble of α-Trees(LEAT)

Input: Data (features {Xi}ni=1 and class Y ), L(threshold), K (Number of Trees)Output: LEAT rule setfor for i = 1 to k do

Grid search a list of α’s that increase α(D)Build a modified α-Tree using algorithm 3

end forSet LEAT Rule Set ← ∅.for for each rule set in α(D) different α-tree rulesets do

LEAT Rule Set ← (LEAT Rule Set ∪Extract Positive Rules(ith α-tree rule set))

end forreturn LEAT Rule Set

7 EXPERIMENTAL RESULTSIn this section, we provide experimental results forEAT and LEAT frameworks.1

7.1 Dataset DescriptionThe twenty datasets used in this paper are fromthe KEEL Imbalanced Datasets2, the Behavioral Risk

1. Code for the different alpha-tree ensembles as well asfor generating the plots in this paper is now available at:https://github.com/yubineal/alphaTree .

2. http://sci2s.ugr.es/keel/datasets.php


TABLE 1: Datasets usedDataset Examples Features Positive Class %

glass4 214 9 6.07%glass5 214 9 4.21%

abalone19 4174 8 0.77%abalone9-18 731 8 5.74%

shuttle-c0-vs-c4 1829 9 6.72%yeast4 1484 8 3.43%yeast5 1484 8 2.97%yeast6 1484 8 2.36%

yeast-2 vs 8 482 8 4.15%yeast-2 vs 4 514 8 9.92%

page-1-3 vs 4 472 10 5.93%ecoli4 336 7 5.95%

vowel0 988 13 9.66%paw02a 800 2 12.5%03subcl5 800 2 12.5%04clover 800 2 12.5%

sick 3772 30 7.924%allhypo 3772 30 8.106%BRFSS ≈ 400K 22 11.82%MIMIC ≈ 300K 28 22.1%

Factor Surveillance System (BRFSS)3, and the Mul-tiparameter Intelligent Monitoring in Intensive Care(MIMIC) database4. Many of the KEEL datasets arederived from UCI Machine Learning Repository5 byeither choosing imbalanced datasets , or collaps-ing/removing some classes to make them to im-balanced binary datasets. paw02a, 03subcl5, and04clover are artificial datasets, which are not fromthe UCI repository. They are designed to simulatenoisy, borderline, and imbalanced examples in [43].Aside from the stated modifications, each dataset fromthe KEEL Repository is used “as is”.

The BRFSS dataset is the world’s largest telephonehealth survey, which began in 1984, tracking healthconditions and risk behaviors in the United States.The data are collected monthly in all 50+ states in theUnited States. The dataset contains information on avariety of diseases like diabetes, hypertension, cancer,asthma, HIV, etc, and in this paper we take the BRFSS2009 dataset and focus on diabetes rather than otherdiseases. The 2009 dataset contains more than 400,000records and 405 variables, and the diabetic (positiveclass) ratio is about 12%. Empty and less informativecolumns are dropped and we finally chose 22 vari-ables to perform our experiments. The selected vari-able includes Age, Body Mass Index (BMI), Educationlevel, and Income level.

The MIMIC dataset is a rich multi-parameter datagathered from more than 30,000 ICU patients. Thedataset contains various demographic information aswell as static and dynamic medical measurements. Inthis paper, we predict “septic shock” in the next 48hours of ICU patients using selected 28 predictors,including gender, age, body temperature, and pastICU experience. Septic shock is a medical condition

3. http://www.cdc.gov/brfss/4. http://mimic.physionet.org/abstract.html5. http://archive.ics.uci.edu/ml/

Fig. 4: Pair-wise correlation between α-Trees over 5×2cross-validations for 20 datasets. Each dot represents avalue obtained from one run, and boxes correspond tolower and upper quartiles. The lines inside the boxesare the sample medians.

Fig. 5: AUROC Gain vs. α-index. α-Index is positivelycorrelated with AUROC Gains. Each point representAUROC Gain of one experiment from 5 × 2 cross-validations for 20 datasets.

resulted from severe infection or sepsis, which canlead to death in the worst case. Table 1 summarizesthe datasets in this paper.

In Section 7.2, experimental results on EAT arepresented. Ensemble classifiers in the experiment, in-cluding a family of EAT, are trained on 10 randomlysampled training bags (with replacement), unless oth-erwise specified. Each base classifier in the ensemblesis a binary-split decision tree, and every base tree isgrown until i) it reaches tree-depth 3 or ii) a numberof instances in a node is less than 30. In Section 7.3,we illustrate experimental results on LEAT. α-Treesare grown until it satisfies the modified stopping rule,and the same rule applies to C4.5 in the comparison.

7.2 Experimental Results on EATAmong many imbalanced classification approaches,we pick two approaches as our baseline, BaggedC4.5 and Under-Bagged C4.5 [44]. In [14], BaggedC4.5 recorded the best performance among classic


Fig. 3: AUROC values from twenty datasets. α-diversified classifiers (EAT, BEAT, and UBEAT) usually recordhigher AUROC values than the corresponding baseline methods. Each dot represents a value obtained fromone run, and boxes correspond to lower and upper quartiles. The lines inside the boxes are the sample medians.

TABLE 2: Wilcoxon and t-Tests (3 bags)Null Alternative Test p-value Selected

Equal UBEAT > UC4.5 t-Test 2.5× 10−9 AlternativeEqual UBEAT > UC4.5 Wilcoxon 2.5× 10−9 AlternativeEqual BEAT > BC4.5 t-Test 8.5× 10−7 AlternativeEqual BEAT > BC4.5 Wilcoxon 2.4× 10−10 AlternativeEqual EAT > C4.5 t-Test 4.218× 10−15 AlternativeEqual EAT > C4.5 Wilcoxon 8× 10−16 AlternativeEqual EAT > UC4.5 t-Test 3.701× 10−7 AlternativeEqual EAT > UC4.5 Wilcoxon 0.0017 Alternative

ensembles including AdaBoost, AdaBoost.M1, andAdaBoost.M2, and Under-Bagged C4.5 was one of the

TABLE 3: Wilcoxon and t-Tests (10 bags)Null Alternative Test p-value Selected

Equal UBEAT > UC4.5 t-Test 0.0022 AlternativeEqual UBEAT > UC4.5 Wilcoxon 0.016 AlternativeEqual BEAT > BC4.5 t-Test 0.0024 AlternativeEqual BEAT > BC4.5 Wilcoxon 0.0096 AlternativeEqual EAT > C4.5 t-Test 0.0073 AlternativeEqual EAT > C4.5 Wilcoxon 0.014 AlternativeEqual EAT > UC4.5 t-Test 2.101× 10−5 AlternativeEqual EAT > UC4.5 Wilcoxon 0.1281 Alternative

best performing techniques for imbalanced classifica-tion among all 40+ algorithms used in the comparison.


TABLE 4: Effect of α-indexEstimate Std. Error t-value Pr(> |t|)

(Intercept) 1.00 0.002 462.81 < 2× 10−16

α-index 0.012 0.0008 15.29 < 2× 10−16

We first show three types of α-diversified classifiersusing EAT framework: EAT (α-diversified single tree),BEAT (α-diversified bagged trees), and UBEAT (α-diversified under-bagged trees) when K = 3. Notethat the EAT framework can be plugged into anybagging-based ensemble algorithms such as over-bagging. Along with the two baseline algorithms anda single C4.5, these six algorithms are applied to thedatasets, and AUROC’s from 5 × 2 cross-validationsare recorded. Figure 3 shows the results from thisexperiment. The results show that α-diversified en-sembles result in better AUROC performance than thecorresponding baseline approaches.

Statistical pairwise comparisons also reveal thatour α-diversified classifiers perform better than theoriginal classifiers. Table 2 and 3 show the results fromWilcoxon paired signed rank test [45] and paired t-testwhen K = 3 and K = 10, respectively. Both statisticaltests examine whether there exist significant differ-ences between a pair of algorithms. p-Values fromthe results provide evidence that α-diversification ac-tually helps imbalanced classification problems. Weobserve that for larger sized ensembles (K = 10),the differences between the α-diversified and theoriginal become less compared to when K = 3. Wecontemplate that this is because AUROC is upper-bounded by one, thus when every classifier performswell recording close to one AUROC values, it is hardto tell which one is better.

In the analysis of Random Forests, Hastie et al. [46]provided a formula for the variance of the ensembleaverage:

ρσ2 +1− ρK

σ2 (16)

where ρ represents positive pairwise correlation be-tween base classifiers, and K indicates the number ofclassifiers in an ensemble framework. As K increases,the effect of the second term decreases, but the firstterm remains. Therefore, reducing the pair-wise cor-relation between base classifiers becomes crucial inincreasing the overall performance. The idea of α-Treeis that it can yield diversified decision trees by varyingthe parameter α, whereas Random Forests approachthe problem by selecting random subset features.Figure 4 shows the pair-wise correlation observed inBEAT and BC4.5 (Bagged C4.5), respectively. The re-sults clearly indicate that α-diversification yields lesscorrelated decision trees; thus, the effect of the firstterm in Equation (16) eventually decreases, resultingin better performance.α-Index from Definition 4.4 measures a degree of

diversity from α-diversification. Figure 5 shows the

relationship between α-index and AUROC Gain, de-fined as follows:

AUROC Gain =α-diversified AUROC

original AUROC(17)

Table 4 shows the parameters for the regressed linein Figure 5. The result says that for a given dataset,adding another α-Tree increases the original AUROCby 1.2% on average.

7.3 Experimental Results on LEATLEATs with K = 5 are applied to the same datasetswith varying confidence levels zβ ’s, and the resul-tant coverage values are recorded for different liftthresholds L. Lift charts are constructed from these liftand coverage pairs for each LEAT and C4.5 with themodified stopping rule. Figure 6 shows the results. Wecan observe that LEAT covers wider areas for a givenlift value than the modified C4.5. For example, forabalone19, ecoli4, and yeast5, LEAT approachesvery close to the optimal performance. However, forallhypo and shuttle-c0-vs-c4, both LEAT andthe modified C4.5 could not adjust their coveragesaccording to the given lift values. Overshooting per-formance observed in sick is due to the lower con-fidence level (90%). 95% confidence level partitionsfrom the same dataset does not overshoot over theideal coverage.

Distinct α-Trees highlight different properties of thedata, and eventually help cover a wider range ofexamples. Figure 7 shows the relationship betweenthe number of trees and the coverage. As can be seen,the coverage increases as the number of distinct α-trees increases. As the modified α-tree is now pa-rameterized by lift threshold, we obtain different α-indices for different lift thresholds. For example, for“sick dataset, α(D) = 5 for lift = 2, while α(D) = 7for lift = 4.5. Note that distinct α-Trees may havesignificant overlaps in their covered examples.

8 DISCUSSION

In this paper, we presented two kinds of ensemblesfor imbalanced data. We briefly summarize the maincontributions of this paper here:

1) We introduce a new decision tree algorithmnamed α-Tree using α-divergence. A generalizedtree induction formula is proposed, which in-cludes Gini, DKM, and C4.5 splitting criteria asspecial cases.

2) We propose an ensemble framework of α-Trees called EAT that consistently results inhigher AUROC values over various imbalanceddatasets.

3) We modify traditional stopping rules in decisiontree algorithms to directly reflect an evaluationmetric based on lift. This leads to an ensemblecalled LEAT that can increase coverage with


Fig. 6: Transposed Lift charts from twenty datasets. LEAT is much closer to the ideal lift curve than a normalC4.5. x-axis represents lift threshold, and y-axis is the corresponding coverage for the lift threshold. As LEATtakes lift threshold as an input, we transposed the traditional lift chart, which has coverage in x-axis and liftin y-axis.

high lift values. We re-emphasize that lift is aprevalent evaluation metric in commercial andmedical applications involving class imbalanceproblems.

The proposed α-Trees with various α are muchmore uncorrelated than bagged C4.5’s. The α-diversification contributes in reducing the variance ofEAT ensemble classifiers, which is one of the maindifficulties in class imbalance problems. The LEATapproach, in contrast, incorporates both the noveldecision criterion parameterized by α and the dis-

junctive ensemble technique. For different α values,the α-divergence criterion results in different splittingfeatures, eventually leading to distinct decision trees.The modified stopping rules in α-tree guarantee thatevery positive node in α-tree would have higher liftvalues than a threshold with a level of significance β.Rules from multiple α-tree outputs are then disjunc-tively combined to increase coverage. LEAT outputsrule sets that increase the rule coverage for a fixedlift value and boost lift values for a fixed coverage.Our experimental results show that LEAT results in


Fig. 7: Number of Distinct α-Trees vs. Coverage increases as the number of distinct trees increases. Note thatα-indices are different for each lift threshold. For example, for “sick” dataset, α(D) = 5 for lift = 2, whileα(D) = 7 for lift = 4.5.

improved lift charts in many applications, and itsaverage performance increases as the number of treesin LEAT increases.

The LEAT framework is especially effective in ad-dressing imbalance class datasets in two ways: (i)increasing lift values for the minority class, and (ii)providing a simple rule set for classifying the minorityclass examples. The LEAT algorithm directly incor-porates a lift-based performance criterion in buildingthe base α-Trees, which results in a more exhaustivecoverage for a fixed lift value. Subgroups based onvarious lift values would provide effective targetingmodels especially in marketing applications whereclass imbalance datasets are prevalent. The simpleinterpretability of a decision tree is also maintainedin the LEAT framework, which help in building ex-pert systems. The disjunctive combination of rulesincreases the number of rules as well as their cov-erage. If a new record is caught by one of the LEAToutput rules, then the disjunctive combination allowsus to conclude that the record would have a higherlift value than the threshold. As LEAT provides rule-based descriptions, the output of LEAT is readilycommunicable with human experts. In domains thatneed supervision of human experts such as medicaland healthcare-related fields, this property is veryattractive.

Both EAT and LEAT can be extended using theextended α-divergence criterion. Moreover, the baseclassifiers of BEAT can be much more uncorrelatedby adopting random feature selection as in RandomForests. Furthermore, in the LEAT framework, over-laps between different α-Tree rules should be studiedto reduce the number of trees needed. Also, over-lapping rules can be reduced by examining logicalrelationships, which simplifies the LEAT output evenmore. We leave these tasks as future work.

ACKNOWLEDGMENTS

This research was supported by NHARP and NSF IIS-1016614.

REFERENCES

[1] H. He and E. A. Garcia, “Learning from imbalanced data,” inIEEE Transactions on Knowledge and Data Engineering, vol. 21,2009.

[2] J. Laurikkala, “Improving identification of difficult smallclasses by blancing class distribution,” in Proceedings of the8th Conference of AI in Medicine in Europe: Artificial IntelligenceMedicine, 2001, pp. 63–66.

[3] G. Weiss and F. Provost, “The effect of class distribution onclassifier learning: An empirical study,” Dept. of ComputerScience, Rutgers University, Tech. Rep., 2001.

[4] K. McCarthy, B. Zarbar, and G. Weiss, “Does cost-sensitivelearning beat sampling for classifying rare classes?” in Proceed-ings of International Workshop Utility-Based Data Mining, 2005,pp. 69–77.

[5] R. Akbani, S. Kwek, and N. Japkowicz, “Applying supportvector machines to imbalanced datasets,” in Proceedings of the15th European Conference on Machine Learning, 2004.

[6] S. Ertekin, J. Huang, and C. L. Giles, “Learning on the border:Active learning in imbalanced data classification,” in Proceed-ings of the 30th annual international ACM SIGIR conference, 2007,pp. 823–824.

[7] N. Japkowicz and S. Stephen, “The class imbalance problem:A systematic study,” in Intelligent Data Analysis, vol. 6, no. 5,2002, pp. 429–449.

[8] G. Weiss and F. Provost, “Learning when training data arecostly: The effect of class distribution on tree induction,” inJournal of Artificial Intelligence Research, vol. 19, 2003, pp. 315–254.

[9] G. E. Batista, R. C. Prati, and M. C. Monard, “A study of thebehavior of several methods for balancing machine learningtraining data,” in ACM SIGKDD Explorations Newsletter, vol. 6,no. 1, 2004, pp. 20–29.

[10] Y. Sun, A. K. C. Wong, and M. S. Kamel, “Classification ofimbalanced data: A review,” International Journal of PatternRecognition, vol. 23, no. 4, pp. 687–719, 2009.

[11] N. V. Chawla, N. Japkowicz, and A. Kotcz, “Editorial: specialissue on learning from imbalanced data sets,” SIGKDD Explor.Newsl., vol. 6, pp. 1–6, 2004.

[12] J. H. Friedman and B. E. Popescu, “Predictive learning via ruleensembles,” The Annals of Applied Statistics, vol. 2, no. 3, pp.916–954, 2008.

[13] D. S. Associates, The New Direct Marketing, 3rd ed. McGraw-Hill, 1999.


[14] M. Galar, A. Fernandez, E. Barrenechea, H. Bustince, andF. Herrera, “A review on ensembles for the class imbalanceproblem: Bagging-, boosting-, and hybrid-based approaches,”IEEE Transactions on Systems, Man, and Cybernetics - Part C:Applications and Reviews, 2011.

[15] D. D. Lewis and J. Catlett, “Heterogeneous uncertainty sam-pling for supervised learning,” in Proceedings of the EleventhInternational Conference on Machine Learning, 1994, pp. 148–156.

[16] M. Kubat and S. Matwin, “Addressing the curse of imbalancedtraining sets: One-sided selection,” in Proceedings of the Four-teenth International Conference on Machine Learning, 1997, pp.179–186.

[17] N. Japkowicz, “The class imbalance problem: Significance andstrategies,” in Proceedings of the 2000 International Conference onArtificial Intelligence, 2000, pp. 111–117.

[18] N. V. Chawla and K. W. Bowyer, “Smote: Synthetic minorityover-sampling technique,” in Journal of Artificial IntelligenceResearch, vol. 16, 2002, pp. 321–357.

[19] J. Zhang and I. Mani, “kNN Approach to Unbalanced DataDistributions: A Case Study involving Information Extrac-tion,” in ICML ’03 Workshop on Learning from Imbalanced DatasetII, 2003.

[20] C. Drummond and R. C. Holte, “C4.5, class imabalance, andcost sensitivity: Why under-sampling beats over-sampling,” inWorkshop on Learning from Imbalanced Datasets II, ICML, 2003.

[21] J. Quinonero-Candela, M. Sugiyama, A. Schwaighofer, andN. D. Lawrence, Eds., Dataset Shift in Machine Learning. TheMIT Press, 2009.

[22] L. Breiman, “Technical note: Some properties of splitting cri-teria,” in Machine Learning, vol. 24, 1996, pp. 41–47.

[23] T. Dietterich, M. Kearns, and Y. Mansour, “Applying theweak learning framework to understand and improve c4.5,” inProceedings of the Thirteenth International Conference on MachineLearning, 1996, pp. 96–104.

[24] M. Kearns and Y. Mansour, “On the boosting ability of top-down decision tree learning algorithms,” in Proceedings ofSTOC, 1996, pp. 459–468.

[25] C. Drummond and R. C. Holte, “Exploiting the cost(in)sensitivity of decision tree splitting criteria,” in Proceedingsof ICML, 2000, pp. 239–246.

[26] P. A. Flach, “The geometry of ROC space: Understandingmahine learning metrics through roc isometrics,” in Proceed-ings of ICML, 2003, pp. 194–201.

[27] D. Karakos, J. Eisner, S. Khudanpur, and C. E. Priebe, “Cross-instance tuning of unsupervised document clustering algo-rithms,” in Proceedings of NAACL HLT, 2007, pp. 252–259.

[28] M. J. Pazzani, C. J. Merz, P. M. Murphy, K. Ali, T. Hume, andC. Brunk, “Reducing misclassification costs,” in Proceedings ofICML, 1994, pp. 217–225.

[29] P. Domingos, “Metacost: A general method for making clas-sifiers cost-sensitive,” in Proceedings of the Fifth InternationalConference on Knowledge Discovery and Data Mining, 1999, pp.155–164.

[30] L. Breiman, J. Friedman, R. A. Olshen, and C. J. Stone, Clas-sification and Regression Trees. Chapman & Hall/CRC Press,1984.

[31] M. A. Maloof, “Learning When Data Sets Are Imbalanced andWhen Costs Are Unequal and Unknown,” in Proceedings ofICML ’03 Workshop on Learning from Imbalanced Dataset II, 2003.

[32] N. V. Chawla, “Many are better than one: Improving prob-abilistic estimates from decision trees,” in Machine LearningChallenges, 2006, pp. 41–55.

[33] R. E. Banfield, L. O. Hall, K. W. Bowyer, and W. P. Kegelmeyer,“A comparison of decision tree ensemble creation techniques,”in IEEE Transactions on Pattern Analysis and Machine Intelligence,2006.

[34] C. Chen, A. Liaw, and L. Breiman, “Using random forest tolearn imbalanced data,” Dept. of Statistics, U.C. Berkeley, Tech.Rep., 2004.

[35] L. Rokach and O. Z. Maimon, Data Mining with Decision Trees:Theory and Applications. World Scientific Publishing, 2008.

[36] J. R. Quinlan, C4.5: programs for machine learning. MorganKaufmann Publishers, 1993.

[37] R. O. Duda, P. E. Hart, and D. G. Stork, Pattern Classification.Wiley, 2001.

[38] H. Zhu and R. Rohwer, “Information geometric measurementsof generalization,” Aston University, Tech. Rep. 4350, 1995.

[39] K. Tumer and J. Ghosh, “Analysis of decision boundariesin linearly combined neural classifiers,” Pattern Recognition,vol. 29, no. 2, pp. 341–348, 1996.

[40] L. I. Kuncheva, Combining Pattern Classifiers: Methods and Al-gorithms. Wiley, 2004.

[41] J. Burez and D. V. den Poel, “Handling class imbalance incustomer churn prediction,” Expert Systems with Applications,vol. 36, no. 3, pp. 4626–4636, 2008.

[42] C. Ling, C. X. Ling, and C. Li, “Data mining for directmarketing: Problems and solutions,” in Proceedings of SIGKDD,1998, pp. 73–79.

[43] K. Napierala, J. Stefanowski, and S. Wilk, “Learning from im-balanced data in presence of noisy and borderline examples,”in 7th International Conference on Rough Sets and Current Trendsin Computing, 2010, pp. 158–167.

[44] R. Barandela, J. S. Sanchez, and R. M. Valdovinos, “Newapplications of ensembles of classifiers,” Pattern Analysis andApplications, vol. 6, no. 3, pp. 245–256, 2003.

[45] F. Wilcoxon, “Individual comparisons by ranking methods,”Biometrics Bull., vol. 1, no. 6, pp. 80–83, 1945.

[46] T. Hastie, R. Tibshirani, and J. Friedman, The Elements of Sta-tistical Learning: Data Mining, Inference, and Prediction, 2nd ed.Springer, 2009.

Yubin Park Yubin Park is currently a Ph.D.student in the Electrical and Computer En-gineering Department at the University ofTexas at Austin. He is a member of IntelligentData Exploration and Analysis Lab (IDEAL),and before joining the lab, he is educated atKAIST (B.Sc, 2008). His research interestsare data mining and machine learning appli-cations in healthcare systems and businessanalytics.

Joydeep Ghosh Joydeep Ghosh is currentlythe Schlumberger Centennial Chair Profes-sor of Electrical and Computer Engineeringat the University of Texas, Austin. He joinedthe UT-Austin faculty in 1988 after being ed-ucated at IIT Kanpur (B. Tech ’83) and TheUniversity of Southern California (Ph.D88).He is the founder-director of IDEAL (Intel-ligent Data Exploration and Analysis Lab)and a Fellow of the IEEE. His research in-terests lie primarily in data mining and web

mining, predictive modeling / predictive analytics, machine learningapproaches such as adaptive multi-learner systems, and their appli-cations to a wide variety of complex engineering and AI problems.

Dr. Ghosh has published more than 300 refereed papers and 50book chapters, and co-edited over 20 books. His research has beensupported by the NSF, Yahoo!, Google, ONR, ARO, AFOSR, Intel,IBM, and several others. He has received 14 Best Paper Awards overthe years, including the 2005 Best Research Paper Award acrossUT and the 1992 Darlington Award given by the IEEE Circuits andSystems Society for the overall Best Paper in the areas of CAS/CAD.Dr. Ghosh has been a plenary/keynote speaker on several occasionssuch as KDIR’10, ISIT’08, ANNIE06 and MCS 2002, and has widelylectured on intelligent analysis of large-scale data. He served as theConference Co-Chair or Program Co-Chair for several top data min-ing oriented conferences, including SDM’12, KDD 2011, CIDM07,ICPR’08 (Pattern Recognition Track) and SDM’06. He was the Conf.Co-Chair for Artificial Neural Networks in Engineering (ANNIE)’93to ’96 and ’99 to ’03 and the founding chair of the Data MiningTech. Committee of the IEEE Computational Intelligence Society. Hehas also co-organized workshops on high dimensional clustering,Web Analytics, Web Mining and Parallel/ Distributed KnowledgeDiscovery, in the recent past.

ensembles of α-trees for imbalanced classification problems

Documents