Download - Commonly Used Classifiers
-
8/11/2019 Commonly Used Classifiers
1/33
Commonly Used Classification
Techniques and Recent Developments
Presented by Ke-Shiuan Lynn
-
8/11/2019 Commonly Used Classifiers
2/33
-
8/11/2019 Commonly Used Classifiers
3/33
-
8/11/2019 Commonly Used Classifiers
4/33
Terminology (cont.)
In practice, input vectors of different classesare rarely so neatly distinguishable. Samplesof different classes may have same input
vectors. Due to such a uncertainty, areas ofinput space can be clouded by a mixture ofsamples of different classes.
Input #2
Input #1
-
8/11/2019 Commonly Used Classifiers
5/33
Terminology (cont.)
The optimal classifieris the one expected to
produce the least number of misclassifications.
Such misclassifications are due to uncertainty in the
problem rather than a deficiency in the decision
regions.
Input #2
Input #1
-
8/11/2019 Commonly Used Classifiers
6/33
-
8/11/2019 Commonly Used Classifiers
7/33
Types of Models
Decision-Region Boundaries
Probability Density Functions
Posterior Probabilities
-
8/11/2019 Commonly Used Classifiers
8/33
Decision-Region Boundaries
This type of model defines decision regions
by explicitly constructing boundaries in the
input space.
These models attempt to minimize the
number of expected misclassifications by
placing boundaries appropriately in the
input space.
-
8/11/2019 Commonly Used Classifiers
9/33
Probability Density Functions (PDFs) The models of this type attempt toconstruct aprobability density function,p(x|C), that maps a pointxin the input
space to class C. Prior probabilities,p(C), is to be estimated
from the given database.
This model assigns the most probable classto an input vectorxby selecting the classmaximizingp(C)p(x|C).
-
8/11/2019 Commonly Used Classifiers
10/33
Posterior Probabilities
Let there be mpossible classes denoted C1,
C2, , Cm. This type of models attempts to
generate mposterior probabilitiesp(Ci
|x),
i=1, 2, , mfor any input vectorx.
The classification is made in the way that
the input vector is assigned to the class
associated with maximalp(Ci|x).
-
8/11/2019 Commonly Used Classifiers
11/33
Approaches to Modeling
Fixed models
Parametric models
Nonparametric models
-
8/11/2019 Commonly Used Classifiers
12/33
Fixed models
Fixed model is used when the exact input-
output relationship is known.
Decision region boundary: A known thresholdvalue (e.g. A particular BMI value for defining
obesity)
PDF: When each classs PDF can be obtained Posterior probability: when the probability that
any observation belongs to each class is know.
-
8/11/2019 Commonly Used Classifiers
13/33
-
8/11/2019 Commonly Used Classifiers
14/33
Parametric Models (cont.)
Decision-region boundary: Linear
discriminant function e.g.
y=ax1
+bx2
+cx3
+d
PDF: Multivariate Gaussian function
Posterior probability: Logistic regression
-
8/11/2019 Commonly Used Classifiers
15/33
Nonparametric Models
Nonparametric model is used when the
relationships between input vectors and
their associated classes are not well
understood.
Models of varying smoothness and
complexity are generated and the one with
best generalizationis chosen.
-
8/11/2019 Commonly Used Classifiers
16/33
Nonparametric Models (cont.)
Decision-region boundary: LearningVector Quantization (LVQ),Knearestneighbor classifier, decision tree.
PDF: Gaussian mixture methods, Pazenswindow.
Posterior probability: Artificial neural
network (ANN), radial basis function(RBF), group method of data handling(GMDH)
-
8/11/2019 Commonly Used Classifiers
17/33
Commonly Used AlgorithmsParametric Nonparametric
Linear regression
Logistic regression
Unimodal Gaussian
Backpropagation
Radial basis function
K nearest neighbor
Gaussian mixture
Nearest clustering
Binary/Linear decision tree
Projection pursuit
Estimate-Maximize clusteringMultivariate Adaptive Regression Spline (MARS)
Group Method of Data Handling (GMDH)
Parzens window
Learning Vector Quentization (LVQ)
-
8/11/2019 Commonly Used Classifiers
18/33
-
8/11/2019 Commonly Used Classifiers
19/33
Memory UsageAlgorithm Memory Usage
Linear / Logistic regression Very low
Unimodal Gaussian Very low
Backpropagation Low
Radial basis function Medium
K nearest neighbor High
Gaussian mixture Medium
Nearest clustering Medium
Binary / Linear decision tree Low
Projection pursuit Low
Estimate-Maximize clustering MediumMARS Low
GMDH Low
Parzens window High
LVQ Medium
-
8/11/2019 Commonly Used Classifiers
20/33
Training TimeAlgorithm Training Time
Linear / Logistic regression Fast-Medium
Unimodal Gaussian Fast-Medium
Backpropagation Slow
Radial basis function Medium
K nearest neighbor No training required
Gaussian mixture Medium-Slow
Nearest clustering Medium
Binary / Linear decision tree Fast
Projection pursuit Medium
Estimate-Maximize clustering MediumMARS Medium
GMDH Fast-Medium
Parzens window Fast
LVQ Slow
-
8/11/2019 Commonly Used Classifiers
21/33
Classification timeAlgorithm Classification time
Linear / Logistic regression Very fast
Unimodal Gaussian Fast
Backpropagation Very fast
Radial basis function Medium
K nearest neighbor Slow
Gaussian mixture Medium
Nearest clustering Fast-medium
Binary / Linear decision tree Very fast
Projection pursuit Fast
Estimate-Maximize clustering MediumMARS Fast
GMDH Fast
Parzens window Slow
LVQ Medium
-
8/11/2019 Commonly Used Classifiers
22/33
Comparison of Algorithms
Linear regression:y = w0+w1x1+w2x2 ++wNxN
Logistic regression:
Linear and Logistic regressions both tend to
explicitly construct the decision-region
boundaries.
Advantages: Easy implementation, easy
explanation of input-output relationship Disadvantages: Limited complexity on the
constructed boundary
)1(1
e
y N
i iixww
10
-
8/11/2019 Commonly Used Classifiers
23/33
Comparison of Algorithms (cont)
Binary decision tree:
Binary and Linear decision trees also tend toexplicitly construct the decision-region
boundaries.
Advantages: Easy implementation, easy
explanation of input-output relationship
Disadvantages: Limited complexity on the
constructed boundary, the tree structure may not
be global optimal.
Root
xi>=c1 xi=c2 xj=c3 xk
-
8/11/2019 Commonly Used Classifiers
24/33
Comparison of Algorithms (cont)
Neural Network:
Feedforward neural network and radial-basisfunction network both tend to implicitly construct
the decision-region boundaries.
Advantages: They can both approximate any
complex decision boundaries provided that enough
nodes are used.
Disadvantages: Long training time
-
8/11/2019 Commonly Used Classifiers
25/33
Comparison of Algorithms (cont) Supporting vector machine
Supporting vector machine also tends to implicitly
construct the decision-region boundaries.
Advantages: This type of classifier has been shown to
have good generalization capability.
-
8/11/2019 Commonly Used Classifiers
26/33
-
8/11/2019 Commonly Used Classifiers
27/33
-
8/11/2019 Commonly Used Classifiers
28/33
Comparison of Algorithms (cont)K nearest neighbor classifier
K nearest neighbor tends to construct posteriorprobabilitiesP(Cj|X)
Advantage: No training is required, confidence
level can be obtained Disadvantage: classification accuracy is low is
complex decision-region boundary exists, largestorage required.
-
8/11/2019 Commonly Used Classifiers
29/33
Other Useful Classifiers
Projection Pursuit: aims to decomposing
the task of high-dimensional modeling into
a sequence of low-dimensional modeling.
This algorithm consists of two stage: the
first stage projects the input data onto a
one-dimensional space while the second
stage construct the mapping from projected
space to the output space.
-
8/11/2019 Commonly Used Classifiers
30/33
Other Useful Classifiers (cont) Multivariate adaptive regression splines (MARS)
tends to approximate the decision-regionboundaries in two stages.
At the first stage, the algorithm partitions the statespace into small portions.
At the second stage, the algorithm construct alow-order polynomial to approximate the
decision-region boundary within each partition. Disadvantage: This algorithm is intractable for
problem with high (> 10) dimensional inputs
-
8/11/2019 Commonly Used Classifiers
31/33
Other Useful Classifiers (cont)
Group method of data handling (GMDH)also aims to approximate the decision-region boundaries using high-order
polynomial functions. The modeling process begins with a low
order polynomial, and then iteratively
combines terms to produce a higher orderpolynomial until the modeling accuracysaturates.
-
8/11/2019 Commonly Used Classifiers
32/33
Keep The Following In Mind
Use multiple algorithms without bias and
let your specific data help determine which
model is best suited for your problem.
Occams Razor: Entities should not be
multiplied unnecessarily -- "when you have
two competing models which make exactlythe same predictions to the data, the one
that is simpler is the better."
-
8/11/2019 Commonly Used Classifiers
33/33
A New Member In Our Group