introduction to pattern recognition chapter 1 (duda et al.) cs479/679 pattern recognition dr. george...
TRANSCRIPT
![Page 1: Introduction to Pattern Recognition Chapter 1 (Duda et al.) CS479/679 Pattern Recognition Dr. George Bebis 1](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56649e315503460f94b2274a/html5/thumbnails/1.jpg)
Introduction to Pattern Recognition
Chapter 1 (Duda et al.)
CS479/679 Pattern RecognitionDr. George Bebis
1
![Page 2: Introduction to Pattern Recognition Chapter 1 (Duda et al.) CS479/679 Pattern Recognition Dr. George Bebis 1](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56649e315503460f94b2274a/html5/thumbnails/2.jpg)
What is a Pattern?
• A pattern could be an object or event.
2
biometric patterns hand gesture patterns
![Page 3: Introduction to Pattern Recognition Chapter 1 (Duda et al.) CS479/679 Pattern Recognition Dr. George Bebis 1](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56649e315503460f94b2274a/html5/thumbnails/3.jpg)
What is a Pattern? (con’t)
• Loan/Credit card applications – Income, # of dependents, mortgage amount for credit worthiness
classification.
• Dating services– Age, hobbies, income for “desirability” classification.
• Web documents– Key-word based descriptions (e.g., documents containing
“football”, “NFL”) for document classification.
3
![Page 4: Introduction to Pattern Recognition Chapter 1 (Duda et al.) CS479/679 Pattern Recognition Dr. George Bebis 1](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56649e315503460f94b2274a/html5/thumbnails/4.jpg)
Pattern Class
• A collection of “similar” objects – two challenges:
– Intra-class variability
– Inter-class variability
4
Letters/Numbers that look similar
The letter “T” in different typefaces
![Page 5: Introduction to Pattern Recognition Chapter 1 (Duda et al.) CS479/679 Pattern Recognition Dr. George Bebis 1](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56649e315503460f94b2274a/html5/thumbnails/5.jpg)
What is Pattern Recognition?
• Assign a pattern to one of several known categories (or classes).
5
Gender Classification
![Page 6: Introduction to Pattern Recognition Chapter 1 (Duda et al.) CS479/679 Pattern Recognition Dr. George Bebis 1](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56649e315503460f94b2274a/html5/thumbnails/6.jpg)
What is Pattern Recognition? (cont’d)
6
Character Recognition
![Page 7: Introduction to Pattern Recognition Chapter 1 (Duda et al.) CS479/679 Pattern Recognition Dr. George Bebis 1](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56649e315503460f94b2274a/html5/thumbnails/7.jpg)
What is Pattern Recognition? (cont’d)
7
Speech Recognition
![Page 8: Introduction to Pattern Recognition Chapter 1 (Duda et al.) CS479/679 Pattern Recognition Dr. George Bebis 1](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56649e315503460f94b2274a/html5/thumbnails/8.jpg)
Modeling Pattern Classes
• Typically expressed in terms of a statistical model.– e.g., probability density function (Gaussian)
8
Gender Classification malefemale
![Page 9: Introduction to Pattern Recognition Chapter 1 (Duda et al.) CS479/679 Pattern Recognition Dr. George Bebis 1](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56649e315503460f94b2274a/html5/thumbnails/9.jpg)
Pattern Recognition Objectives
• Hypothesize the models that describe each pattern class (e.g., recover the process that generated the patterns).
• Given a novel pattern, choose the best-fitting model for it and then assign it to the pattern class associated with the model.
9
![Page 10: Introduction to Pattern Recognition Chapter 1 (Duda et al.) CS479/679 Pattern Recognition Dr. George Bebis 1](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56649e315503460f94b2274a/html5/thumbnails/10.jpg)
Classification vs Clustering
– Classification (known categories) – Clustering (creation of categories)
10
Category “A”
Category “B”
Classification (Recognition) (Supervised Classification)
Clustering(Unsupervised Classification)
![Page 11: Introduction to Pattern Recognition Chapter 1 (Duda et al.) CS479/679 Pattern Recognition Dr. George Bebis 1](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56649e315503460f94b2274a/html5/thumbnails/11.jpg)
Pattern Recognition Applications
11
![Page 12: Introduction to Pattern Recognition Chapter 1 (Duda et al.) CS479/679 Pattern Recognition Dr. George Bebis 1](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56649e315503460f94b2274a/html5/thumbnails/12.jpg)
Handwriting Recognition
12
![Page 13: Introduction to Pattern Recognition Chapter 1 (Duda et al.) CS479/679 Pattern Recognition Dr. George Bebis 1](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56649e315503460f94b2274a/html5/thumbnails/13.jpg)
License Plate Recognition
13
![Page 14: Introduction to Pattern Recognition Chapter 1 (Duda et al.) CS479/679 Pattern Recognition Dr. George Bebis 1](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56649e315503460f94b2274a/html5/thumbnails/14.jpg)
Face Detection
14
Example of unbalanced classes (i.e., faces vs non-faces)
![Page 15: Introduction to Pattern Recognition Chapter 1 (Duda et al.) CS479/679 Pattern Recognition Dr. George Bebis 1](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56649e315503460f94b2274a/html5/thumbnails/15.jpg)
Gender Classification
15
Example of balanced classes (i.e., male vs female)
![Page 16: Introduction to Pattern Recognition Chapter 1 (Duda et al.) CS479/679 Pattern Recognition Dr. George Bebis 1](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56649e315503460f94b2274a/html5/thumbnails/16.jpg)
Fingerprint Classification
16
![Page 17: Introduction to Pattern Recognition Chapter 1 (Duda et al.) CS479/679 Pattern Recognition Dr. George Bebis 1](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56649e315503460f94b2274a/html5/thumbnails/17.jpg)
Biometric Recognition
17
![Page 18: Introduction to Pattern Recognition Chapter 1 (Duda et al.) CS479/679 Pattern Recognition Dr. George Bebis 1](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56649e315503460f94b2274a/html5/thumbnails/18.jpg)
Land Cover Classification(from aerial or satellite images)
18
![Page 19: Introduction to Pattern Recognition Chapter 1 (Duda et al.) CS479/679 Pattern Recognition Dr. George Bebis 1](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56649e315503460f94b2274a/html5/thumbnails/19.jpg)
“Hot” Pattern Recognition Applications
• Recommendation systems– Amazon, Netflix
• Targeted advertising
19
![Page 20: Introduction to Pattern Recognition Chapter 1 (Duda et al.) CS479/679 Pattern Recognition Dr. George Bebis 1](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56649e315503460f94b2274a/html5/thumbnails/20.jpg)
The Netflix Prize
• Predict how much someone is going to enjoy a movie based on their movie preferences– $1M awarded in Sept. 2009
• Can software recommend movies to customers?– Not Rambo to Woody Allen fans– Not Saw VI if you’ve seen all previous Saw movies
20
![Page 21: Introduction to Pattern Recognition Chapter 1 (Duda et al.) CS479/679 Pattern Recognition Dr. George Bebis 1](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56649e315503460f94b2274a/html5/thumbnails/21.jpg)
Main Classification Approaches
Assumption: x is the input vector (pattern) y is the class label (class)
•Generative – Model the joint probability, p( x, y)– Make predictions by using Bayes rules to calculate p(ylx)– Pick the most likely label y
•Discriminative – Model p(ylx) directly, or learn a direct map from inputs x to the
class labels y.– Pick the most likely label y
21
![Page 22: Introduction to Pattern Recognition Chapter 1 (Duda et al.) CS479/679 Pattern Recognition Dr. George Bebis 1](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56649e315503460f94b2274a/html5/thumbnails/22.jpg)
Syntactic Pattern Recognition Approach
• Represent patterns in terms of simple primitives.
• Describe patterns using deterministic grammars or formal languages.
22
![Page 23: Introduction to Pattern Recognition Chapter 1 (Duda et al.) CS479/679 Pattern Recognition Dr. George Bebis 1](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56649e315503460f94b2274a/html5/thumbnails/23.jpg)
Complexity of PR – An Example
23
Problem: Sorting incoming fish on a conveyor belt.
Assumption: Two kind of fish: (1) sea bass (2) salmon
![Page 24: Introduction to Pattern Recognition Chapter 1 (Duda et al.) CS479/679 Pattern Recognition Dr. George Bebis 1](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56649e315503460f94b2274a/html5/thumbnails/24.jpg)
Pre-processing
24
(1) Image enhancement
(2) Separating touchingor occluding fish
(3) Finding the boundary of each fish
![Page 25: Introduction to Pattern Recognition Chapter 1 (Duda et al.) CS479/679 Pattern Recognition Dr. George Bebis 1](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56649e315503460f94b2274a/html5/thumbnails/25.jpg)
Feature Extraction
• Assume a fisherman told us that a sea bass is generally longer than a salmon.
• We can use length as a feature and decide between sea bass and salmon according to a threshold on length.
• How can we choose the threshold?
25
![Page 26: Introduction to Pattern Recognition Chapter 1 (Duda et al.) CS479/679 Pattern Recognition Dr. George Bebis 1](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56649e315503460f94b2274a/html5/thumbnails/26.jpg)
“Length” Histograms
• Even though sea bass is longer than salmon on the average, there are many examples of fish where this observation does not hold.
26
threshold l*
![Page 27: Introduction to Pattern Recognition Chapter 1 (Duda et al.) CS479/679 Pattern Recognition Dr. George Bebis 1](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56649e315503460f94b2274a/html5/thumbnails/27.jpg)
“Average Lightness” Histograms• Consider a different feature such as “average
lightness”
• It seems easier to choose the threshold x* but we still cannot make a perfect decision.
27
![Page 28: Introduction to Pattern Recognition Chapter 1 (Duda et al.) CS479/679 Pattern Recognition Dr. George Bebis 1](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56649e315503460f94b2274a/html5/thumbnails/28.jpg)
Cost of miss-classifications
• There are two possible classification errors:
(1) Deciding the fish was a sea bass when it was a salmon.
(2) Deciding the fish was a salmon when it was a sea bass.
• Are both errors equally important ?
28
![Page 29: Introduction to Pattern Recognition Chapter 1 (Duda et al.) CS479/679 Pattern Recognition Dr. George Bebis 1](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56649e315503460f94b2274a/html5/thumbnails/29.jpg)
Cost of miss-classifications (cont’d)
• Suppose the fish packing company knows that:– Customers who buy salmon will object vigorously if
they see sea bass in their cans.– Customers who buy sea bass will not be unhappy if
they occasionally see some expensive salmon in their cans.
• How does this knowledge affect our decision?
29
![Page 30: Introduction to Pattern Recognition Chapter 1 (Duda et al.) CS479/679 Pattern Recognition Dr. George Bebis 1](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56649e315503460f94b2274a/html5/thumbnails/30.jpg)
Multiple Features
• To improve recognition accuracy, we might have to use more than one features at a time.– Single features might not yield the best performance.– Using combinations of features might yield better
performance.
1
2
x
x
1
2
:
:
x lightness
x width
30
![Page 31: Introduction to Pattern Recognition Chapter 1 (Duda et al.) CS479/679 Pattern Recognition Dr. George Bebis 1](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56649e315503460f94b2274a/html5/thumbnails/31.jpg)
Multiple Features (cont’d)• Partition the feature space into two regions by
finding the decision boundary that minimizes the error.
31
![Page 32: Introduction to Pattern Recognition Chapter 1 (Duda et al.) CS479/679 Pattern Recognition Dr. George Bebis 1](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56649e315503460f94b2274a/html5/thumbnails/32.jpg)
How Many Features?
• Does adding more features always improve performance?– It might be difficult and computationally expensive to
extract certain features.– Correlated features do not improve performance.– “Curse” of dimensionality …
32
![Page 33: Introduction to Pattern Recognition Chapter 1 (Duda et al.) CS479/679 Pattern Recognition Dr. George Bebis 1](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56649e315503460f94b2274a/html5/thumbnails/33.jpg)
Curse of Dimensionality• Adding too many features can, paradoxically, lead to a
worsening of performance.– Divide each of the input features into a number of intervals, so that
the value of a feature can be specified approximately by saying in which interval it lies.
– If each input feature is divided into M divisions, then the total number of cells is Md (d: # of features) which grows exponentially with d.
– Since each cell must contain at least one point, the number of training data grows exponentially!
33
![Page 34: Introduction to Pattern Recognition Chapter 1 (Duda et al.) CS479/679 Pattern Recognition Dr. George Bebis 1](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56649e315503460f94b2274a/html5/thumbnails/34.jpg)
Model Complexity
• We can get perfect classification performance on the training data by choosing complex models.
• Complex models are tuned to the particular training samples, rather than on the characteristics of the true model.
34
How well can the model generalize to unknown samples?
overfitting
![Page 35: Introduction to Pattern Recognition Chapter 1 (Duda et al.) CS479/679 Pattern Recognition Dr. George Bebis 1](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56649e315503460f94b2274a/html5/thumbnails/35.jpg)
Generalization• Generalization is defined as the ability of a classifier to
produce correct results on novel patterns.• How can we improve generalization performance ?
– More training examples (i.e., better model estimates).– Simpler models usually yield better performance.
35
complex model simpler model
![Page 36: Introduction to Pattern Recognition Chapter 1 (Duda et al.) CS479/679 Pattern Recognition Dr. George Bebis 1](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56649e315503460f94b2274a/html5/thumbnails/36.jpg)
More on model complexity
36
Regression example: •Consider the following 10 sample points assuming some noise.•Green curve is the true function that generated the data.•Approximate the true function from the sample points.
![Page 37: Introduction to Pattern Recognition Chapter 1 (Duda et al.) CS479/679 Pattern Recognition Dr. George Bebis 1](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56649e315503460f94b2274a/html5/thumbnails/37.jpg)
More on model complexity (cont’d)
37
Polynomial curve fitting: polynomials having various orders, shown as red curves, fitted to the set of 10 sample points.
![Page 38: Introduction to Pattern Recognition Chapter 1 (Duda et al.) CS479/679 Pattern Recognition Dr. George Bebis 1](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56649e315503460f94b2274a/html5/thumbnails/38.jpg)
More on complexity (cont’d)
38
Polynomial curve fitting: 9’th order polynomials fitted to 15 and 100 sample points.
![Page 39: Introduction to Pattern Recognition Chapter 1 (Duda et al.) CS479/679 Pattern Recognition Dr. George Bebis 1](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56649e315503460f94b2274a/html5/thumbnails/39.jpg)
PR System – Two Phases
39
Training PhaseTest Phase
![Page 40: Introduction to Pattern Recognition Chapter 1 (Duda et al.) CS479/679 Pattern Recognition Dr. George Bebis 1](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56649e315503460f94b2274a/html5/thumbnails/40.jpg)
PR System (cont’d)• Sensing:
– Use a sensor (camera or microphone)– PR depends on bandwidth, resolution, sensitivity,
distortion of the sensor.
• Pre-processing:– Removal of noise in data.– Segmentation (i.e., isolation of patterns of interest
from background).
40
![Page 41: Introduction to Pattern Recognition Chapter 1 (Duda et al.) CS479/679 Pattern Recognition Dr. George Bebis 1](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56649e315503460f94b2274a/html5/thumbnails/41.jpg)
PR System (cont’d)
• Training/Test data– How do we know that we have collected an
adequately large and representative set of examples for training/testing the system?
41
![Page 42: Introduction to Pattern Recognition Chapter 1 (Duda et al.) CS479/679 Pattern Recognition Dr. George Bebis 1](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56649e315503460f94b2274a/html5/thumbnails/42.jpg)
PR System (cont’d)• Feature extraction:
– Discriminative features
– Invariant features (e.g., translation, rotation and scale)
– How many should we use ?– Are there ways to automatically learn which
features are best ?42
![Page 43: Introduction to Pattern Recognition Chapter 1 (Duda et al.) CS479/679 Pattern Recognition Dr. George Bebis 1](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56649e315503460f94b2274a/html5/thumbnails/43.jpg)
PR System (cont’d)
• Missing features:– Certain features might be missing (e.g., due to
occlusion).– How should we train the classifier with missing
features ?– How should the classifier make the best
decision with missing features ?
43
![Page 44: Introduction to Pattern Recognition Chapter 1 (Duda et al.) CS479/679 Pattern Recognition Dr. George Bebis 1](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56649e315503460f94b2274a/html5/thumbnails/44.jpg)
PR System (cont’d)• Model learning and estimation:
– Models complex than necessary lead to overfitting (i.e., good performance on the training data but poor performance on novel data).
– How can we adjust the complexity of the model ? (i.e., not very complex or simple).
44
![Page 45: Introduction to Pattern Recognition Chapter 1 (Duda et al.) CS479/679 Pattern Recognition Dr. George Bebis 1](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56649e315503460f94b2274a/html5/thumbnails/45.jpg)
PR System (cont’d)• Classification:
– Using features and learned models to assign a novel pattern to a category.
– Performance can be improved using a "pool" of classifiers.
– How should we build and combine multiple classifiers ?
45
"pool" of classifiers
![Page 46: Introduction to Pattern Recognition Chapter 1 (Duda et al.) CS479/679 Pattern Recognition Dr. George Bebis 1](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56649e315503460f94b2274a/html5/thumbnails/46.jpg)
PR System (cont’d)• Post-processing:
– Exploit context to improve performance.
46
How m ch info mation are y u mi
sing?
![Page 47: Introduction to Pattern Recognition Chapter 1 (Duda et al.) CS479/679 Pattern Recognition Dr. George Bebis 1](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56649e315503460f94b2274a/html5/thumbnails/47.jpg)
PR System (cont’d)
• Computational Complexity:– How does an algorithm scale with the number
of:• features• patterns • categories
– Consider tradeoffs between computational complexity and performance.
47
![Page 48: Introduction to Pattern Recognition Chapter 1 (Duda et al.) CS479/679 Pattern Recognition Dr. George Bebis 1](https://reader036.vdocuments.mx/reader036/viewer/2022062301/56649e315503460f94b2274a/html5/thumbnails/48.jpg)
General Purpose PR Systems
• Humans have the ability to switch rapidly and seamlessly between different pattern recognition tasks.
• It is very difficult to design a system that is capable of performing a variety of classification tasks.– Different decision tasks may require different features.– Different features might yield different solutions.– Different tradeoffs exist for different tasks.
48