lecture 1: course introduction - cs course...

Introduction to Pattern Recognition Ricardo Gutierrez-OsunaWright State University

1

Lecture 1: Course introduction

g Course organizationn Grading policyn Outline and calendar

g Introduction to pattern recognitionn Definitions and related termsn Features and patternsn Decision regions and discriminant functions

g Pattern recognition examplesg Pattern recognition approaches

n Statisticaln Neuraln Structural


2

Course organizationg Instructor

n Ricardo Gutierrez-Osunan Office: 401 RCn Tel: (937) 775-5120n Email: [email protected] http://www.cs.wright.edu/~rgutiern Office hours: MW 4:00-5:30 PM

g Meeting time and locationn 105 Biological Sciences Building, Tu-Th 5:35-6:50PM

g Gradingn Homework

g Bi-weekly (first 6 weeks)

n Examsg 1 midterm and 1 final

n Term Projectg Open-endedg In-class presentation

Weight (%) Homework 40

Project 20 Midterm 20

Final Exam 20


3

Course outline

g Introduction to pattern recognition (1)n What is pattern recognition?n Approaches to pattern recognition: statistical,

neural and structuralg Overview of background material (2)

n Random variables and Probabilityn Linear Algebran MATLAB®

g Decision Theory (1)n Likelihood Ratio Testn Probability of error, Bayes Risk

g Dimensionality reduction (2)n The curse of dimensionalityn Principal Components Analysisn Linear Discriminant Analysis

g Classification using distance metrics (2)n Linear and quadratic classifiersn The K Nearest Neighbor (KNN) classifier

g Classification using density functions (2)n Parameter estimation: Maximum Likelihoodn Non-Parametric density estimation: Histograms,

Kernels, KNNn Optimal and Naïve Bayes Classifiers

g Classification using Multilayer Perceptrons (2)

n Historical overviewn Learning: back-prop and enhancementsn The role of hidden and output units

g Validation (1)n Holdout, cross-validation, bootstrapn Data splits

g Unsupervised Learning (2)n K-means and ISODATAn Hierarchical clusteringn Competitive Learningn KohonenSelf-Organizing Maps

g Feature Selection (2)n Search strategies: exhaustive, sequential,

randomizedn Evaluation strategies: filter, wrapper


4

Tentative quarter calendar Date Topic Reading Assignments

1/4 Course introduction 1.1, 1.2, 1.5, 1.6 HW1

Wee

k 1

1/6 Random variables, Probability 1.8, 2.1.1, 2.1.2 HW1

1/11 Linear Algebra, MATLAB® App. A HW1 HW1 assigned

Wee

k 2

1/13 Bayesian Decision Theory 1.9 HW1

1/18 The curse of dimensionality, Principal Components Analysis 1.4, 8.6 HW1

Wee

k 3

1/20 Linear Discriminants Analysis 3.6 HW2

1/25 Linear and quadratic classifiers HW2 HW1 due HW2 assigned

Wee

k 4

1/27 The K Nearest Neighbors classifier HW2

2/1 Midterm W

eek

5 2/3 Parameter estimation, histograms,

KNN 2.1, 2.2,

2.3 HW3

2/8 Kernel Density Estimation 2.5 HW3 HW2 due HW3 assigned

Wee

k 6

2/10 MLPs: history, back-prop, enhancements

4.1-4.3, 4.8, 7.5 HW3

2/15 MLPs: the role of hidden and output units

6.5, 6.6, 6.11 HW3

Wee

k 7

2/17 Validation 9.1, 9.8 HW3

2/22 Unsupervised learning: statistical clustering 2.6, 5.9.3 HW3 due

Wee

k 8

2/24 Unsupervised learning: Competitive Learning, Kohonen 5.9.3 Project

Proposal due

2/29 Feature selection I: objective functions, sequential FS 8.5

Wee

k 9

3/2 Feature selection II: exponential and randomized FS 8.5

3/7 Review

Wee

k 10

3/9 Final Exam

3/14

Wee

k 11

3/16 Project Presentations 5:30-7:30 PM, BH 105 Project Report

due


5

Definition of pattern recognition

g Definitions from the literaturen “The assignment of a physical object or event to one of several pre-

specified categories” --Duda & Hartn “A problem of estimating density functions in a high-dimensional space

and dividing the space into the regions of categories or classes” --Fukunaga

n “Given some examples of complex signals and the correct decisions for them, make decisions automatically for a stream of future examples” –Ripley

n “The science that concerns the description or classification (recognition) of measurements” --Schalkoff

n “The process of giving names ω to observations x”, --Schürmannn Pattern Recognition is concerned with answering the question “What is

this?” --Morse


6

More on Pattern Recognition

g Related fieldsn Adaptive Signal Processingn Machine Learningn Artificial Neural Networks n Robotics and Visionn Cognitive Sciencesn Mathematical Statisticsn Nonlinear Optimization n Exploratory Data Analysisn Fuzzy and Genetic systemsn Detection and Estimation

Theoryn Formal Languagesn Structural Modeling

g Applicationsn Image Preprocessing /

Segmentationn Computer Visionn Speech Recognitionn Automated Target Recognition n Optical Character Recognitionn Seismic Analysisn Man and Machine Diagnosticsn Fingerprint Identificationn Industrial Inspectionn Financial Forecastn Medical Diagnosisn EKG Signal Analysis


7

Components of a pattern recognition systemg A typical pattern recognition system contains

n A sensorn A preprocessing mechanismn A feature extraction mechanism (manual or automated)n A classification or description algorithmn A set of examples (training set) already classified or described

Sensor / transducer

Preprocessing and

enhancement

Feature extraction

Classification algorithm

Description algorithm

The “real world”

Class assignment

Description

Feedback / Adaptation


8

Features and patterns (1)g Feature

n Feature is any distinctive aspect, quality or characteristicg Features may be symbolic (i.e., color) or numeric (i.e., height)

n Definitionsg The combination of d features is represented as a d -dimensional column vector called a feature vectorg The d-dimensional space defined by the feature vector is called the feature spaceg Objects are represented as points in feature space. This representation is called a scatter plot

g Patternn Pattern is a composite of traits or features characteristic of an individualn For our purposes, a pattern is a pair of variables {x,ω} where

g x is a collection of observations or features (feature vector)g ω is the concept behind the observation (label)

=

d

2

1

x

xx

x

x3

x1 x2

x

Feature vector Feature space (3D)

Fea

ture

2

Feature 1

Class 1

Class 2

Class 3

Scatter plot (2D)


9

Features and patterns (2)g What makes a “good” feature vector?

n The quality of a feature vector is related to its ability to discriminate examples from different classes

g Examples from the same class should have similar feature valuesg Examples from different classes have different feature values

g More feature properties

“Good” features “Bad” features

Highly correlated featuresNon-linear separabilityLinear separability Multi-modal


10

g A classifier can be represented as a set of discriminant functionsn The classifier assigns a feature vector x to class ωι if

Classifiersg The task of a classifier is to partition feature

space into class-labeled decision regionsn Borders between decision regions are called

decision boundariesn The classification of feature vector x consists of

determining which decision region it belongs to, and assign x to this class

R1R2

R3

R1

R2

R3

R4

ij)(g)(g ji ≠∀> xx

x2x2 x3

x3 xdxd

g1(x)g1(x)

x1x1

g2(x)g2(x) gC(x)gC(x)

Select maxSelect max

CostsCosts

Class assignment

Discriminant functions

Features


11

A simple pattern recognition problem

g Consider the problem of recognizing the letters L,P,O,E,Qn Determine a sufficient set of featuresn Design a tree-structured classifier

C>0?

V>0?

O>0?

H>0?P

E L

OQ

Start

YES NO

YESNO

YES NO

YES

Features

Character Verticalstraight

lines

Horizontalstraight

lines

Obliquestraight

lines

Curvedlines

L 1 1 0 0P 1 0 0 1O 0 0 0 1E 1 3 0 0Q 0 0 1 1


12

A realistic pattern recognition system (1)

g Consider the following scenario*n A fish processing plan wants to automate the process of sorting incoming

fish according to species (salmon or sea bass)n The automation system consists of

g a conveyor belt for incoming productsg two conveyor belts for sorted products g a pick-and-place robotic armg a vision system with an overhead CCD camerag a computer to analyze images and control the robot arm

*Adapted from Duda, Hart and Stork, Pattern Classification, 2nd Ed.

Conveyor belt

CCD camera

Conveyor belt (bass)

Conveyor belt (salmon)

Robotarm

computer


13

A realistic pattern recognition system (2)g Sensor

n The vision system captures an image as a new fish enters the sorting area

g Preprocessingn Image processing algorithms

g adjustments for average intensity levelsg segmentation to separate fish from background

g Feature Extractionn Suppose we know that, on the average, sea bass is larger than salmon

g From the segmented image we estimate the length of the fish

count

length

Sea bassSalmon

Decision boundary

g Classificationn Collect a set of examples from both speciesn Compute the distribution of lengths for both

classesn Determine a decision boundary (threshold)

that minimizes the classification errorn We estimate the classifier’s probability of

error and obtain a discouraging result of 40% n What do we do now?


14

Avg. scale intensityle

ngth Decision

boundary

Sea bass Salmon

count

Avg. scale intensity

Sea bass Salmon

Decision boundary

A realistic pattern recognition system (3)g Improving the performance of our PR system

n Determined to achieve a recognition rate of 95%, we try a number of featuresg Width, Area, Position of the eyes w.r.t. mouth...g only to find out that these features contain no discriminatory information

n Finally we find a “good” feature: average intensity of the scales

n We combine “length” and “average intensity of the scales” to improve class separability

n We compute a linear discriminant function to separate the two classes, and obtain a classification rate of 95.7%


15

A realistic pattern recognition system (4)

g Cost Versus Classification raten Our linear classifier was designed to minimize the overall

misclassification raten Is this the best objective function for our fish processing plant?

g The cost of misclassifying salmon as sea bass is that the end customer will occasionally find a tasty piece of salmon when he purchases sea bass

g The cost of misclassifying sea bass as salmon is an end customer upset when he finds a piece of sea bass purchased at the price of salmon

n Intuitively, we could adjust the decision boundary to minimize this cost function


leng

th Decision boundary

Sea bass Salmon


leng

th NewDecision boundary

Sea bass

Salmon


16

A realistic pattern recognition system (5)g The issue of generalization

n The recognition rate of our linear classifier (95.7%) met the design specs, but we still think we can improve the performance of the system

g We then design an artificial neural network with five hidden layers, a combination of logistic and hyperbolic tangent activation functions, train it with the Levenberg-Marquardt algorithm and obtain an impressive classification rate of 99.9975% with the following decision boundary

n Satisfied with our classifier, we integrate the system and deploy it to the fish processing plant

g After a few days, the plant manager calls to complain that the system is misclassifying an average of 25% of the fish

g What went wrong?


leng

th

SalmonSea bass


17

Pattern recognition approachesg Statistical (StatPR)

n Patterns classified based on an underlying statistical model of the features g The statistical model is defined by a family of class-conditional probability density

functions Pr(x| ci) (Probability of feature vector x given class ci)

g Syntactic (SyntPR)n Patterns classified based on measures of structural similarity

g Structure is represented by means of formal grammars or relational descriptions (graphs)

n SyntPR is used not only for classification, but also for descriptiong Typically, SyntPR approaches formulate hierarchical descriptions of complex patterns

built up from simpler sub patterns.

g Neural (NeurPR)n Classification is based on the response of a network of processing units

(neurons) to an input stimuli (pattern)g The response of the network is determined by the connectivity and strength of the

synaptic weightsn NeurPR is a trainable, non-algorithmic, black-box strategyn NeurPR is very attractive since

g it requires minimum a priori knowledgeg with enough layers and neurons, an ANN can create any complex decision region


18

-1.97 -1.96 -1.95 -1.94 -1.93 -1.92 -1.91 -1.9 -1.89 -1.88 -1.87

0.3

0.32

0.34

0.36

0.38

0.4

0.42

11111 1

1

5

1

55

1

1

1

1

1

5

1

3 11

13 1

11

11

5 55

1

1

31

5

1

55

2

13 3

5

3

1

11

5

2

5

5

2

5

2

331

1

3 2

5

22

5

3

5

2

2

1

3

5

11

2

5

43

1

5

33

1

3

5

3

5

1

2222

3

35

3

5 5

3

2

5

5

1

5

1

33

2

55

3

5

3

15

3

2

5

23 31

3

2

55

3

2

5

2

5

2

1

23

2

5

3

5

13

1

2

2

2

3

22

3

4

2

3

4

3

2

2

44

2

2

2

3

23

5

2

4

5

2

4

5

2

34

4

3

4

2

5

2

4

5

34

34

44

3

334

2

4

4

4

44

44

2

4

4

44

4 44

44

44

4

2

4

4

44

4

2

4

4

44

4

LDA

axis 1

axis

2

Statistical pattern recognition: an example

0 50 100 150 200

-40

-20

0

20

40

60

Sulawesy Kenya Arabian Sumatra Colombia

Sen

sor r

espo

nse

normalized data

Dimensionality reduction

Densityestimation


19

Structural pattern recognition: an example

Example Model Description

CB

D AT

DirectedGraph

A B C

T

D

ononon on

"the programcrashes thecomputer"

Grammar

P = {SENTENCE → NP + VPNOUN PHRASE → ART + NVERB PHRASE → V + NPNOUN → computer|programVERB → crashesARTICLE → the}

S

NP

ART N

VP

V NP

ART N

the program crashes the computer


20

Neural pattern recognition: an example

Inputlayer

Hiddenlayer

Outputlayer

weights

examples

examples

Inpu

ts (f

eatu

res)

Out

puts

(cl

asse

s)

Medium

Weak

Strong

lecture 1: course introduction - cs course...

Documents