Matlab8:K-NearestNeighborClassifiers

Cheng-HsinHsuNa#onalTsingHuaUniversity

DepartmentofComputerScience

SlidesarebasedonthematerialsfromProf.RogerJang

CS3330ScienAficCompuAng 1

04/12/16 2

Concept of KNNC • Steps:

1.  Find the first k nearest neighbors of a given point

2.  Determine the class of the given point by a majority vote among these k neighbors

Feature1

Feature2 :class-Apoint

:class-Bpoint:pointwithunknownclass

3-nearestneighborsThepointisclassBvia3NNC.

04/12/16 3

Flowchart for KNNC

Featureextrac*on

Datamodeling

Evalua*on

GeneralflowchartofPR: KNNC:

Fromrawdatatofeatures

DistanceComputa@onForKNNC

Clustering(op@onal)

04/12/16 4

Decision Boundary for 1NNC Delaunay triangulation: maximize the minimum angle of all the angles of each triangle Voronoi diagram: piecewise linear boundary

Characteristics of KNNC

•  Strengths of KNNC – Very Intuitive – No data modeling required

•  Drawbacks of KNNC – Massive computation required when dataset is

big – No straightforward way to choose the value of K – Rescaling the dataset along each dimension may

be tricky

04/12/16 5

04/12/16 6

Preprocessing/Variants for KNNC

• Preprocessing: – Data rescaling to have zero mean and unit

variance along each feature – Value of K obtained via trials and errors

• Variants: – Weighted votes – Nearest prototype classification – Edited nearest neighbor classification – k+k-nearest neighbor

Given (x1, x2, . . . , xN

), x

0n

=x

n

� µ

x

�

x

8n = 1, 2, . . . , N

Demos by Cleve

•  Delaunay triangles and Voronoi diagram – http://mirlab.org/jang/books/dcpr/example.rar

04/12/16 7

Natural Examples of Voronoi Diagram

04/12/16 8

Natural Examples of Voronoi Diagram (cont.)

04/12/16 9

1NNC Decision Boundaries

•  1NNC Decision boundaries

04/12/16 10

1NNC Distance/Posterior as Surfaces and Contours

04/12/16 11

Using Prototypes in KNNC

•  No. of prototypes for each class is 4.

04/12/16 12

10 20 30 40 50 60

5

10

15

20

25

sepal length

sepa

l wid

th

iris

10 20 30 40 50 60

5

10

15

20

25

Input_1

Inpu

t_2

Dataset and decision boundary

Decision Boundaries of Different Classifiers

• Quadratic classifier

• 1NNC classifier

• Naive Bayes classifier

04/12/16 13

10 20 30 40 50 60

5

10

15

20

25

sepal lengthse

pal w

idth

iris

Matlab#7Homework(M7)1.  (1%)Giventwosetsof2D

points:Redonesat{(0,1),(2,3),(4,5)};andblueonesat{(2,0),(5,2),(7,3)}.PleaseuseEuclideandistanceinthisexercise.a)  DrawtheKNNCdecision

boundaryforK=1.Markthecolorsforeachpoint.

b)  MulAplethe6points’xvaluesby10.Replotthefigure,andexplainhowthechangeaffectsthedecisionboundary.Howwouldthisaffectrealworldproblem?

c)  Whatistheoutputof3-NNCofthepoint(1,2)?Ifyoucanaddpointsinthe2Dpoints,inordertofliptheclassificaAonof(1,2)?Showyourwork,e.g.,whereyouaddthepointsandthevaluesofthem.

d)  WhatisthecomplexityofKNNCalgorithm?Showyourwork.

CS3330ScienAficCompuAng 14

QuesAons?

CS3330ScienAficCompuAng 15