![Page 1: Felix Mbuga Math 251 HW 03 - sjsu.edu€¦ · Problem 1: LDA vs PCA Black - setosa Red - versicolor Green - virginica-10 -5 0 5 4 5 6 7 8 9 Iris - LDA Linear Discriminant 1 2 t n](https://reader034.vdocuments.mx/reader034/viewer/2022050222/5f6791205ef52e15e842027b/html5/thumbnails/1.jpg)
MATH 251 CLASSIFICATION HW 03Felix Mbuga
![Page 2: Felix Mbuga Math 251 HW 03 - sjsu.edu€¦ · Problem 1: LDA vs PCA Black - setosa Red - versicolor Green - virginica-10 -5 0 5 4 5 6 7 8 9 Iris - LDA Linear Discriminant 1 2 t n](https://reader034.vdocuments.mx/reader034/viewer/2022050222/5f6791205ef52e15e842027b/html5/thumbnails/2.jpg)
Problem 1: LDA vs PCA
Sepal.Length
2.0
2.5
3.0
3.5
4.0
4.5 5.5 6.5 7.5
0.5
1.0
1.5
2.0
2.5
2.0 2.5 3.0 3.5 4.0
Sepal.Width
Petal.Length
1 2 3 4 5 6 7
0.5 1.0 1.5 2.0 2.5
4.5
5.5
6.5
7.5
12
34
56
7
Petal.Widthsetosaversicolorvirginica
Raw data
![Page 3: Felix Mbuga Math 251 HW 03 - sjsu.edu€¦ · Problem 1: LDA vs PCA Black - setosa Red - versicolor Green - virginica-10 -5 0 5 4 5 6 7 8 9 Iris - LDA Linear Discriminant 1 2 t n](https://reader034.vdocuments.mx/reader034/viewer/2022050222/5f6791205ef52e15e842027b/html5/thumbnails/3.jpg)
Problem 1: LDA vs PCA
Black - setosaRed - versicolorGreen - virginica
-10 -5 0 5
45
67
89
Iris - LDA
Linear Discriminant 1
Line
ar D
iscr
imin
ant 2
-3 -2 -1 0 1 2 3
-2-1
01
2
Iris - PCA
Principal Component 1
Prin
cipa
l Com
pone
nt 2
![Page 4: Felix Mbuga Math 251 HW 03 - sjsu.edu€¦ · Problem 1: LDA vs PCA Black - setosa Red - versicolor Green - virginica-10 -5 0 5 4 5 6 7 8 9 Iris - LDA Linear Discriminant 1 2 t n](https://reader034.vdocuments.mx/reader034/viewer/2022050222/5f6791205ef52e15e842027b/html5/thumbnails/4.jpg)
Problem 3: PCA 95% + LDA
-15 -10 -5 0
-10
-50
5
USPS - PCA 95% - 0 & 1
Principal Component 1
Prin
cipa
l Com
pone
nt 2
01
![Page 5: Felix Mbuga Math 251 HW 03 - sjsu.edu€¦ · Problem 1: LDA vs PCA Black - setosa Red - versicolor Green - virginica-10 -5 0 5 4 5 6 7 8 9 Iris - LDA Linear Discriminant 1 2 t n](https://reader034.vdocuments.mx/reader034/viewer/2022050222/5f6791205ef52e15e842027b/html5/thumbnails/5.jpg)
Problem 3: PCA 95% + LDA
0 500 1000 1500 2000
05
10
USPS - PCA 95% + LDA - 0 & 1
Index
usps.train.0and1.pca95.lda.proj
01
![Page 6: Felix Mbuga Math 251 HW 03 - sjsu.edu€¦ · Problem 1: LDA vs PCA Black - setosa Red - versicolor Green - virginica-10 -5 0 5 4 5 6 7 8 9 Iris - LDA Linear Discriminant 1 2 t n](https://reader034.vdocuments.mx/reader034/viewer/2022050222/5f6791205ef52e15e842027b/html5/thumbnails/6.jpg)
Problem 3: PCA 95% + LDA
-12 -10 -8 -6 -4
-50
510
USPS - PCA 95% - 4 & 9
Principal Component 1
Prin
cipa
l Com
pone
nt 2
49
![Page 7: Felix Mbuga Math 251 HW 03 - sjsu.edu€¦ · Problem 1: LDA vs PCA Black - setosa Red - versicolor Green - virginica-10 -5 0 5 4 5 6 7 8 9 Iris - LDA Linear Discriminant 1 2 t n](https://reader034.vdocuments.mx/reader034/viewer/2022050222/5f6791205ef52e15e842027b/html5/thumbnails/7.jpg)
Problem 3: PCA 95% + LDA
0 200 400 600 800 1000 1200
-8-6
-4-2
02
USPS - PCA 95% + LDA - 4 & 9
Index
usps.train.4and9.pca95.lda.proj
49
![Page 8: Felix Mbuga Math 251 HW 03 - sjsu.edu€¦ · Problem 1: LDA vs PCA Black - setosa Red - versicolor Green - virginica-10 -5 0 5 4 5 6 7 8 9 Iris - LDA Linear Discriminant 1 2 t n](https://reader034.vdocuments.mx/reader034/viewer/2022050222/5f6791205ef52e15e842027b/html5/thumbnails/8.jpg)
Problem 3: PCA 95% + LDA
-14 -12 -10 -8 -6 -4 -2
-10
-8-6
-4-2
02
4
USPS - PCA 95% - 1, 2 & 3
Principal Component 1
Prin
cipa
l Com
pone
nt 2
123
![Page 9: Felix Mbuga Math 251 HW 03 - sjsu.edu€¦ · Problem 1: LDA vs PCA Black - setosa Red - versicolor Green - virginica-10 -5 0 5 4 5 6 7 8 9 Iris - LDA Linear Discriminant 1 2 t n](https://reader034.vdocuments.mx/reader034/viewer/2022050222/5f6791205ef52e15e842027b/html5/thumbnails/9.jpg)
Problem 3: PCA 95% + LDA
-10 -5 0
-4-2
02
46
8
USPS - PCA 95% + LDA - 1, 2 & 3
Linear Discriminant 1
Line
ar D
iscr
imin
ant 2
123
![Page 10: Felix Mbuga Math 251 HW 03 - sjsu.edu€¦ · Problem 1: LDA vs PCA Black - setosa Red - versicolor Green - virginica-10 -5 0 5 4 5 6 7 8 9 Iris - LDA Linear Discriminant 1 2 t n](https://reader034.vdocuments.mx/reader034/viewer/2022050222/5f6791205ef52e15e842027b/html5/thumbnails/10.jpg)
Problem 3: PCA 95% + LDA
-12 -10 -8 -6 -4
-50
5
USPS - PCA 95% - 3, 5 & 8
Principal Component 1
Prin
cipa
l Com
pone
nt 2
358
![Page 11: Felix Mbuga Math 251 HW 03 - sjsu.edu€¦ · Problem 1: LDA vs PCA Black - setosa Red - versicolor Green - virginica-10 -5 0 5 4 5 6 7 8 9 Iris - LDA Linear Discriminant 1 2 t n](https://reader034.vdocuments.mx/reader034/viewer/2022050222/5f6791205ef52e15e842027b/html5/thumbnails/11.jpg)
Problem 3: PCA 95% + LDA
-4 -2 0 2 4 6
-20
24
6
USPS - PCA 95% + LDA - 3, 5 & 8
Linear Discriminant 1
Line
ar D
iscr
imin
ant 2
358
![Page 12: Felix Mbuga Math 251 HW 03 - sjsu.edu€¦ · Problem 1: LDA vs PCA Black - setosa Red - versicolor Green - virginica-10 -5 0 5 4 5 6 7 8 9 Iris - LDA Linear Discriminant 1 2 t n](https://reader034.vdocuments.mx/reader034/viewer/2022050222/5f6791205ef52e15e842027b/html5/thumbnails/12.jpg)
Problem 4: PCA 95% + LDA + kNN vs PCA 95% + kNN
2 4 6 8 10
0.05
0.06
0.07
0.08
0.09
0.10
0.11
k (number of nearest neighbors used)
Mis
clas
sific
atio
n E
rror
Rat
e
PCA 95% + LDAPCA 95%
PCA 95% + LDA has higher misclassification error rate for all k. Lowest misclassification error rates are ~ 5.1% (PCA 95%, k = 1) and ~ 9.3% (PCA 95% + LDA , k = 9)
![Page 13: Felix Mbuga Math 251 HW 03 - sjsu.edu€¦ · Problem 1: LDA vs PCA Black - setosa Red - versicolor Green - virginica-10 -5 0 5 4 5 6 7 8 9 Iris - LDA Linear Discriminant 1 2 t n](https://reader034.vdocuments.mx/reader034/viewer/2022050222/5f6791205ef52e15e842027b/html5/thumbnails/13.jpg)
Problem 5: PCA 95% + LDA + Nearest Local Centroid vs PCA 95% + Nearest Local Centroid
PCA 95% + LDA has higher misclassification error rate for all k. Lowest misclassification error rates are ~ 4.2% (PCA 95%, k = 4) and ~ 8.6% (PCA 95% + LDA , k = 10)
2 4 6 8 10
0.04
0.05
0.06
0.07
0.08
0.09
0.10
0.11
USPS Dataset
Nearest Local Centroid with PCA 95% vs with PCA 95% and LDAk (number of nearest neighbors used)
Mis
clas
sific
atio
n E
rror
Rat
e
PCA 95%PCA 95% + LDA