![Page 1: ECE#5424:#Introduction#to# Machine#Learningf16ece5424/slides/L8_regression.pptx.pdfECE#5424:#Introduction#to# Machine#Learning Stefan#Lee Virginia#Tech Topics:# – Regression Readings:#Barber#17.1,#17.2](https://reader033.vdocuments.mx/reader033/viewer/2022052719/5f0827b57e708231d4209cbd/html5/thumbnails/1.jpg)
ECE 5424: Introduction to Machine Learning
Stefan LeeVirginia Tech
Topics: – Regression
Readings: Barber 17.1, 17.2
![Page 2: ECE#5424:#Introduction#to# Machine#Learningf16ece5424/slides/L8_regression.pptx.pdfECE#5424:#Introduction#to# Machine#Learning Stefan#Lee Virginia#Tech Topics:# – Regression Readings:#Barber#17.1,#17.2](https://reader033.vdocuments.mx/reader033/viewer/2022052719/5f0827b57e708231d4209cbd/html5/thumbnails/2.jpg)
Administrativia• HW1
– 39 Submissions– 58 Students Enrolled– 19 MIA (?????)
• Project Proposal– Due: 09/21, 11:55 pm (LESS THAN A WEEK!)– <= 2pages, NIPS format
• HW2 – Out today– Due on Wednesday 09/28, 11:55pm– Please please please please please start early– Implement Linear Regression, Naïve Bayes, Logistic Regression
(C) Dhruv Batra 2
![Page 3: ECE#5424:#Introduction#to# Machine#Learningf16ece5424/slides/L8_regression.pptx.pdfECE#5424:#Introduction#to# Machine#Learning Stefan#Lee Virginia#Tech Topics:# – Regression Readings:#Barber#17.1,#17.2](https://reader033.vdocuments.mx/reader033/viewer/2022052719/5f0827b57e708231d4209cbd/html5/thumbnails/3.jpg)
Recap of last time
(C) Dhruv Batra 3
![Page 4: ECE#5424:#Introduction#to# Machine#Learningf16ece5424/slides/L8_regression.pptx.pdfECE#5424:#Introduction#to# Machine#Learning Stefan#Lee Virginia#Tech Topics:# – Regression Readings:#Barber#17.1,#17.2](https://reader033.vdocuments.mx/reader033/viewer/2022052719/5f0827b57e708231d4209cbd/html5/thumbnails/4.jpg)
Learning a Gaussian• Collect a bunch of data
– Hopefully, i.i.d. samples– e.g., exam scores
• Learn parameters– Mean– Variance
(C) Dhruv Batra 4
![Page 5: ECE#5424:#Introduction#to# Machine#Learningf16ece5424/slides/L8_regression.pptx.pdfECE#5424:#Introduction#to# Machine#Learning Stefan#Lee Virginia#Tech Topics:# – Regression Readings:#Barber#17.1,#17.2](https://reader033.vdocuments.mx/reader033/viewer/2022052719/5f0827b57e708231d4209cbd/html5/thumbnails/5.jpg)
MLE for Gaussian• Prob. of i.i.d. samples D={x1,…,xN}:
• Log-likelihood of data:
Slide Credit: Carlos Guestrin(C) Dhruv Batra 5
![Page 6: ECE#5424:#Introduction#to# Machine#Learningf16ece5424/slides/L8_regression.pptx.pdfECE#5424:#Introduction#to# Machine#Learning Stefan#Lee Virginia#Tech Topics:# – Regression Readings:#Barber#17.1,#17.2](https://reader033.vdocuments.mx/reader033/viewer/2022052719/5f0827b57e708231d4209cbd/html5/thumbnails/6.jpg)
Your second learning algorithm:MLE for mean of a Gaussian
• What’s MLE for mean?
Slide Credit: Carlos Guestrin(C) Dhruv Batra 6
![Page 7: ECE#5424:#Introduction#to# Machine#Learningf16ece5424/slides/L8_regression.pptx.pdfECE#5424:#Introduction#to# Machine#Learning Stefan#Lee Virginia#Tech Topics:# – Regression Readings:#Barber#17.1,#17.2](https://reader033.vdocuments.mx/reader033/viewer/2022052719/5f0827b57e708231d4209cbd/html5/thumbnails/7.jpg)
Learning Gaussian parameters
• MLE:
(C) Dhruv Batra 7
![Page 8: ECE#5424:#Introduction#to# Machine#Learningf16ece5424/slides/L8_regression.pptx.pdfECE#5424:#Introduction#to# Machine#Learning Stefan#Lee Virginia#Tech Topics:# – Regression Readings:#Barber#17.1,#17.2](https://reader033.vdocuments.mx/reader033/viewer/2022052719/5f0827b57e708231d4209cbd/html5/thumbnails/8.jpg)
• Conjugate priors– Mean: Gaussian prior– Variance: Inverse Gamma or Wishart Distribution
• Prior for mean:
Slide Credit: Carlos Guestrin(C) Dhruv Batra 8
Bayesian learning of Gaussian parameters
![Page 9: ECE#5424:#Introduction#to# Machine#Learningf16ece5424/slides/L8_regression.pptx.pdfECE#5424:#Introduction#to# Machine#Learning Stefan#Lee Virginia#Tech Topics:# – Regression Readings:#Barber#17.1,#17.2](https://reader033.vdocuments.mx/reader033/viewer/2022052719/5f0827b57e708231d4209cbd/html5/thumbnails/9.jpg)
MAP for mean of Gaussian
Slide Credit: Carlos Guestrin(C) Dhruv Batra 9
![Page 10: ECE#5424:#Introduction#to# Machine#Learningf16ece5424/slides/L8_regression.pptx.pdfECE#5424:#Introduction#to# Machine#Learning Stefan#Lee Virginia#Tech Topics:# – Regression Readings:#Barber#17.1,#17.2](https://reader033.vdocuments.mx/reader033/viewer/2022052719/5f0827b57e708231d4209cbd/html5/thumbnails/10.jpg)
New Topic: Regression
(C) Dhruv Batra 10
![Page 11: ECE#5424:#Introduction#to# Machine#Learningf16ece5424/slides/L8_regression.pptx.pdfECE#5424:#Introduction#to# Machine#Learning Stefan#Lee Virginia#Tech Topics:# – Regression Readings:#Barber#17.1,#17.2](https://reader033.vdocuments.mx/reader033/viewer/2022052719/5f0827b57e708231d4209cbd/html5/thumbnails/11.jpg)
1-NN for Regression• Often bumpy (overfits)
(C) Dhruv Batra 11Figure Credit: Andrew Moore
![Page 12: ECE#5424:#Introduction#to# Machine#Learningf16ece5424/slides/L8_regression.pptx.pdfECE#5424:#Introduction#to# Machine#Learning Stefan#Lee Virginia#Tech Topics:# – Regression Readings:#Barber#17.1,#17.2](https://reader033.vdocuments.mx/reader033/viewer/2022052719/5f0827b57e708231d4209cbd/html5/thumbnails/12.jpg)
(C) Dhruv Batra 12Slide Credit: Greg Shakhnarovich
![Page 13: ECE#5424:#Introduction#to# Machine#Learningf16ece5424/slides/L8_regression.pptx.pdfECE#5424:#Introduction#to# Machine#Learning Stefan#Lee Virginia#Tech Topics:# – Regression Readings:#Barber#17.1,#17.2](https://reader033.vdocuments.mx/reader033/viewer/2022052719/5f0827b57e708231d4209cbd/html5/thumbnails/13.jpg)
(C) Dhruv Batra 13Slide Credit: Greg Shakhnarovich
![Page 14: ECE#5424:#Introduction#to# Machine#Learningf16ece5424/slides/L8_regression.pptx.pdfECE#5424:#Introduction#to# Machine#Learning Stefan#Lee Virginia#Tech Topics:# – Regression Readings:#Barber#17.1,#17.2](https://reader033.vdocuments.mx/reader033/viewer/2022052719/5f0827b57e708231d4209cbd/html5/thumbnails/14.jpg)
(C) Dhruv Batra 14Slide Credit: Greg Shakhnarovich
![Page 15: ECE#5424:#Introduction#to# Machine#Learningf16ece5424/slides/L8_regression.pptx.pdfECE#5424:#Introduction#to# Machine#Learning Stefan#Lee Virginia#Tech Topics:# – Regression Readings:#Barber#17.1,#17.2](https://reader033.vdocuments.mx/reader033/viewer/2022052719/5f0827b57e708231d4209cbd/html5/thumbnails/15.jpg)
Linear Regression• Demo
– http://hspm.sph.sc.edu/courses/J716/demos/LeastSquares/LeastSquaresDemo.html
(C) Dhruv Batra 15
![Page 16: ECE#5424:#Introduction#to# Machine#Learningf16ece5424/slides/L8_regression.pptx.pdfECE#5424:#Introduction#to# Machine#Learning Stefan#Lee Virginia#Tech Topics:# – Regression Readings:#Barber#17.1,#17.2](https://reader033.vdocuments.mx/reader033/viewer/2022052719/5f0827b57e708231d4209cbd/html5/thumbnails/16.jpg)
(C) Dhruv Batra 16Slide Credit: Greg Shakhnarovich
![Page 17: ECE#5424:#Introduction#to# Machine#Learningf16ece5424/slides/L8_regression.pptx.pdfECE#5424:#Introduction#to# Machine#Learning Stefan#Lee Virginia#Tech Topics:# – Regression Readings:#Barber#17.1,#17.2](https://reader033.vdocuments.mx/reader033/viewer/2022052719/5f0827b57e708231d4209cbd/html5/thumbnails/17.jpg)
Plan for Today• Regression
– Linear Regression Recap• Some matrix calculus review
– Connections with Gaussians• The outlier problem L
– Robust Least Squares
(C) Dhruv Batra 17
![Page 18: ECE#5424:#Introduction#to# Machine#Learningf16ece5424/slides/L8_regression.pptx.pdfECE#5424:#Introduction#to# Machine#Learning Stefan#Lee Virginia#Tech Topics:# – Regression Readings:#Barber#17.1,#17.2](https://reader033.vdocuments.mx/reader033/viewer/2022052719/5f0827b57e708231d4209cbd/html5/thumbnails/18.jpg)
(C) Dhruv Batra 18Slide Credit: Greg Shakhnarovich
![Page 19: ECE#5424:#Introduction#to# Machine#Learningf16ece5424/slides/L8_regression.pptx.pdfECE#5424:#Introduction#to# Machine#Learning Stefan#Lee Virginia#Tech Topics:# – Regression Readings:#Barber#17.1,#17.2](https://reader033.vdocuments.mx/reader033/viewer/2022052719/5f0827b57e708231d4209cbd/html5/thumbnails/19.jpg)
(C) Dhruv Batra 19Slide Credit: Greg Shakhnarovich
![Page 20: ECE#5424:#Introduction#to# Machine#Learningf16ece5424/slides/L8_regression.pptx.pdfECE#5424:#Introduction#to# Machine#Learning Stefan#Lee Virginia#Tech Topics:# – Regression Readings:#Barber#17.1,#17.2](https://reader033.vdocuments.mx/reader033/viewer/2022052719/5f0827b57e708231d4209cbd/html5/thumbnails/20.jpg)
(C) Dhruv Batra 20Slide Credit: Greg Shakhnarovich
![Page 21: ECE#5424:#Introduction#to# Machine#Learningf16ece5424/slides/L8_regression.pptx.pdfECE#5424:#Introduction#to# Machine#Learning Stefan#Lee Virginia#Tech Topics:# – Regression Readings:#Barber#17.1,#17.2](https://reader033.vdocuments.mx/reader033/viewer/2022052719/5f0827b57e708231d4209cbd/html5/thumbnails/21.jpg)
(C) Dhruv Batra 21Slide Credit: Greg Shakhnarovich
![Page 22: ECE#5424:#Introduction#to# Machine#Learningf16ece5424/slides/L8_regression.pptx.pdfECE#5424:#Introduction#to# Machine#Learning Stefan#Lee Virginia#Tech Topics:# – Regression Readings:#Barber#17.1,#17.2](https://reader033.vdocuments.mx/reader033/viewer/2022052719/5f0827b57e708231d4209cbd/html5/thumbnails/22.jpg)
(C) Dhruv Batra 22Slide Credit: Greg Shakhnarovich
![Page 23: ECE#5424:#Introduction#to# Machine#Learningf16ece5424/slides/L8_regression.pptx.pdfECE#5424:#Introduction#to# Machine#Learning Stefan#Lee Virginia#Tech Topics:# – Regression Readings:#Barber#17.1,#17.2](https://reader033.vdocuments.mx/reader033/viewer/2022052719/5f0827b57e708231d4209cbd/html5/thumbnails/23.jpg)
(C) Dhruv Batra 23Slide Credit: Greg Shakhnarovich
![Page 24: ECE#5424:#Introduction#to# Machine#Learningf16ece5424/slides/L8_regression.pptx.pdfECE#5424:#Introduction#to# Machine#Learning Stefan#Lee Virginia#Tech Topics:# – Regression Readings:#Barber#17.1,#17.2](https://reader033.vdocuments.mx/reader033/viewer/2022052719/5f0827b57e708231d4209cbd/html5/thumbnails/24.jpg)
• Why sum squared error???• Gaussians, Watson, Gaussians…
But, why?
24(C) Dhruv Batra
![Page 25: ECE#5424:#Introduction#to# Machine#Learningf16ece5424/slides/L8_regression.pptx.pdfECE#5424:#Introduction#to# Machine#Learning Stefan#Lee Virginia#Tech Topics:# – Regression Readings:#Barber#17.1,#17.2](https://reader033.vdocuments.mx/reader033/viewer/2022052719/5f0827b57e708231d4209cbd/html5/thumbnails/25.jpg)
(C) Dhruv Batra 25Slide Credit: Greg Shakhnarovich
![Page 26: ECE#5424:#Introduction#to# Machine#Learningf16ece5424/slides/L8_regression.pptx.pdfECE#5424:#Introduction#to# Machine#Learning Stefan#Lee Virginia#Tech Topics:# – Regression Readings:#Barber#17.1,#17.2](https://reader033.vdocuments.mx/reader033/viewer/2022052719/5f0827b57e708231d4209cbd/html5/thumbnails/26.jpg)
MLE Under Gaussian Model• On board
(C) Dhruv Batra 26
![Page 27: ECE#5424:#Introduction#to# Machine#Learningf16ece5424/slides/L8_regression.pptx.pdfECE#5424:#Introduction#to# Machine#Learning Stefan#Lee Virginia#Tech Topics:# – Regression Readings:#Barber#17.1,#17.2](https://reader033.vdocuments.mx/reader033/viewer/2022052719/5f0827b57e708231d4209cbd/html5/thumbnails/27.jpg)
Is OLS Robust?• Demo
– http://www.calpoly.edu/~srein/StatDemo/All.html
• Bad things happen when the data does not comefrom your model!
• How do we fix this?
(C) Dhruv Batra 27
![Page 28: ECE#5424:#Introduction#to# Machine#Learningf16ece5424/slides/L8_regression.pptx.pdfECE#5424:#Introduction#to# Machine#Learning Stefan#Lee Virginia#Tech Topics:# – Regression Readings:#Barber#17.1,#17.2](https://reader033.vdocuments.mx/reader033/viewer/2022052719/5f0827b57e708231d4209cbd/html5/thumbnails/28.jpg)
Robust Linear Regression• y ~ Lap(w’x, b)
• On board
(C) Dhruv Batra 28
0 0.2 0.4 0.6 0.8 1−6
−5
−4
−3
−2
−1
0
1
2
3
4Linear data with noise and outliers
least squareslaplace
−3 −2 −1 0 1 2 3−0.5
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
5
L2
L1
huber