![Page 1: Method of Lagrange Multipliersmgormley/courses/606-607-f18/slides606/... · 2019-01-11 · Method of Lagrange Multipliers 1 10-606 Mathematical Foundations for Machine Learning Matt](https://reader034.vdocuments.mx/reader034/viewer/2022050714/5f0a72727e708231d42bad63/html5/thumbnails/1.jpg)
Method of Lagrange Multipliers
1
10-606 Mathematical Foundations for Machine Learning
Matt Gormley
Lecture 10
Oct. 1, 2018
Machine Learning Department
School of Computer Science
Carnegie Mellon University
![Page 2: Method of Lagrange Multipliersmgormley/courses/606-607-f18/slides606/... · 2019-01-11 · Method of Lagrange Multipliers 1 10-606 Mathematical Foundations for Machine Learning Matt](https://reader034.vdocuments.mx/reader034/viewer/2022050714/5f0a72727e708231d42bad63/html5/thumbnails/2.jpg)
Reminders
• Homework 2: Linear Algebra + Calculus
– Out: Tue, Sep. 25
– Due: Tue, Oct. 2 at 11:59pm
• Quiz 1: Linear Algebra
– In-class, Wed, Oct. 3
• Homework 3: Calculus + Probability
– Out: Wed, Oct. 3
– Due: Wed, Oct. 10 at 11:59pm
• Quiz 2: Matrix Calculus + Probability
– In-class, Wed, Oct. 10
3
![Page 3: Method of Lagrange Multipliersmgormley/courses/606-607-f18/slides606/... · 2019-01-11 · Method of Lagrange Multipliers 1 10-606 Mathematical Foundations for Machine Learning Matt](https://reader034.vdocuments.mx/reader034/viewer/2022050714/5f0a72727e708231d42bad63/html5/thumbnails/3.jpg)
Q&A
4
![Page 4: Method of Lagrange Multipliersmgormley/courses/606-607-f18/slides606/... · 2019-01-11 · Method of Lagrange Multipliers 1 10-606 Mathematical Foundations for Machine Learning Matt](https://reader034.vdocuments.mx/reader034/viewer/2022050714/5f0a72727e708231d42bad63/html5/thumbnails/4.jpg)
MATRIX CALCULUS
6
![Page 5: Method of Lagrange Multipliersmgormley/courses/606-607-f18/slides606/... · 2019-01-11 · Method of Lagrange Multipliers 1 10-606 Mathematical Foundations for Machine Learning Matt](https://reader034.vdocuments.mx/reader034/viewer/2022050714/5f0a72727e708231d42bad63/html5/thumbnails/5.jpg)
Method of Lagrange Multipliers
Chalkboard– Motivation: Constrained Optimization
– Method of Lagrange Multipliers
– Extending to multiple constraints
– Extending to inequality constraints
7
![Page 6: Method of Lagrange Multipliersmgormley/courses/606-607-f18/slides606/... · 2019-01-11 · Method of Lagrange Multipliers 1 10-606 Mathematical Foundations for Machine Learning Matt](https://reader034.vdocuments.mx/reader034/viewer/2022050714/5f0a72727e708231d42bad63/html5/thumbnails/6.jpg)
Method of Lagrange Multipliers
8
![Page 7: Method of Lagrange Multipliersmgormley/courses/606-607-f18/slides606/... · 2019-01-11 · Method of Lagrange Multipliers 1 10-606 Mathematical Foundations for Machine Learning Matt](https://reader034.vdocuments.mx/reader034/viewer/2022050714/5f0a72727e708231d42bad63/html5/thumbnails/7.jpg)
Method of Lagrange Multipliers
9
![Page 8: Method of Lagrange Multipliersmgormley/courses/606-607-f18/slides606/... · 2019-01-11 · Method of Lagrange Multipliers 1 10-606 Mathematical Foundations for Machine Learning Matt](https://reader034.vdocuments.mx/reader034/viewer/2022050714/5f0a72727e708231d42bad63/html5/thumbnails/8.jpg)
Method of Lagrange Multipliers
10
![Page 9: Method of Lagrange Multipliersmgormley/courses/606-607-f18/slides606/... · 2019-01-11 · Method of Lagrange Multipliers 1 10-606 Mathematical Foundations for Machine Learning Matt](https://reader034.vdocuments.mx/reader034/viewer/2022050714/5f0a72727e708231d42bad63/html5/thumbnails/9.jpg)
Method of Lagrange Multipliers
11
![Page 10: Method of Lagrange Multipliersmgormley/courses/606-607-f18/slides606/... · 2019-01-11 · Method of Lagrange Multipliers 1 10-606 Mathematical Foundations for Machine Learning Matt](https://reader034.vdocuments.mx/reader034/viewer/2022050714/5f0a72727e708231d42bad63/html5/thumbnails/10.jpg)
Method of Lagrange Multipliers
12
![Page 11: Method of Lagrange Multipliersmgormley/courses/606-607-f18/slides606/... · 2019-01-11 · Method of Lagrange Multipliers 1 10-606 Mathematical Foundations for Machine Learning Matt](https://reader034.vdocuments.mx/reader034/viewer/2022050714/5f0a72727e708231d42bad63/html5/thumbnails/11.jpg)
Method of Lagrange Multipliers
13
Figure from http://tutorial.math.lamar.edu/Classes/CalcIII/LagrangeMultipliers.aspx
![Page 12: Method of Lagrange Multipliersmgormley/courses/606-607-f18/slides606/... · 2019-01-11 · Method of Lagrange Multipliers 1 10-606 Mathematical Foundations for Machine Learning Matt](https://reader034.vdocuments.mx/reader034/viewer/2022050714/5f0a72727e708231d42bad63/html5/thumbnails/12.jpg)
Method of Lagrange Multipliers
14
Figure from http://tutorial.math.lamar.edu/Classes/CalcIII/LagrangeMultipliers.aspx
![Page 13: Method of Lagrange Multipliersmgormley/courses/606-607-f18/slides606/... · 2019-01-11 · Method of Lagrange Multipliers 1 10-606 Mathematical Foundations for Machine Learning Matt](https://reader034.vdocuments.mx/reader034/viewer/2022050714/5f0a72727e708231d42bad63/html5/thumbnails/13.jpg)
DIMENSIONALITY REDUCTION
15
![Page 14: Method of Lagrange Multipliersmgormley/courses/606-607-f18/slides606/... · 2019-01-11 · Method of Lagrange Multipliers 1 10-606 Mathematical Foundations for Machine Learning Matt](https://reader034.vdocuments.mx/reader034/viewer/2022050714/5f0a72727e708231d42bad63/html5/thumbnails/14.jpg)
PCA Outline
• Dimensionality Reduction– High-dimensional data
– Learning (low dimensional) representations
• Principal Component Analysis (PCA)– Examples: 2D and 3D
– Data for PCA
– PCA Definition
– Objective functions for PCA
– PCA, Eigenvectors, and Eigenvalues
– Algorithms for finding Eigenvectors /
Eigenvalues
• PCA Examples– Face Recognition
– Image Compression
16
![Page 15: Method of Lagrange Multipliersmgormley/courses/606-607-f18/slides606/... · 2019-01-11 · Method of Lagrange Multipliers 1 10-606 Mathematical Foundations for Machine Learning Matt](https://reader034.vdocuments.mx/reader034/viewer/2022050714/5f0a72727e708231d42bad63/html5/thumbnails/15.jpg)
High Dimension Data
Examples of high dimensional data:
– High resolution images (millions of pixels)
17
![Page 16: Method of Lagrange Multipliersmgormley/courses/606-607-f18/slides606/... · 2019-01-11 · Method of Lagrange Multipliers 1 10-606 Mathematical Foundations for Machine Learning Matt](https://reader034.vdocuments.mx/reader034/viewer/2022050714/5f0a72727e708231d42bad63/html5/thumbnails/16.jpg)
High Dimension Data
Examples of high dimensional data:
– Multilingual News Stories
(vocabulary of hundreds of thousands of words)
18
![Page 17: Method of Lagrange Multipliersmgormley/courses/606-607-f18/slides606/... · 2019-01-11 · Method of Lagrange Multipliers 1 10-606 Mathematical Foundations for Machine Learning Matt](https://reader034.vdocuments.mx/reader034/viewer/2022050714/5f0a72727e708231d42bad63/html5/thumbnails/17.jpg)
High Dimension Data
Examples of high dimensional data:
– Brain Imaging Data (100s of MBs per scan)
19
Image from https://pixabay.com/en/brain-mrt-magnetic-resonance-imaging-1728449/
Image from (Wehbe et al., 2014)
![Page 18: Method of Lagrange Multipliersmgormley/courses/606-607-f18/slides606/... · 2019-01-11 · Method of Lagrange Multipliers 1 10-606 Mathematical Foundations for Machine Learning Matt](https://reader034.vdocuments.mx/reader034/viewer/2022050714/5f0a72727e708231d42bad63/html5/thumbnails/18.jpg)
High Dimension Data
Examples of high dimensional data:
– Customer Purchase Data
20