![Page 1: Nonlinear Systems II: Multiple Variables · Multiple Variables CS 205A: Mathematical Methods for Robotics, Vision, and Graphics Justin Solomon CS 205A: Mathematical Methods Nonlinear](https://reader036.vdocuments.mx/reader036/viewer/2022062402/5fc2616c6f1e3516dc134849/html5/thumbnails/1.jpg)
Multivariable Roots First-Order Approximations Quasi-Newton Automatic Differentiation
Nonlinear Systems II:Multiple Variables
CS 205A:Mathematical Methods for Robotics, Vision, and Graphics
Justin Solomon
CS 205A: Mathematical Methods Nonlinear Systems II: Multiple Variables 1 / 19
![Page 2: Nonlinear Systems II: Multiple Variables · Multiple Variables CS 205A: Mathematical Methods for Robotics, Vision, and Graphics Justin Solomon CS 205A: Mathematical Methods Nonlinear](https://reader036.vdocuments.mx/reader036/viewer/2022062402/5fc2616c6f1e3516dc134849/html5/thumbnails/2.jpg)
Multivariable Roots First-Order Approximations Quasi-Newton Automatic Differentiation
Today’s Root Problems
f : Rn→ Rm
CS 205A: Mathematical Methods Nonlinear Systems II: Multiple Variables 2 / 19
![Page 3: Nonlinear Systems II: Multiple Variables · Multiple Variables CS 205A: Mathematical Methods for Robotics, Vision, and Graphics Justin Solomon CS 205A: Mathematical Methods Nonlinear](https://reader036.vdocuments.mx/reader036/viewer/2022062402/5fc2616c6f1e3516dc134849/html5/thumbnails/3.jpg)
Multivariable Roots First-Order Approximations Quasi-Newton Automatic Differentiation
One Easy Instance
f (~x) = A~x−~b
CS 205A: Mathematical Methods Nonlinear Systems II: Multiple Variables 3 / 19
![Page 4: Nonlinear Systems II: Multiple Variables · Multiple Variables CS 205A: Mathematical Methods for Robotics, Vision, and Graphics Justin Solomon CS 205A: Mathematical Methods Nonlinear](https://reader036.vdocuments.mx/reader036/viewer/2022062402/5fc2616c6f1e3516dc134849/html5/thumbnails/4.jpg)
Multivariable Roots First-Order Approximations Quasi-Newton Automatic Differentiation
Usual Assumption
f : Rn→ Rm
−→ n ≥ m
CS 205A: Mathematical Methods Nonlinear Systems II: Multiple Variables 4 / 19
![Page 5: Nonlinear Systems II: Multiple Variables · Multiple Variables CS 205A: Mathematical Methods for Robotics, Vision, and Graphics Justin Solomon CS 205A: Mathematical Methods Nonlinear](https://reader036.vdocuments.mx/reader036/viewer/2022062402/5fc2616c6f1e3516dc134849/html5/thumbnails/5.jpg)
Multivariable Roots First-Order Approximations Quasi-Newton Automatic Differentiation
Jacobian
(Df )ij ≡∂fi∂xj
How big is Df forf : Rn→ Rm?
CS 205A: Mathematical Methods Nonlinear Systems II: Multiple Variables 5 / 19
![Page 6: Nonlinear Systems II: Multiple Variables · Multiple Variables CS 205A: Mathematical Methods for Robotics, Vision, and Graphics Justin Solomon CS 205A: Mathematical Methods Nonlinear](https://reader036.vdocuments.mx/reader036/viewer/2022062402/5fc2616c6f1e3516dc134849/html5/thumbnails/6.jpg)
Multivariable Roots First-Order Approximations Quasi-Newton Automatic Differentiation
Jacobian
(Df )ij ≡∂fi∂xj
How big is Df forf : Rn→ Rm?
CS 205A: Mathematical Methods Nonlinear Systems II: Multiple Variables 5 / 19
![Page 7: Nonlinear Systems II: Multiple Variables · Multiple Variables CS 205A: Mathematical Methods for Robotics, Vision, and Graphics Justin Solomon CS 205A: Mathematical Methods Nonlinear](https://reader036.vdocuments.mx/reader036/viewer/2022062402/5fc2616c6f1e3516dc134849/html5/thumbnails/7.jpg)
Multivariable Roots First-Order Approximations Quasi-Newton Automatic Differentiation
First-Order Approximation off : Rn → Rn
f (~x) ≈ f (~xk) + Df (~xk) · (~x− ~xk)
Newton’s Method:
~xk+1 = ~xk − [Df (~xk)]−1f (~xk)
Review: Do we explicitly compute [Df (~xk)]−1?
CS 205A: Mathematical Methods Nonlinear Systems II: Multiple Variables 6 / 19
![Page 8: Nonlinear Systems II: Multiple Variables · Multiple Variables CS 205A: Mathematical Methods for Robotics, Vision, and Graphics Justin Solomon CS 205A: Mathematical Methods Nonlinear](https://reader036.vdocuments.mx/reader036/viewer/2022062402/5fc2616c6f1e3516dc134849/html5/thumbnails/8.jpg)
Multivariable Roots First-Order Approximations Quasi-Newton Automatic Differentiation
First-Order Approximation off : Rn → Rn
f (~x) ≈ f (~xk) + Df (~xk) · (~x− ~xk)
Newton’s Method:
~xk+1 = ~xk − [Df (~xk)]−1f (~xk)
Review: Do we explicitly compute [Df (~xk)]−1?
CS 205A: Mathematical Methods Nonlinear Systems II: Multiple Variables 6 / 19
![Page 9: Nonlinear Systems II: Multiple Variables · Multiple Variables CS 205A: Mathematical Methods for Robotics, Vision, and Graphics Justin Solomon CS 205A: Mathematical Methods Nonlinear](https://reader036.vdocuments.mx/reader036/viewer/2022062402/5fc2616c6f1e3516dc134849/html5/thumbnails/9.jpg)
Multivariable Roots First-Order Approximations Quasi-Newton Automatic Differentiation
First-Order Approximation off : Rn → Rn
f (~x) ≈ f (~xk) + Df (~xk) · (~x− ~xk)
Newton’s Method:
~xk+1 = ~xk − [Df (~xk)]−1f (~xk)
Review: Do we explicitly compute [Df (~xk)]−1?
CS 205A: Mathematical Methods Nonlinear Systems II: Multiple Variables 6 / 19
![Page 10: Nonlinear Systems II: Multiple Variables · Multiple Variables CS 205A: Mathematical Methods for Robotics, Vision, and Graphics Justin Solomon CS 205A: Mathematical Methods Nonlinear](https://reader036.vdocuments.mx/reader036/viewer/2022062402/5fc2616c6f1e3516dc134849/html5/thumbnails/10.jpg)
Multivariable Roots First-Order Approximations Quasi-Newton Automatic Differentiation
Convergence Sketch
1. ~xk+1 = g(~xk) converges when the
maximum-magnitude eigenvalue of Dg is less
than 1
2. Extend observations about (quadratic)
convergence in multiple dimensions
CS 205A: Mathematical Methods Nonlinear Systems II: Multiple Variables 7 / 19
![Page 11: Nonlinear Systems II: Multiple Variables · Multiple Variables CS 205A: Mathematical Methods for Robotics, Vision, and Graphics Justin Solomon CS 205A: Mathematical Methods Nonlinear](https://reader036.vdocuments.mx/reader036/viewer/2022062402/5fc2616c6f1e3516dc134849/html5/thumbnails/11.jpg)
Multivariable Roots First-Order Approximations Quasi-Newton Automatic Differentiation
Two Problems
1. Differentiation is hard
2. Df (~xk) changes everyiteration
CS 205A: Mathematical Methods Nonlinear Systems II: Multiple Variables 8 / 19
![Page 12: Nonlinear Systems II: Multiple Variables · Multiple Variables CS 205A: Mathematical Methods for Robotics, Vision, and Graphics Justin Solomon CS 205A: Mathematical Methods Nonlinear](https://reader036.vdocuments.mx/reader036/viewer/2022062402/5fc2616c6f1e3516dc134849/html5/thumbnails/12.jpg)
Multivariable Roots First-Order Approximations Quasi-Newton Automatic Differentiation
Two Problems
1. Differentiation is hard
2. Df (~xk) changes everyiteration
CS 205A: Mathematical Methods Nonlinear Systems II: Multiple Variables 8 / 19
![Page 13: Nonlinear Systems II: Multiple Variables · Multiple Variables CS 205A: Mathematical Methods for Robotics, Vision, and Graphics Justin Solomon CS 205A: Mathematical Methods Nonlinear](https://reader036.vdocuments.mx/reader036/viewer/2022062402/5fc2616c6f1e3516dc134849/html5/thumbnails/13.jpg)
Multivariable Roots First-Order Approximations Quasi-Newton Automatic Differentiation
Extend Secant Method?
Direct extensions arenot obvious:
Not enough data points!
CS 205A: Mathematical Methods Nonlinear Systems II: Multiple Variables 9 / 19
![Page 14: Nonlinear Systems II: Multiple Variables · Multiple Variables CS 205A: Mathematical Methods for Robotics, Vision, and Graphics Justin Solomon CS 205A: Mathematical Methods Nonlinear](https://reader036.vdocuments.mx/reader036/viewer/2022062402/5fc2616c6f1e3516dc134849/html5/thumbnails/14.jpg)
Multivariable Roots First-Order Approximations Quasi-Newton Automatic Differentiation
Extend Secant Method?
Direct extensions arenot obvious:
Not enough data points!
CS 205A: Mathematical Methods Nonlinear Systems II: Multiple Variables 9 / 19
![Page 15: Nonlinear Systems II: Multiple Variables · Multiple Variables CS 205A: Mathematical Methods for Robotics, Vision, and Graphics Justin Solomon CS 205A: Mathematical Methods Nonlinear](https://reader036.vdocuments.mx/reader036/viewer/2022062402/5fc2616c6f1e3516dc134849/html5/thumbnails/15.jpg)
Multivariable Roots First-Order Approximations Quasi-Newton Automatic Differentiation
Observation: Directional Derivative
D~vf = Df · ~v
CS 205A: Mathematical Methods Nonlinear Systems II: Multiple Variables 10 / 19
![Page 16: Nonlinear Systems II: Multiple Variables · Multiple Variables CS 205A: Mathematical Methods for Robotics, Vision, and Graphics Justin Solomon CS 205A: Mathematical Methods Nonlinear](https://reader036.vdocuments.mx/reader036/viewer/2022062402/5fc2616c6f1e3516dc134849/html5/thumbnails/16.jpg)
Multivariable Roots First-Order Approximations Quasi-Newton Automatic Differentiation
Secant-Like Approximation
J · (~xk − ~xk−1)
≈ f (~xk)− f (~xk−1)
where J ≈ Df (~xk)
“Broyden’s Method”
CS 205A: Mathematical Methods Nonlinear Systems II: Multiple Variables 11 / 19
![Page 17: Nonlinear Systems II: Multiple Variables · Multiple Variables CS 205A: Mathematical Methods for Robotics, Vision, and Graphics Justin Solomon CS 205A: Mathematical Methods Nonlinear](https://reader036.vdocuments.mx/reader036/viewer/2022062402/5fc2616c6f1e3516dc134849/html5/thumbnails/17.jpg)
Multivariable Roots First-Order Approximations Quasi-Newton Automatic Differentiation
Secant-Like Approximation
J · (~xk − ~xk−1)
≈ f (~xk)− f (~xk−1)
where J ≈ Df (~xk)
“Broyden’s Method”CS 205A: Mathematical Methods Nonlinear Systems II: Multiple Variables 11 / 19
![Page 18: Nonlinear Systems II: Multiple Variables · Multiple Variables CS 205A: Mathematical Methods for Robotics, Vision, and Graphics Justin Solomon CS 205A: Mathematical Methods Nonlinear](https://reader036.vdocuments.mx/reader036/viewer/2022062402/5fc2616c6f1e3516dc134849/html5/thumbnails/18.jpg)
Multivariable Roots First-Order Approximations Quasi-Newton Automatic Differentiation
Broyden’s Method: Outline
I Maintain current iterate ~xk and
approximation Jk of Jacobian near ~xkI Update ~xk using Newton-like step
I Update Jk using secant-like formula
CS 205A: Mathematical Methods Nonlinear Systems II: Multiple Variables 12 / 19
![Page 19: Nonlinear Systems II: Multiple Variables · Multiple Variables CS 205A: Mathematical Methods for Robotics, Vision, and Graphics Justin Solomon CS 205A: Mathematical Methods Nonlinear](https://reader036.vdocuments.mx/reader036/viewer/2022062402/5fc2616c6f1e3516dc134849/html5/thumbnails/19.jpg)
Multivariable Roots First-Order Approximations Quasi-Newton Automatic Differentiation
Deriving the Broyden Step
minimizeJk ‖Jk − Jk−1‖2Frosuch that Jk · (~xk − ~xk−1) = f (~xk)− f (~xk−1)
Jk = Jk−1 +(f (~xk)− f (~xk−1)− Jk ·∆~x)
‖~xk − ~xk−1‖22(∆~x)>
CS 205A: Mathematical Methods Nonlinear Systems II: Multiple Variables 13 / 19
![Page 20: Nonlinear Systems II: Multiple Variables · Multiple Variables CS 205A: Mathematical Methods for Robotics, Vision, and Graphics Justin Solomon CS 205A: Mathematical Methods Nonlinear](https://reader036.vdocuments.mx/reader036/viewer/2022062402/5fc2616c6f1e3516dc134849/html5/thumbnails/20.jpg)
Multivariable Roots First-Order Approximations Quasi-Newton Automatic Differentiation
Deriving the Broyden Step
minimizeJk ‖Jk − Jk−1‖2Frosuch that Jk · (~xk − ~xk−1) = f (~xk)− f (~xk−1)
Jk = Jk−1 +(f (~xk)− f (~xk−1)− Jk ·∆~x)
‖~xk − ~xk−1‖22(∆~x)>
CS 205A: Mathematical Methods Nonlinear Systems II: Multiple Variables 13 / 19
![Page 21: Nonlinear Systems II: Multiple Variables · Multiple Variables CS 205A: Mathematical Methods for Robotics, Vision, and Graphics Justin Solomon CS 205A: Mathematical Methods Nonlinear](https://reader036.vdocuments.mx/reader036/viewer/2022062402/5fc2616c6f1e3516dc134849/html5/thumbnails/21.jpg)
Multivariable Roots First-Order Approximations Quasi-Newton Automatic Differentiation
The Newton Step
~xk+1 = ~xk − J−1k f (~xk)
CS 205A: Mathematical Methods Nonlinear Systems II: Multiple Variables 14 / 19
![Page 22: Nonlinear Systems II: Multiple Variables · Multiple Variables CS 205A: Mathematical Methods for Robotics, Vision, and Graphics Justin Solomon CS 205A: Mathematical Methods Nonlinear](https://reader036.vdocuments.mx/reader036/viewer/2022062402/5fc2616c6f1e3516dc134849/html5/thumbnails/22.jpg)
Multivariable Roots First-Order Approximations Quasi-Newton Automatic Differentiation
Implementation Details
I How to initialize J0?
Common choice: J0 ≡ I
I Long “memory”—bad approximation?
Limited-memory methods
I Still have to invert Jk in each step!
CS 205A: Mathematical Methods Nonlinear Systems II: Multiple Variables 15 / 19
![Page 23: Nonlinear Systems II: Multiple Variables · Multiple Variables CS 205A: Mathematical Methods for Robotics, Vision, and Graphics Justin Solomon CS 205A: Mathematical Methods Nonlinear](https://reader036.vdocuments.mx/reader036/viewer/2022062402/5fc2616c6f1e3516dc134849/html5/thumbnails/23.jpg)
Multivariable Roots First-Order Approximations Quasi-Newton Automatic Differentiation
Implementation Details
I How to initialize J0?
Common choice: J0 ≡ I
I Long “memory”—bad approximation?
Limited-memory methods
I Still have to invert Jk in each step!
CS 205A: Mathematical Methods Nonlinear Systems II: Multiple Variables 15 / 19
![Page 24: Nonlinear Systems II: Multiple Variables · Multiple Variables CS 205A: Mathematical Methods for Robotics, Vision, and Graphics Justin Solomon CS 205A: Mathematical Methods Nonlinear](https://reader036.vdocuments.mx/reader036/viewer/2022062402/5fc2616c6f1e3516dc134849/html5/thumbnails/24.jpg)
Multivariable Roots First-Order Approximations Quasi-Newton Automatic Differentiation
Implementation Details
I How to initialize J0?
Common choice: J0 ≡ I
I Long “memory”—bad approximation?
Limited-memory methods
I Still have to invert Jk in each step!
CS 205A: Mathematical Methods Nonlinear Systems II: Multiple Variables 15 / 19
![Page 25: Nonlinear Systems II: Multiple Variables · Multiple Variables CS 205A: Mathematical Methods for Robotics, Vision, and Graphics Justin Solomon CS 205A: Mathematical Methods Nonlinear](https://reader036.vdocuments.mx/reader036/viewer/2022062402/5fc2616c6f1e3516dc134849/html5/thumbnails/25.jpg)
Multivariable Roots First-Order Approximations Quasi-Newton Automatic Differentiation
Implementation Details
I How to initialize J0?
Common choice: J0 ≡ I
I Long “memory”—bad approximation?
Limited-memory methods
I Still have to invert Jk in each step!
CS 205A: Mathematical Methods Nonlinear Systems II: Multiple Variables 15 / 19
![Page 26: Nonlinear Systems II: Multiple Variables · Multiple Variables CS 205A: Mathematical Methods for Robotics, Vision, and Graphics Justin Solomon CS 205A: Mathematical Methods Nonlinear](https://reader036.vdocuments.mx/reader036/viewer/2022062402/5fc2616c6f1e3516dc134849/html5/thumbnails/26.jpg)
Multivariable Roots First-Order Approximations Quasi-Newton Automatic Differentiation
Implementation Details
I How to initialize J0?
Common choice: J0 ≡ I
I Long “memory”—bad approximation?
Limited-memory methods
I Still have to invert Jk in each step!
CS 205A: Mathematical Methods Nonlinear Systems II: Multiple Variables 15 / 19
![Page 27: Nonlinear Systems II: Multiple Variables · Multiple Variables CS 205A: Mathematical Methods for Robotics, Vision, and Graphics Justin Solomon CS 205A: Mathematical Methods Nonlinear](https://reader036.vdocuments.mx/reader036/viewer/2022062402/5fc2616c6f1e3516dc134849/html5/thumbnails/27.jpg)
Multivariable Roots First-Order Approximations Quasi-Newton Automatic Differentiation
Revisiting the Broyden Step
Jk = Jk−1 +(f (~xk)− f (~xk−1)− Jk ·∆~x)
‖~xk − ~xk−1‖22(∆~x)>
Simpler form:
Jk = Jk−1 + ~uk~v>k
CS 205A: Mathematical Methods Nonlinear Systems II: Multiple Variables 16 / 19
![Page 28: Nonlinear Systems II: Multiple Variables · Multiple Variables CS 205A: Mathematical Methods for Robotics, Vision, and Graphics Justin Solomon CS 205A: Mathematical Methods Nonlinear](https://reader036.vdocuments.mx/reader036/viewer/2022062402/5fc2616c6f1e3516dc134849/html5/thumbnails/28.jpg)
Multivariable Roots First-Order Approximations Quasi-Newton Automatic Differentiation
Revisiting the Broyden Step
Jk = Jk−1 +(f (~xk)− f (~xk−1)− Jk ·∆~x)
‖~xk − ~xk−1‖22(∆~x)>
Simpler form:
Jk = Jk−1 + ~uk~v>k
CS 205A: Mathematical Methods Nonlinear Systems II: Multiple Variables 16 / 19
![Page 29: Nonlinear Systems II: Multiple Variables · Multiple Variables CS 205A: Mathematical Methods for Robotics, Vision, and Graphics Justin Solomon CS 205A: Mathematical Methods Nonlinear](https://reader036.vdocuments.mx/reader036/viewer/2022062402/5fc2616c6f1e3516dc134849/html5/thumbnails/29.jpg)
Multivariable Roots First-Order Approximations Quasi-Newton Automatic Differentiation
Sherman-Morrison Formula
(A + ~u~v>)−1 = A−1 − A−1~u~v>A−1
1 + ~v>A−1~u
Homework (if I remember): Check this
#sorrynotsorry
CS 205A: Mathematical Methods Nonlinear Systems II: Multiple Variables 17 / 19
![Page 30: Nonlinear Systems II: Multiple Variables · Multiple Variables CS 205A: Mathematical Methods for Robotics, Vision, and Graphics Justin Solomon CS 205A: Mathematical Methods Nonlinear](https://reader036.vdocuments.mx/reader036/viewer/2022062402/5fc2616c6f1e3516dc134849/html5/thumbnails/30.jpg)
Multivariable Roots First-Order Approximations Quasi-Newton Automatic Differentiation
Broyden Without Inversion
I Start with a J0 for which you know J−10
(e.g. identity)
I Update J−10 directly!
Question: Limited-memory strategy?
Question: Large-scale strategy?
CS 205A: Mathematical Methods Nonlinear Systems II: Multiple Variables 18 / 19
![Page 31: Nonlinear Systems II: Multiple Variables · Multiple Variables CS 205A: Mathematical Methods for Robotics, Vision, and Graphics Justin Solomon CS 205A: Mathematical Methods Nonlinear](https://reader036.vdocuments.mx/reader036/viewer/2022062402/5fc2616c6f1e3516dc134849/html5/thumbnails/31.jpg)
Multivariable Roots First-Order Approximations Quasi-Newton Automatic Differentiation
Broyden Without Inversion
I Start with a J0 for which you know J−10
(e.g. identity)
I Update J−10 directly!
Question: Limited-memory strategy?
Question: Large-scale strategy?
CS 205A: Mathematical Methods Nonlinear Systems II: Multiple Variables 18 / 19
![Page 32: Nonlinear Systems II: Multiple Variables · Multiple Variables CS 205A: Mathematical Methods for Robotics, Vision, and Graphics Justin Solomon CS 205A: Mathematical Methods Nonlinear](https://reader036.vdocuments.mx/reader036/viewer/2022062402/5fc2616c6f1e3516dc134849/html5/thumbnails/32.jpg)
Multivariable Roots First-Order Approximations Quasi-Newton Automatic Differentiation
Broyden Without Inversion
I Start with a J0 for which you know J−10
(e.g. identity)
I Update J−10 directly!
Question: Limited-memory strategy?
Question: Large-scale strategy?
CS 205A: Mathematical Methods Nonlinear Systems II: Multiple Variables 18 / 19
![Page 33: Nonlinear Systems II: Multiple Variables · Multiple Variables CS 205A: Mathematical Methods for Robotics, Vision, and Graphics Justin Solomon CS 205A: Mathematical Methods Nonlinear](https://reader036.vdocuments.mx/reader036/viewer/2022062402/5fc2616c6f1e3516dc134849/html5/thumbnails/33.jpg)
Multivariable Roots First-Order Approximations Quasi-Newton Automatic Differentiation
Automatic Differentiation
(x,
dx
dt
);
(y,
dy
dt
)7→(
x + y,dx
dt+
dy
dt
)(xy,
dx
dty + x
dy
dt
) (x
y,ydxdt + xdy
dt
y2
)· · ·
Next
CS 205A: Mathematical Methods Nonlinear Systems II: Multiple Variables 19 / 19