+ quadratic programming and duality sivaraman balakrishnan
TRANSCRIPT
+
Quadratic Programming and DualitySivaraman Balakrishnan
+Outline
Quadratic Programs
General Lagrangian Duality
Lagrangian Duality in QPs
+Norm approximation
Problem
Interpretation Geometric – try to find projection of b into ran(A) Statistical – try to find solution to b = Ax + v
v is a measurement noise (choose norm so that v is small in that norm)
Several others
+Examples
-- Least Squares Regression
-- Chebyshev
-- Least Median Regression
More generally can use *any* convex penalty function
+Picture from BV
+Least norm
Perfect measurements
Not enough of them
Heart of something known as compressed sensing
Related to regularized regression in the noisy case
+Smooth signal reconstruction
S(x) is a smoothness penalty
Least squares penalty Smooths out noise and sharp transitions
Total variation (peak to valley intuition) Smooths out noise but preserves sharp transitions
+Euclidean Projection
Very fundamental idea in constrained minimization
Efficient algorithms to project onto many many convex sets (norm balls, special polyhedra etc)
More generally finding minimum distance between polyhedra is a QP
+Quadratic Programming Duality
+General recipe
Form Lagrangian
How to figure out signs?
+Primal & Dual Functions
Primal
Dual
+Primal & Dual Programs
Primal Programs
Constraints are now implicit in the primal
Dual Program
+Lagrangian Properties
Can extract primal and dual problem
Dual problem is always concave Proof
Dual problem is always a lower bound on primal Proof
Strong duality gives complementary slackness Proof
+Some examples of QP duality
Consider the example from class
Lets try to derive dual using Lagrangian
+General PSD QP
Primal
Dual
+SVM – Lagrange Dual
Primal SVM
Dual SVM
Recovering Primal Variables and Complementary Slackness