Foundations & Core in Computer Vision: A System Perspective

Ce Liu

Microsoft Research New England

Vision vs. Learning

• Computer vision: visual application of machine learning?

• Data features algorithms data

• ML: design algorithms given input and output data

• CV: find the best input and output data given available algorithms

Theoretical vs. Experimental

• Theoretical analysis of a visual system– Best & worst cases

– Average performance

• Theoretical analysis is challenging as many visual distributions are hard to model (signal processing: 2nd order processes, machine learning: exponential families)

• Experimental approach: full spectrum of system performance as a function of the amount of data, annotation, number of categories, noise, and other conditions

Quality vs. Speed

• HD videos, billions of images to index• Real time & 90% vs. one hour per frame & 95%?• Mechanism to balance quality and speed in modeling

Automatic vs. semi-automatic

• Common review feedback: parameters are hand-tuned; not clear how to set the parameters

• Vision system user feedback: I don’t know how to tweak parameters!

• Computer-oriented vs. human-oriented representations

• Human-in-the-loop (collaborative) vision– How to optimally use humans (what, which and how

accurate) beyond traditional active learning

– Model design by crowd-sourcing

– Learning by subtraction

Algorithms vs. Sensors

• Two approaches to solving a vision problem– Look at images, design algorithms, experiment, improve…

– Look at cameras, design new/better sensors, …

• Cameras for full-spectrum, high res, low noise, depth, motion, occluding boundary, object, …

• What’s the optimal sensor/device for solving a vision problem?

• What’s the limit of sensors?

Thank you!

Ce Liu

Microsoft Research New England