Regression

[slides prises du cours cs294-10 UC Berkeley (2006 / 2009)]http://www.cs.berkeley.edu/~jordan/courses/294-fall09/lectures/regression/

Classification (reminder)

X ! YAnything:

• continuous (, d, …)

• discrete ({0,1}, {1,…k}, …)

• structured (tree, string, …)

• …

• discrete:

– {0,1} binary

– {1,…k} multi-class

– tree, etc. structured

Classification (reminder)

XAnything:

• continuous (, d, …)

• discrete ({0,1}, {1,…k}, …)

• structured (tree, string, …)

• …

Classification (reminder)

XAnything:

• continuous (, d, …)

• discrete ({0,1}, {1,…k}, …)

• structured (tree, string, …)

• …

Perceptron

Logistic Regression

Support Vector Machine

Decision TreeRandom Forest

Kernel trick

Regression

X ! Y• continuous:– , d

Anything:

• continuous (, d, …)

• discrete ({0,1}, {1,…k}, …)

• structured (tree, string, …)

• …

1

Overfitting in regression...

degree 15

overfitting!

Between two models / hypotheses which explain as well the data, choose the simplest one

In Machine Learning:◦ we usually need to tradeoff between

training error model complexity

◦ can be formalized precisely in statistics (bias-variance tradeoff, etc.)

Occam’s razor principle:

training error model complexity

Logiciels:◦ Weka (Java): http://www.cs.waikato.ac.nz/ml/weka/

◦ RapidMiner (nicer GUI?): http://rapid-i.com/

◦ SciKit Learn (Python): http://scikit-learn.org

Livres:◦ Pattern Classification (Duda, Hart & Stork)◦ Pattern Recognition and Machine Learning

(Bishop)◦ Data Mining (Witten, Frank & Hall)◦ The Elements of Statistical Learning (Hastie, Tibshirani, Friedman)

Programmer en python:◦ cours cs188 de Dan Klein à Berkeley: http://inst.eecs.berkeley.edu/~cs188/fa10/lectures.html

Ressources

Kernel Regression

0 2 4 6 8 10 12 14 16 18 20-10

-5

0

5

10

15Kernel regression (sigma=1)