logistic regression

of 25 /25
Machine learning workshop [email protected] Machine learning introduc7on Logis&c regression Feature selec7on Boos7ng, tree boos7ng See more machine learning post: h>p://dongguo.me

Author: -

Post on 24-Jan-2015

3.774 views

Category:

Technology


1 download

Embed Size (px)

DESCRIPTION

introduce logistic regression, inference with maximize likelihood with gradient descent, compare L1 and L2 regularization, generalized linear model

TRANSCRIPT

  • 1. Machinelearningworkshop [email protected]&cregression Featureselec7on Boos7ng,treeboos7ng Seemoremachinelearningpost:h>p://dongguo.me

2. Overviewofmachinelearning MachineLearningUnsupervised LearningSupervised LearningClassica7on Logis7c regressionSemi-supervised LearningRegression 3. Howtochooseasuitablemodel? Characteris&cNave BayesTrees KNearest neighborLogis&c regressionNeural SVM NetworksComputa7onal scalability331311Interpretability221211Predic7vepower113233Naturalhandling dataofmixedtype131111Robustnessto outliersininput space333311IIP351 4. Whymodelcantperformperfectlyonunseendata Expectedrisk Empiricalrisk Choosefunc7onfamilyforpredic7onfunc7ons Error 5. Logis7cregression 6. Outline Introduc7on Inference Regulariza7on Experiments More Mul7-nominalLR Generalizedlinearmodel Applica7on 7. Logitfunc7onandlogis7cfunc7on Logitfunc7on logis7cfunc7on:Inversedlogit 8. Logis7cregression Predic7onfunc7on 9. Inferencewithmaximizelikelihood(1) Likelihood Inference 10. Inferencewithmaximizelikelihood(2) Inferencecont. Usegradientdescent Stochas7cgradientdescent 11. Regulariza7on Penalizelargeweighttoavoidover`ng L2regulariza7on L1regulariza7on 12. Regulariza7on:Maximumaposteriori MAP 13. L2regulariza7on:GaussianPrior Gaussianprior MAP Gradientdescentstep 14. L1regulariza7on:LaplacePrior Laplaceprior MAP Gradientdescentstep 15. Implementa7on L2LR _weightOfFeatures[fea] += step * (feaValue * error - reguParam * _weightOfFeatures[fea]); L1LR if (_weightOfFeatures[fea] > 0) { _weightOfFeatures[fea] += step * (feaValue * error) - step * reguParam; if (_weightOfFeatures[fea] < 0) _weightOfFeatures[fea] = 0; }else if (_weightOfFeatures[fea] < 0) { _weightOfFeatures[fea] += step * (feaValue * error) + step * reguParam; if (_weightOfFeatures[fea] > 0) _weightOfFeatures[fea] = 0; }else{ _weightOfFeatures[fea] += step * (feaValue * error); } 16. L2VS.L1 L2regulariza7on Almostallweightsarenotequaltozero Notsuitablewhentrainingsamplesarescarce L1regulariza7on Producessparseparametervectors Moresuitablewhenmostfeaturesareirrelevant Couldhandlescarcetrainingsamplesbe>er 17. Experiments Dataset Goal:genderpredic7on Dataset:trainsamples(431k),testsamples(167k) Comparisonalgorithms A:gradientdescentwithL1regulariza7on B:gradientdescentwithL2regulariza7on C:OWL-QN(L-BFGSbasedop7miza7onwithL1regulariza7on) Parameterschoice Regulariza7onvalue Step(learningspeed) Decayra7o Itera7onovercondi7on Maxitera7on7mes(50)||AUCchangep://www.docin.com/p-376254439.html Hulu Demographictarge7ng Otherad-targe7ngproject Customchurnpredic7on More 23. Reference ScalableTrainingofL1-RegularizedLog-Linear ModelsICML07 h>p://www.docin.com/p-376254439.html# Genera-veanddiscrimina-veclassiers:Nave Bayesandlogis-cregressionbyMitchell Featureselec-on,L1vs.L2regulariza-on,and rota-onalinvarianceICML04 24. Recommendedresources MachineLearningopenclassbyAndrewNg //10.20.0.130/TempShare/Machine-LearningOpenClass h>p://www.cnblogs.com/vivounicorn/archive/ 2012/02/24/2365328.html logis7cregressionImplementa7on[link] //10.20.0.130/TempShare/guodong/Logis7cregressionImplementa7on/ Supportbinomialandmul7nominalLRwithL1andL2regulariza7on OWL-QN //10.20.0.130/TempShare/guodong/OWL-QN/ 25. Thanks