ensemble methods. “no free lunch theorem” wolpert and macready 1995
TRANSCRIPT
![Page 1: Ensemble Methods. “No free lunch theorem” Wolpert and Macready 1995](https://reader036.vdocuments.mx/reader036/viewer/2022062315/5697bfd91a28abf838caf680/html5/thumbnails/1.jpg)
Ensemble Methods
![Page 2: Ensemble Methods. “No free lunch theorem” Wolpert and Macready 1995](https://reader036.vdocuments.mx/reader036/viewer/2022062315/5697bfd91a28abf838caf680/html5/thumbnails/2.jpg)
“No free lunch theorem” Wolpert and Macready 1995
![Page 3: Ensemble Methods. “No free lunch theorem” Wolpert and Macready 1995](https://reader036.vdocuments.mx/reader036/viewer/2022062315/5697bfd91a28abf838caf680/html5/thumbnails/3.jpg)
“No free lunch theorem” Wolpert and Macready 1995
Solution search also involves searching for learners
![Page 4: Ensemble Methods. “No free lunch theorem” Wolpert and Macready 1995](https://reader036.vdocuments.mx/reader036/viewer/2022062315/5697bfd91a28abf838caf680/html5/thumbnails/4.jpg)
Different algorithms
![Page 5: Ensemble Methods. “No free lunch theorem” Wolpert and Macready 1995](https://reader036.vdocuments.mx/reader036/viewer/2022062315/5697bfd91a28abf838caf680/html5/thumbnails/5.jpg)
Different algorithmsDifferent parameters
![Page 6: Ensemble Methods. “No free lunch theorem” Wolpert and Macready 1995](https://reader036.vdocuments.mx/reader036/viewer/2022062315/5697bfd91a28abf838caf680/html5/thumbnails/6.jpg)
Different algorithmsDifferent parametersDifferent input
representations/features
![Page 7: Ensemble Methods. “No free lunch theorem” Wolpert and Macready 1995](https://reader036.vdocuments.mx/reader036/viewer/2022062315/5697bfd91a28abf838caf680/html5/thumbnails/7.jpg)
Different algorithmsDifferent parametersDifferent input
representations/featuresDifferent data
![Page 8: Ensemble Methods. “No free lunch theorem” Wolpert and Macready 1995](https://reader036.vdocuments.mx/reader036/viewer/2022062315/5697bfd91a28abf838caf680/html5/thumbnails/8.jpg)
Base learner
![Page 9: Ensemble Methods. “No free lunch theorem” Wolpert and Macready 1995](https://reader036.vdocuments.mx/reader036/viewer/2022062315/5697bfd91a28abf838caf680/html5/thumbnails/9.jpg)
Diversity over accuracy
![Page 10: Ensemble Methods. “No free lunch theorem” Wolpert and Macready 1995](https://reader036.vdocuments.mx/reader036/viewer/2022062315/5697bfd91a28abf838caf680/html5/thumbnails/10.jpg)
Model combination
![Page 11: Ensemble Methods. “No free lunch theorem” Wolpert and Macready 1995](https://reader036.vdocuments.mx/reader036/viewer/2022062315/5697bfd91a28abf838caf680/html5/thumbnails/11.jpg)
VotingBaggingBoostingCascading
![Page 12: Ensemble Methods. “No free lunch theorem” Wolpert and Macready 1995](https://reader036.vdocuments.mx/reader036/viewer/2022062315/5697bfd91a28abf838caf680/html5/thumbnails/12.jpg)
![Page 13: Ensemble Methods. “No free lunch theorem” Wolpert and Macready 1995](https://reader036.vdocuments.mx/reader036/viewer/2022062315/5697bfd91a28abf838caf680/html5/thumbnails/13.jpg)
![Page 14: Ensemble Methods. “No free lunch theorem” Wolpert and Macready 1995](https://reader036.vdocuments.mx/reader036/viewer/2022062315/5697bfd91a28abf838caf680/html5/thumbnails/14.jpg)
![Page 15: Ensemble Methods. “No free lunch theorem” Wolpert and Macready 1995](https://reader036.vdocuments.mx/reader036/viewer/2022062315/5697bfd91a28abf838caf680/html5/thumbnails/15.jpg)
Data set = [1,2,3,4,5,6,7,8,9,10]
Samples: Input to learner 1 = [10,2,5,10,3] Input to learner 2 = [4,5,2,7,6,3] Input to learner 3 = [8,8,4,9,1]
![Page 16: Ensemble Methods. “No free lunch theorem” Wolpert and Macready 1995](https://reader036.vdocuments.mx/reader036/viewer/2022062315/5697bfd91a28abf838caf680/html5/thumbnails/16.jpg)
Create complementary learners
![Page 17: Ensemble Methods. “No free lunch theorem” Wolpert and Macready 1995](https://reader036.vdocuments.mx/reader036/viewer/2022062315/5697bfd91a28abf838caf680/html5/thumbnails/17.jpg)
Create complementary learnersTrain successive learners on the
mistakes of predecessors
![Page 18: Ensemble Methods. “No free lunch theorem” Wolpert and Macready 1995](https://reader036.vdocuments.mx/reader036/viewer/2022062315/5697bfd91a28abf838caf680/html5/thumbnails/18.jpg)
Weak learners combine to a strong learner
![Page 19: Ensemble Methods. “No free lunch theorem” Wolpert and Macready 1995](https://reader036.vdocuments.mx/reader036/viewer/2022062315/5697bfd91a28abf838caf680/html5/thumbnails/19.jpg)
![Page 20: Ensemble Methods. “No free lunch theorem” Wolpert and Macready 1995](https://reader036.vdocuments.mx/reader036/viewer/2022062315/5697bfd91a28abf838caf680/html5/thumbnails/20.jpg)
![Page 21: Ensemble Methods. “No free lunch theorem” Wolpert and Macready 1995](https://reader036.vdocuments.mx/reader036/viewer/2022062315/5697bfd91a28abf838caf680/html5/thumbnails/21.jpg)
![Page 22: Ensemble Methods. “No free lunch theorem” Wolpert and Macready 1995](https://reader036.vdocuments.mx/reader036/viewer/2022062315/5697bfd91a28abf838caf680/html5/thumbnails/22.jpg)
![Page 23: Ensemble Methods. “No free lunch theorem” Wolpert and Macready 1995](https://reader036.vdocuments.mx/reader036/viewer/2022062315/5697bfd91a28abf838caf680/html5/thumbnails/23.jpg)
Adaboost – Adaptive Boosting
![Page 24: Ensemble Methods. “No free lunch theorem” Wolpert and Macready 1995](https://reader036.vdocuments.mx/reader036/viewer/2022062315/5697bfd91a28abf838caf680/html5/thumbnails/24.jpg)
Adaboost – Adaptive BoostingAllows for a smaller training set
![Page 25: Ensemble Methods. “No free lunch theorem” Wolpert and Macready 1995](https://reader036.vdocuments.mx/reader036/viewer/2022062315/5697bfd91a28abf838caf680/html5/thumbnails/25.jpg)
Adaboost – Adaptive BoostingAllows for a smaller training setSimple classifiers
![Page 26: Ensemble Methods. “No free lunch theorem” Wolpert and Macready 1995](https://reader036.vdocuments.mx/reader036/viewer/2022062315/5697bfd91a28abf838caf680/html5/thumbnails/26.jpg)
Adaboost – Adaptive BoostingAllows for a smaller training setSimple classifiersBinary
![Page 27: Ensemble Methods. “No free lunch theorem” Wolpert and Macready 1995](https://reader036.vdocuments.mx/reader036/viewer/2022062315/5697bfd91a28abf838caf680/html5/thumbnails/27.jpg)
Modify probability of drawing examples from a training set based on errors
![Page 28: Ensemble Methods. “No free lunch theorem” Wolpert and Macready 1995](https://reader036.vdocuments.mx/reader036/viewer/2022062315/5697bfd91a28abf838caf680/html5/thumbnails/28.jpg)
![Page 29: Ensemble Methods. “No free lunch theorem” Wolpert and Macready 1995](https://reader036.vdocuments.mx/reader036/viewer/2022062315/5697bfd91a28abf838caf680/html5/thumbnails/29.jpg)
![Page 30: Ensemble Methods. “No free lunch theorem” Wolpert and Macready 1995](https://reader036.vdocuments.mx/reader036/viewer/2022062315/5697bfd91a28abf838caf680/html5/thumbnails/30.jpg)
![Page 31: Ensemble Methods. “No free lunch theorem” Wolpert and Macready 1995](https://reader036.vdocuments.mx/reader036/viewer/2022062315/5697bfd91a28abf838caf680/html5/thumbnails/31.jpg)
![Page 32: Ensemble Methods. “No free lunch theorem” Wolpert and Macready 1995](https://reader036.vdocuments.mx/reader036/viewer/2022062315/5697bfd91a28abf838caf680/html5/thumbnails/32.jpg)
€
α1= 12log(
1− error
error)
€
α1= 12log(
1− .33
.33)
€
α1= 0.35€
error = 0.33
Step 3
![Page 33: Ensemble Methods. “No free lunch theorem” Wolpert and Macready 1995](https://reader036.vdocuments.mx/reader036/viewer/2022062315/5697bfd91a28abf838caf680/html5/thumbnails/33.jpg)
![Page 34: Ensemble Methods. “No free lunch theorem” Wolpert and Macready 1995](https://reader036.vdocuments.mx/reader036/viewer/2022062315/5697bfd91a28abf838caf680/html5/thumbnails/34.jpg)
![Page 35: Ensemble Methods. “No free lunch theorem” Wolpert and Macready 1995](https://reader036.vdocuments.mx/reader036/viewer/2022062315/5697bfd91a28abf838caf680/html5/thumbnails/35.jpg)
![Page 36: Ensemble Methods. “No free lunch theorem” Wolpert and Macready 1995](https://reader036.vdocuments.mx/reader036/viewer/2022062315/5697bfd91a28abf838caf680/html5/thumbnails/36.jpg)
Demo
![Page 37: Ensemble Methods. “No free lunch theorem” Wolpert and Macready 1995](https://reader036.vdocuments.mx/reader036/viewer/2022062315/5697bfd91a28abf838caf680/html5/thumbnails/37.jpg)
Sequence classifiers by complexity
![Page 38: Ensemble Methods. “No free lunch theorem” Wolpert and Macready 1995](https://reader036.vdocuments.mx/reader036/viewer/2022062315/5697bfd91a28abf838caf680/html5/thumbnails/38.jpg)
Sequence classifiers by complexityUse classifier j+1 if classifier j
doesn’t meet a confidence threshold
![Page 39: Ensemble Methods. “No free lunch theorem” Wolpert and Macready 1995](https://reader036.vdocuments.mx/reader036/viewer/2022062315/5697bfd91a28abf838caf680/html5/thumbnails/39.jpg)
Sequence classifiers by complexityUse classifier j+1 if classifier j
doesn’t meet a confidence thresholdTrain cascading classifiers on
instances the previous classifier is not confident about
![Page 40: Ensemble Methods. “No free lunch theorem” Wolpert and Macready 1995](https://reader036.vdocuments.mx/reader036/viewer/2022062315/5697bfd91a28abf838caf680/html5/thumbnails/40.jpg)
Sequence classifiers by complexityUse classifier j+1 if classifier j
doesn’t meet a confidence thresholdTrain cascading classifiers on
instances the previous classifier is not confident about
Most examples classified quickly, harder ones passed to more expensive classifiers
![Page 41: Ensemble Methods. “No free lunch theorem” Wolpert and Macready 1995](https://reader036.vdocuments.mx/reader036/viewer/2022062315/5697bfd91a28abf838caf680/html5/thumbnails/41.jpg)
Boosting and Cascading
![Page 42: Ensemble Methods. “No free lunch theorem” Wolpert and Macready 1995](https://reader036.vdocuments.mx/reader036/viewer/2022062315/5697bfd91a28abf838caf680/html5/thumbnails/42.jpg)
![Page 43: Ensemble Methods. “No free lunch theorem” Wolpert and Macready 1995](https://reader036.vdocuments.mx/reader036/viewer/2022062315/5697bfd91a28abf838caf680/html5/thumbnails/43.jpg)
![Page 44: Ensemble Methods. “No free lunch theorem” Wolpert and Macready 1995](https://reader036.vdocuments.mx/reader036/viewer/2022062315/5697bfd91a28abf838caf680/html5/thumbnails/44.jpg)
![Page 45: Ensemble Methods. “No free lunch theorem” Wolpert and Macready 1995](https://reader036.vdocuments.mx/reader036/viewer/2022062315/5697bfd91a28abf838caf680/html5/thumbnails/45.jpg)
![Page 46: Ensemble Methods. “No free lunch theorem” Wolpert and Macready 1995](https://reader036.vdocuments.mx/reader036/viewer/2022062315/5697bfd91a28abf838caf680/html5/thumbnails/46.jpg)
![Page 47: Ensemble Methods. “No free lunch theorem” Wolpert and Macready 1995](https://reader036.vdocuments.mx/reader036/viewer/2022062315/5697bfd91a28abf838caf680/html5/thumbnails/47.jpg)
Object detection/trackingCollaborative filteringNeural networksOptical character recognition ++BiometricsData mining
![Page 48: Ensemble Methods. “No free lunch theorem” Wolpert and Macready 1995](https://reader036.vdocuments.mx/reader036/viewer/2022062315/5697bfd91a28abf838caf680/html5/thumbnails/48.jpg)
Ensemble methods are proven effective, but why?