mle’s, bayesian classifiers and naïve bayes machine learning 10-601 tom m. mitchell machine...

MLE’s, Bayesian Classifiers and Naïve Bayes Machine Learning 10-601 Tom M. Mitchell Machine Learning Department Carnegie Mellon University January 30, 2008 Required reading: • Mitchell draft chapter, sections 1 and 2. (available on class website)

Upload: lesley-kennedy

Post on 05-Jan-2016

224 views

Category:

Documents

5 download

Report

Download

Embed Size (px):

TRANSCRIPT

MLE’s, Bayesian Classifiers and Naïve Bayes

Machine Learning 10-601

Tom M. MitchellMachine Learning Department

Carnegie Mellon University

January 30, 2008

Required reading:

• Mitchell draft chapter, sections 1 and 2. (available on class website)

Page 2: MLE’s, Bayesian Classifiers and Naïve Bayes Machine Learning 10-601 Tom M. Mitchell Machine Learning Department Carnegie Mellon University January 30,

Naïve Bayes in a Nutshell

Bayes rule:

Assuming conditional independence among Xi’s:

So, classification rule for Xnew = < X1, …, Xn > is:

Page 3: MLE’s, Bayesian Classifiers and Naïve Bayes Machine Learning 10-601 Tom M. Mitchell Machine Learning Department Carnegie Mellon University January 30,

Naïve Bayes Algorithm – discrete Xi

• Train Naïve Bayes (examples)

for each* value yk

estimate

for each* value xij of each attribute Xi

estimate

• Classify (Xnew)

* probabilities must sum to 1, so need estimate only n-1 parameters...

Page 4: MLE’s, Bayesian Classifiers and Naïve Bayes Machine Learning 10-601 Tom M. Mitchell Machine Learning Department Carnegie Mellon University January 30,

Estimating Parameters: Y, Xi discrete-valued

Maximum likelihood estimates (MLE’s):

Number of items in set D for which Y=yk

Page 5: MLE’s, Bayesian Classifiers and Naïve Bayes Machine Learning 10-601 Tom M. Mitchell Machine Learning Department Carnegie Mellon University January 30,

Example: Live in Sq Hill? P(S|G,D,M)• S=1 iff live in Squirrel Hill• G=1 iff shop at Giant Eagle

• D=1 iff Drive to CMU• M=1 iff Dave Matthews fan

Page 6: MLE’s, Bayesian Classifiers and Naïve Bayes Machine Learning 10-601 Tom M. Mitchell Machine Learning Department Carnegie Mellon University January 30,

Example: Live in Sq Hill? P(S|G,D,M)• S=1 iff live in Squirrel Hill• G=1 iff shop at Giant Eagle

• D=1 iff Drive to CMU• M=1 iff Dave Matthews fan

Page 7: MLE’s, Bayesian Classifiers and Naïve Bayes Machine Learning 10-601 Tom M. Mitchell Machine Learning Department Carnegie Mellon University January 30,

Naïve Bayes: Subtlety #1

If unlucky, our MLE estimate for P(Xi | Y) may be zero. (e.g., X373= Birthday_Is_January30)

• Why worry about just one parameter out of many?

• What can be done to avoid this?

Page 8: MLE’s, Bayesian Classifiers and Naïve Bayes Machine Learning 10-601 Tom M. Mitchell Machine Learning Department Carnegie Mellon University January 30,

Estimating Parameters: Y, Xi discrete-valued

Maximum likelihood estimates:

MAP estimates (Dirichlet priors):

Only difference: “imaginary” examples

Page 9: MLE’s, Bayesian Classifiers and Naïve Bayes Machine Learning 10-601 Tom M. Mitchell Machine Learning Department Carnegie Mellon University January 30,

Naïve Bayes: Subtlety #2

Often the Xi are not really conditionally independent

• We use Naïve Bayes in many cases anyway, and it often works pretty well– often the right classification, even when not the right

probability (see [Domingos&Pazzani, 1996])

• What is effect on estimated P(Y|X)?– Special case: what if we add two copies: Xi = Xk

Page 10: MLE’s, Bayesian Classifiers and Naïve Bayes Machine Learning 10-601 Tom M. Mitchell Machine Learning Department Carnegie Mellon University January 30,

Learning to classify text documents

• Classify which emails are spam

• Classify which emails are meeting invites

• Classify which web pages are student home pages

How shall we represent text documents for Naïve Bayes?

Page 11: MLE’s, Bayesian Classifiers and Naïve Bayes Machine Learning 10-601 Tom M. Mitchell Machine Learning Department Carnegie Mellon University January 30,

Page 12: MLE’s, Bayesian Classifiers and Naïve Bayes Machine Learning 10-601 Tom M. Mitchell Machine Learning Department Carnegie Mellon University January 30,

Page 13: MLE’s, Bayesian Classifiers and Naïve Bayes Machine Learning 10-601 Tom M. Mitchell Machine Learning Department Carnegie Mellon University January 30,

Baseline: Bag of Words Approach

aardvark 0

about 2

all 2

Africa 1

apple 0

anxious 0

...

gas 1

...

oil 1

…

Zaire 0

Page 14: MLE’s, Bayesian Classifiers and Naïve Bayes Machine Learning 10-601 Tom M. Mitchell Machine Learning Department Carnegie Mellon University January 30,

Page 15: MLE’s, Bayesian Classifiers and Naïve Bayes Machine Learning 10-601 Tom M. Mitchell Machine Learning Department Carnegie Mellon University January 30,

For code and data, see www.cs.cmu.edu/~tom/mlbook.html click on “Software and Data”

Page 16: MLE’s, Bayesian Classifiers and Naïve Bayes Machine Learning 10-601 Tom M. Mitchell Machine Learning Department Carnegie Mellon University January 30,

Page 17: MLE’s, Bayesian Classifiers and Naïve Bayes Machine Learning 10-601 Tom M. Mitchell Machine Learning Department Carnegie Mellon University January 30,

Page 18: MLE’s, Bayesian Classifiers and Naïve Bayes Machine Learning 10-601 Tom M. Mitchell Machine Learning Department Carnegie Mellon University January 30,

What you should know:

• Training and using classifiers based on Bayes rule

• Conditional independence– What it is– Why it’s important

• Naïve Bayes– What it is– Why we use it so much– Training using MLE, MAP estimates– Discrete variables (Bernoulli) and continuous (Gaussian)

Page 19: MLE’s, Bayesian Classifiers and Naïve Bayes Machine Learning 10-601 Tom M. Mitchell Machine Learning Department Carnegie Mellon University January 30,

Questions:

• Can you use Naïve Bayes for a combination of discrete and real-valued Xi?

• How can we easily model just 2 of n attributes as dependent?

• What does the decision surface of a Naïve Bayes classifier look like?

Page 20: MLE’s, Bayesian Classifiers and Naïve Bayes Machine Learning 10-601 Tom M. Mitchell Machine Learning Department Carnegie Mellon University January 30,

What is form of decision surface for Naïve Bayes classifier?

DAT340/DIT866 Applied Machine Learning: Machine learning ...richajo/dit866/lectures/l11/l11.pdfDAT340/DIT866 Applied Machine Learning: Machine learning and ethics Vilhelm Verendel

Computer Security: A Machine Learning Approachmedia.techtarget.com/searchSecurity/downloads/REVISED_RH9_Sabn… · Royal Holloway series Machine learning approaches • MACHINE LEARNING

Section 1Section 1 Machine Learning basic concepts · Machine Learning TutorialMachine Learning Tutorial ... Section 1Section 1 Machine Learning basic concepts ... Learning Tutorial

Deep learning - Machine learning overviewce.sharif.edu/courses/98-99/1/ce719-1/resources/root/...Deep learning j What is machine learning? Types of machine learning Machine learning

Machine Learning with MATLAB · 2 Agenda Machine Learning Overview Machine Learning with MATLAB –Unsupervised Learning Clustering –Supervised Learning Classification Regression

Hawaii Machine Learning Meetup · Introduction –Why Machine Learning? Machine Learning Quotes ^A breakthrough in machine learning would be worth ten Microsofts. ―Bill Gates For

Machine Learning Made Easy€¦ · Machine Learning –What is Machine Learning and why do we need it? –Common challenges in Machine Learning Example 1: Human activity learning

MACHINE REASONING: A PERSPECTIVE AND …...“From machine learning to machine reasoning”. Machine Learning Volume 94 Issue 2, 2004, Pages 133-149 Machine Learning Volume …

Machine Learning using MapReduce - Boise State Universitycs.boisestate.edu/.../mapreduce-machine-learning-handout.pdf · 2016-10-27 · What is Machine Learning I Machine learning

Machine learning for neuroimaging with scikit-learn · Machine learning for neuroimaging with scikit-learn. ... a Python machine learning library, ... Abraham et al. Machine learning

Introducing Machine Learning · 4 ntroducing Machine Learning How Machine Learning Works Machine learning uses two types of techniques: supervised learning, which trains a model on

MACHINE LEARNING AN INTRODUCTION - Sas Institute · 2016-03-11 · MACHINE LEARNING - AN INTRODUCTION WHAT IS MACHINE LEARNING? Wikipedia: Machine learning, a branch of artificial

1. Introduction to Machine - lis-lab.fr › bernard.espinasse › ... · § 1. Introduction to Machine Learning § Machine Learning Problem § Machine Learning: Supervised & Unsupervised

ARTIFICIAL INTELLIGENCE & MACHINE LEARNING · Intelligence & Machine Learning technologies and applications including Machine Learning, Deep Learning, Computer Vision, Natural Language

An Introduction to Machine Learning - GitHub Pages · Introduction Machine learning The machine learning process What’s machine learning History Supervised learning Non-supervised

Machine Learning in Logistics: Machine Learning Algorithmsltu.diva-portal.org/smash/get/diva2:1118764/FULLTEXT02.pdf · Machine Learning in Logistics: Machine Learning ... Machine

Machine Learning - 0. Overview€¦ · Machine Learning 1. What is Machine Learning? One area of research, many names (and aspects) machine learning historically, stresses learning

MLE’s, Bayesian Classifiers and Naïve Bayes

Advanced Topics in Machine Learning: Bayesian Machine Learning · 2020. 2. 3. · Advanced Topics in Machine Learning: Bayesian Machine Learning Tom Rainforth Department of Computer

Introduction To Azure Machine Learning...Expected Learning Outcomes Azure Machine Learning @tetranoodle Supervised Machine Learning Machine Learning Algorithms in Azure ML Workflow

¿Qué es Machine Learning? Usos de Machine Learning

11 Machine Learning Important Issues in Machine Learning

Methods MACHINE LEARNING FROM MACHINE LEARNING TO …

COMP 562: Introduction to Machine Learning · Introduction to Machine Learning Machine learning: What and Why? Quiz: De ning the Learning Task "Machine Learning is the study of algorithms

Predicting Housing Prices With Azure Machine Learning · Azure Machine Learning. Expected Learning Outcomes Azure Machine Learning @tetranoodle Build Regression Machine Learning Model

Machine Learning Probabilistic Machine Learning · Machine Learning Probabilistic Machine Learning learning as inference, Bayesian Kernel Ridge regression = Gaussian Processes, Bayesian

Machine Learning: Machine Learning: Introduction Introduction

1-Machine Learning in Oracle - ITOUG2017/12/01 · – Oracle Machine Learning (on Oracle Autonomous Data Warehouse Cloud Service) – Automated Machine Learning => Machine Learning

Chapter- 6 : Machine Learning - IOE Notesioenotes.edu.np › media › ...Chapter-6-machine-learning... · Chapter- 6 : Machine Learning-Machine learning is a branch of AI that uses

Machine Learning with MATLAB - … · 2 Agenda Machine Learning Overview Machine Learning with MATLAB –Unsupervised Learning Clustering –Supervised Learning Classification

Machine Learning FabSpace March 2018 - irit.fr Learning FabSpace March 2018... · Machine Learning 5 Machine Learning • Supervised : Train / test • Predictiveanalysis Machine

Learning with large datasets Machine Learning Large scale machine learning

Machine Learning Introduction Machine learningbejar/ia/transpas/teoria.mti/6-AP-aprendizaj… · Machine Learning Introduction Types of machine learning Inductive Learning: Models

Fast Track Machine Learning Part 1 (Machine Learning Overview) - Types of Machine Learning Problems

Machine Learning-Deep Learning Fondement et biaisfiles.meetup.com/19203187/FondetBiaisMLDLonline.pdf · Machine Learning-Deep Learning Fondement et biais Machine learning Aix-Marseille