overview - edx · 2015-06-29 · distributed computing data machine learning ml is applied...
TRANSCRIPT
![Page 1: Overview - edX · 2015-06-29 · Distributed Computing Data Machine Learning ML is Applied Everywhere E.g., Personalized recommendations, Speech recognition, Face detection, Protein](https://reader034.vdocuments.mx/reader034/viewer/2022042108/5e88a05c3c7f457f6a39cd43/html5/thumbnails/1.jpg)
Overview
![Page 2: Overview - edX · 2015-06-29 · Distributed Computing Data Machine Learning ML is Applied Everywhere E.g., Personalized recommendations, Speech recognition, Face detection, Protein](https://reader034.vdocuments.mx/reader034/viewer/2022042108/5e88a05c3c7f457f6a39cd43/html5/thumbnails/2.jpg)
Rapid growth of massive datasets E.g., Online activity, Science, Sensor networks
Data
![Page 3: Overview - edX · 2015-06-29 · Distributed Computing Data Machine Learning ML is Applied Everywhere E.g., Personalized recommendations, Speech recognition, Face detection, Protein](https://reader034.vdocuments.mx/reader034/viewer/2022042108/5e88a05c3c7f457f6a39cd43/html5/thumbnails/3.jpg)
Distributed Computing
Data
Distributed Clusters are Pervasive
![Page 4: Overview - edX · 2015-06-29 · Distributed Computing Data Machine Learning ML is Applied Everywhere E.g., Personalized recommendations, Speech recognition, Face detection, Protein](https://reader034.vdocuments.mx/reader034/viewer/2022042108/5e88a05c3c7f457f6a39cd43/html5/thumbnails/4.jpg)
Distributed Computing
Data Machine Learning
Mature Methods for Common Problems e.g., classification, regression, collaborative filtering, clustering
![Page 5: Overview - edX · 2015-06-29 · Distributed Computing Data Machine Learning ML is Applied Everywhere E.g., Personalized recommendations, Speech recognition, Face detection, Protein](https://reader034.vdocuments.mx/reader034/viewer/2022042108/5e88a05c3c7f457f6a39cd43/html5/thumbnails/5.jpg)
Distributed Computing
Data Machine Learning
ML is Applied Everywhere E.g., Personalized recommendations, Speech recognition, Face detection, Protein structure, Fraud detection, Spam filtering, Playing chess or Jeopardy, Unassisted
vehicle control, Medical diagnosis, ...
![Page 6: Overview - edX · 2015-06-29 · Distributed Computing Data Machine Learning ML is Applied Everywhere E.g., Personalized recommendations, Speech recognition, Face detection, Protein](https://reader034.vdocuments.mx/reader034/viewer/2022042108/5e88a05c3c7f457f6a39cd43/html5/thumbnails/6.jpg)
Distributed Computing
Data Machine Learning
ML is Applied Everywhere E.g., Personalized recommendations, Speech recognition, Face detection, Protein structure, Fraud detection, Spam filtering, Playing chess or Jeopardy, Unassisted
vehicle control, Medical diagnosis, ...
![Page 7: Overview - edX · 2015-06-29 · Distributed Computing Data Machine Learning ML is Applied Everywhere E.g., Personalized recommendations, Speech recognition, Face detection, Protein](https://reader034.vdocuments.mx/reader034/viewer/2022042108/5e88a05c3c7f457f6a39cd43/html5/thumbnails/7.jpg)
Classic ML techniques are not always suitable for modern datasets
0"
10"
20"
30"
40"
50"
60"
2010" 2011" 2012" 2013" 2014" 2015"
Moore's"Law"Overall"Data"Par8cle"Accel."DNA"Sequencers"
Data"Grows"faster"than"Moore’s"Law"[IDC%report,%Kathy%Yelick,%LBNL]%
Challenge: Scalability
Machine Learning
Data
Distributed Computing
Data Grows Faster than Moore’s Law [IDC report, Kathy Yelick, LBNL]
![Page 8: Overview - edX · 2015-06-29 · Distributed Computing Data Machine Learning ML is Applied Everywhere E.g., Personalized recommendations, Speech recognition, Face detection, Protein](https://reader034.vdocuments.mx/reader034/viewer/2022042108/5e88a05c3c7f457f6a39cd43/html5/thumbnails/8.jpg)
Focus on scalability challenges for common ML tasks
How can we use raw data to train statistical models?• Study typical ML pipelines• Classification, regression, exploratory analysis
How can we do so at scale?• Study distributed machine learning algorithms• Implement distributed pipelines in Apache Spark
using real datasets• Understand details of MLlib (Spark’s ML library)
Course Goals
Machine Learning
Data
Distributed Computing
![Page 9: Overview - edX · 2015-06-29 · Distributed Computing Data Machine Learning ML is Applied Everywhere E.g., Personalized recommendations, Speech recognition, Face detection, Protein](https://reader034.vdocuments.mx/reader034/viewer/2022042108/5e88a05c3c7f457f6a39cd43/html5/thumbnails/9.jpg)
Basic Python, ML, math background • First week provides review of ML and useful math concepts
Self-assessment exam has pointers to review material • http://cs.ucla.edu/~ameet/self_assessment.pdf
Assumes no knowledge of Spark • Second week introduces Spark
Prerequisites
![Page 10: Overview - edX · 2015-06-29 · Distributed Computing Data Machine Learning ML is Applied Everywhere E.g., Personalized recommendations, Speech recognition, Face detection, Protein](https://reader034.vdocuments.mx/reader034/viewer/2022042108/5e88a05c3c7f457f6a39cd43/html5/thumbnails/10.jpg)
5 weeks of lectures, 5 Spark coding labs, 1 setup lab • Week 0: Setup software environment • Week 1: ML overview, Math review • Week 2: Spark Overview [from CS100.1x] • Week 3: Distributed ML Principles and Linear Regression • Week 4: Classification with Click-through Rate Prediction • Week 5: Exploratory Analysis with Brain Imaging Data
Schedule
![Page 11: Overview - edX · 2015-06-29 · Distributed Computing Data Machine Learning ML is Applied Everywhere E.g., Personalized recommendations, Speech recognition, Face detection, Protein](https://reader034.vdocuments.mx/reader034/viewer/2022042108/5e88a05c3c7f457f6a39cd43/html5/thumbnails/11.jpg)
Distributed Computing and Apache Spark
![Page 12: Overview - edX · 2015-06-29 · Distributed Computing Data Machine Learning ML is Applied Everywhere E.g., Personalized recommendations, Speech recognition, Face detection, Protein](https://reader034.vdocuments.mx/reader034/viewer/2022042108/5e88a05c3c7f457f6a39cd43/html5/thumbnails/12.jpg)
Need more hardware to store / process modern dataScale-up (one big machine)• Can be very fast for medium scale problems• Expensive, specialized hardware• Eventually hit a wall
How to Handle Massive Data?
CPU
Disk
RAM
CPUDisk
RAM
![Page 13: Overview - edX · 2015-06-29 · Distributed Computing Data Machine Learning ML is Applied Everywhere E.g., Personalized recommendations, Speech recognition, Face detection, Protein](https://reader034.vdocuments.mx/reader034/viewer/2022042108/5e88a05c3c7f457f6a39cd43/html5/thumbnails/13.jpg)
How to Handle Massive Data?Need more hardware to store / process modern dataScale-out (distributed, e.g., cloud-based)• Commodity hardware, scales to massive problems• Need to deal with network communication• Added software complexity
Network
CPU
Disk
RAM …
CPU
Disk
RAM
CPU
Disk
RAM
CPU
Disk
RAM
![Page 14: Overview - edX · 2015-06-29 · Distributed Computing Data Machine Learning ML is Applied Everywhere E.g., Personalized recommendations, Speech recognition, Face detection, Protein](https://reader034.vdocuments.mx/reader034/viewer/2022042108/5e88a05c3c7f457f6a39cd43/html5/thumbnails/14.jpg)
General, open-source cluster computing engine
Well-suited for machine learning• Fast iterative procedures• Efficient communication primitives
Simple and Expressive• APIs in Scala, Java, Python, R• Interactive Shell
Integrated Higher-Level Libraries
What is Apache ?
SparkSQL
Apache Spark
Spark Streaming MLlib GraphX
![Page 15: Overview - edX · 2015-06-29 · Distributed Computing Data Machine Learning ML is Applied Everywhere E.g., Personalized recommendations, Speech recognition, Face detection, Protein](https://reader034.vdocuments.mx/reader034/viewer/2022042108/5e88a05c3c7f457f6a39cd43/html5/thumbnails/15.jpg)
What is Machine Learning?
![Page 16: Overview - edX · 2015-06-29 · Distributed Computing Data Machine Learning ML is Applied Everywhere E.g., Personalized recommendations, Speech recognition, Face detection, Protein](https://reader034.vdocuments.mx/reader034/viewer/2022042108/5e88a05c3c7f457f6a39cd43/html5/thumbnails/16.jpg)
Constructing and studying methods that learn from and make predictions on data
Broad area involving tools and ideas from various domains• Computer Science• Probability and Statistics • Optimization• Linear Algebra
A Definition
![Page 17: Overview - edX · 2015-06-29 · Distributed Computing Data Machine Learning ML is Applied Everywhere E.g., Personalized recommendations, Speech recognition, Face detection, Protein](https://reader034.vdocuments.mx/reader034/viewer/2022042108/5e88a05c3c7f457f6a39cd43/html5/thumbnails/17.jpg)
Some ExamplesFace recognition
Link prediction
Text or document classification, e.g., spam detection
Protein structure prediction
Games, e.g., Backgammon or Jeopardy
![Page 18: Overview - edX · 2015-06-29 · Distributed Computing Data Machine Learning ML is Applied Everywhere E.g., Personalized recommendations, Speech recognition, Face detection, Protein](https://reader034.vdocuments.mx/reader034/viewer/2022042108/5e88a05c3c7f457f6a39cd43/html5/thumbnails/18.jpg)
Observations. Items or entities used for learning or evaluation, e.g., emails
Features. Attributes (typically numeric) used to represent an observation, e.g., length, date, presence of keywords
Labels. Values / categories assigned to observations, e.g., spam, not-spam
Training and Test Data. Observations used to train and evaluate a learning algorithm, e.g., a set of emails along with their labels• Training data is given to the algorithm for training• Test data is withheld at train time
Terminology
![Page 19: Overview - edX · 2015-06-29 · Distributed Computing Data Machine Learning ML is Applied Everywhere E.g., Personalized recommendations, Speech recognition, Face detection, Protein](https://reader034.vdocuments.mx/reader034/viewer/2022042108/5e88a05c3c7f457f6a39cd43/html5/thumbnails/19.jpg)
Two Common Learning Settings
Supervised learning. Learning from labeled observations• Labels ‘teach’ algorithm to learn mapping from observations to labelsUnsupervised learning. Learning from unlabeled observations• Learning algorithm must find latent structure from features alone• Can be goal in itself (discover hidden patterns, exploratory data analysis)• Can be means to an end (preprocessing for supervised task)
![Page 20: Overview - edX · 2015-06-29 · Distributed Computing Data Machine Learning ML is Applied Everywhere E.g., Personalized recommendations, Speech recognition, Face detection, Protein](https://reader034.vdocuments.mx/reader034/viewer/2022042108/5e88a05c3c7f457f6a39cd43/html5/thumbnails/20.jpg)
Examples of Supervised Learning
Classification. Assign a category to each item, e.g., spam detection• Categories are discrete• Generally no notion of ‘closeness’ in multi-class setting
Regression. Predict a real value for each item, e.g., stock prices• Labels are continuous• Can define ‘closeness’ when comparing prediction with label
![Page 21: Overview - edX · 2015-06-29 · Distributed Computing Data Machine Learning ML is Applied Everywhere E.g., Personalized recommendations, Speech recognition, Face detection, Protein](https://reader034.vdocuments.mx/reader034/viewer/2022042108/5e88a05c3c7f457f6a39cd43/html5/thumbnails/21.jpg)
Examples of Unsupervised Learning
Clustering. Partition observations into homogeneous regions, e.g., to identify “communities” within large groups of people in social networks
Dimensionality Reduction. Transform an initial feature representation into a more concise representation, e.g., representing digital images
![Page 22: Overview - edX · 2015-06-29 · Distributed Computing Data Machine Learning ML is Applied Everywhere E.g., Personalized recommendations, Speech recognition, Face detection, Protein](https://reader034.vdocuments.mx/reader034/viewer/2022042108/5e88a05c3c7f457f6a39cd43/html5/thumbnails/22.jpg)
Typical Supervised Learning Pipeline
![Page 23: Overview - edX · 2015-06-29 · Distributed Computing Data Machine Learning ML is Applied Everywhere E.g., Personalized recommendations, Speech recognition, Face detection, Protein](https://reader034.vdocuments.mx/reader034/viewer/2022042108/5e88a05c3c7f457f6a39cd43/html5/thumbnails/23.jpg)
Raw data comes from many sourcesObtain Raw Data
![Page 24: Overview - edX · 2015-06-29 · Distributed Computing Data Machine Learning ML is Applied Everywhere E.g., Personalized recommendations, Speech recognition, Face detection, Protein](https://reader034.vdocuments.mx/reader034/viewer/2022042108/5e88a05c3c7f457f6a39cd43/html5/thumbnails/24.jpg)
Data Types
Web hypertext
![Page 25: Overview - edX · 2015-06-29 · Distributed Computing Data Machine Learning ML is Applied Everywhere E.g., Personalized recommendations, Speech recognition, Face detection, Protein](https://reader034.vdocuments.mx/reader034/viewer/2022042108/5e88a05c3c7f457f6a39cd43/html5/thumbnails/25.jpg)
Data Types
![Page 26: Overview - edX · 2015-06-29 · Distributed Computing Data Machine Learning ML is Applied Everywhere E.g., Personalized recommendations, Speech recognition, Face detection, Protein](https://reader034.vdocuments.mx/reader034/viewer/2022042108/5e88a05c3c7f457f6a39cd43/html5/thumbnails/26.jpg)
Data Types
Genomic Data, e.g., SNPs
![Page 27: Overview - edX · 2015-06-29 · Distributed Computing Data Machine Learning ML is Applied Everywhere E.g., Personalized recommendations, Speech recognition, Face detection, Protein](https://reader034.vdocuments.mx/reader034/viewer/2022042108/5e88a05c3c7f457f6a39cd43/html5/thumbnails/27.jpg)
Data Types
Images
![Page 28: Overview - edX · 2015-06-29 · Distributed Computing Data Machine Learning ML is Applied Everywhere E.g., Personalized recommendations, Speech recognition, Face detection, Protein](https://reader034.vdocuments.mx/reader034/viewer/2022042108/5e88a05c3c7f457f6a39cd43/html5/thumbnails/28.jpg)
Data Types
(Social) Networks / Graphs
![Page 29: Overview - edX · 2015-06-29 · Distributed Computing Data Machine Learning ML is Applied Everywhere E.g., Personalized recommendations, Speech recognition, Face detection, Protein](https://reader034.vdocuments.mx/reader034/viewer/2022042108/5e88a05c3c7f457f6a39cd43/html5/thumbnails/29.jpg)
User Ratings
Data Types
![Page 30: Overview - edX · 2015-06-29 · Distributed Computing Data Machine Learning ML is Applied Everywhere E.g., Personalized recommendations, Speech recognition, Face detection, Protein](https://reader034.vdocuments.mx/reader034/viewer/2022042108/5e88a05c3c7f457f6a39cd43/html5/thumbnails/30.jpg)
Initial observations can be in arbitrary formatWe extract features to represent observationsWe can incorporate domain knowledgeWe typically want numeric featuresSuccess of entire pipeline often depends on choosing good descriptions of observations!!
Obtain Raw Data
Feature Extraction
![Page 31: Overview - edX · 2015-06-29 · Distributed Computing Data Machine Learning ML is Applied Everywhere E.g., Personalized recommendations, Speech recognition, Face detection, Protein](https://reader034.vdocuments.mx/reader034/viewer/2022042108/5e88a05c3c7f457f6a39cd43/html5/thumbnails/31.jpg)
Obtain Raw Data
Unsupervised Learning
Feature Extraction
Preprocessing step for supervised learning
![Page 32: Overview - edX · 2015-06-29 · Distributed Computing Data Machine Learning ML is Applied Everywhere E.g., Personalized recommendations, Speech recognition, Face detection, Protein](https://reader034.vdocuments.mx/reader034/viewer/2022042108/5e88a05c3c7f457f6a39cd43/html5/thumbnails/32.jpg)
Train a supervised model using labeled data, e.g., Classification or Regression model
Obtain Raw Data
Feature Extraction
Supervised Learning
![Page 33: Overview - edX · 2015-06-29 · Distributed Computing Data Machine Learning ML is Applied Everywhere E.g., Personalized recommendations, Speech recognition, Face detection, Protein](https://reader034.vdocuments.mx/reader034/viewer/2022042108/5e88a05c3c7f457f6a39cd43/html5/thumbnails/33.jpg)
Q: How do we determine the quality of the model we’ve just trained?A: We can evaluate it on test / hold-out data, i.e., labeled data not used for trainingIf we don’t like the results, we iterate…
Obtain Raw Data
Feature Extraction
Supervised Learning
Evaluation
![Page 34: Overview - edX · 2015-06-29 · Distributed Computing Data Machine Learning ML is Applied Everywhere E.g., Personalized recommendations, Speech recognition, Face detection, Protein](https://reader034.vdocuments.mx/reader034/viewer/2022042108/5e88a05c3c7f457f6a39cd43/html5/thumbnails/34.jpg)
Obtain Raw Data
Feature Extraction
Predict
Evaluation
Supervised LearningOnce we’re happy with our model, we can use it to make predictions on future observations, i.e., data without a known label
![Page 35: Overview - edX · 2015-06-29 · Distributed Computing Data Machine Learning ML is Applied Everywhere E.g., Personalized recommendations, Speech recognition, Face detection, Protein](https://reader034.vdocuments.mx/reader034/viewer/2022042108/5e88a05c3c7f457f6a39cd43/html5/thumbnails/35.jpg)
Sample Classification Pipeline
![Page 36: Overview - edX · 2015-06-29 · Distributed Computing Data Machine Learning ML is Applied Everywhere E.g., Personalized recommendations, Speech recognition, Face detection, Protein](https://reader034.vdocuments.mx/reader034/viewer/2022042108/5e88a05c3c7f457f6a39cd43/html5/thumbnails/36.jpg)
Classification
Goal: Learn a mapping from observations to discrete labels given a set of training examples (supervised learning)
Example: Spam Classification• Observations are emails• Labels are {spam, not-spam} (Binary Classification)• Given a set of labeled emails, we want to predict whether
a new email is spam or not-spam
![Page 37: Overview - edX · 2015-06-29 · Distributed Computing Data Machine Learning ML is Applied Everywhere E.g., Personalized recommendations, Speech recognition, Face detection, Protein](https://reader034.vdocuments.mx/reader034/viewer/2022042108/5e88a05c3c7f457f6a39cd43/html5/thumbnails/37.jpg)
Other Examples
Fraud detection: User activity → {fraud, not fraud}
Face detection: Images → set of people
Link prediction: Users → {suggest link, don’t suggest link}
Clickthrough rate prediction: User and ads → {click, no click}
Many others…
![Page 38: Overview - edX · 2015-06-29 · Distributed Computing Data Machine Learning ML is Applied Everywhere E.g., Personalized recommendations, Speech recognition, Face detection, Protein](https://reader034.vdocuments.mx/reader034/viewer/2022042108/5e88a05c3c7f457f6a39cd43/html5/thumbnails/38.jpg)
training set
Classification Pipeline
Raw data consists of a set of labeled training observations
Obtain Raw Data
Feature Extraction
Predict
Evaluation
Supervised Learning
![Page 39: Overview - edX · 2015-06-29 · Distributed Computing Data Machine Learning ML is Applied Everywhere E.g., Personalized recommendations, Speech recognition, Face detection, Protein](https://reader034.vdocuments.mx/reader034/viewer/2022042108/5e88a05c3c7f457f6a39cd43/html5/thumbnails/39.jpg)
E.g., Spam ClassificationExample:(Spam(Classifica<on(
From: [email protected]
"Eliminate your debt by giving us your money..."
From: [email protected]
"Hi, it's been a while! How are you? ..."
spam
not-spam
Observation
spam
not-spam
Label
Obtain Raw Data
Feature Extraction
Predict
Evaluation
Supervised Learning
![Page 40: Overview - edX · 2015-06-29 · Distributed Computing Data Machine Learning ML is Applied Everywhere E.g., Personalized recommendations, Speech recognition, Face detection, Protein](https://reader034.vdocuments.mx/reader034/viewer/2022042108/5e88a05c3c7f457f6a39cd43/html5/thumbnails/40.jpg)
training set
Classification Pipeline
Feature extraction typically transforms each observations into a vector of real numbers (features)
Success or failure of a classifier often depends on choosing good descriptions of observations!!
Obtain Raw Data
Feature Extraction
Predict
Evaluation
Supervised Learning
![Page 41: Overview - edX · 2015-06-29 · Distributed Computing Data Machine Learning ML is Applied Everywhere E.g., Personalized recommendations, Speech recognition, Face detection, Protein](https://reader034.vdocuments.mx/reader034/viewer/2022042108/5e88a05c3c7f457f6a39cd43/html5/thumbnails/41.jpg)
E.g., “Bag of Words”
Observations are documentsBuild VocabularyDerive feature vectors from Vocabulary
Example:(Spam(Classifica<on(
From: [email protected]
"Eliminate your debt by giving us your money..."
From: [email protected]
"Hi, it's been a while! How are you? ..."
Vocabularybeendebt
eliminategivinghowit'smoneywhile
Example:(Spam(Classifica<on(
From: [email protected]
"Eliminate your debt by giving us your money..."
From: [email protected]
"Hi, it's been a while! How are you? ..."
Vocabularybeendebt
eliminategivinghowit'smoneywhile
Example:(Spam(Classifica<on(
From: [email protected]
"Eliminate your debt by giving us your money..."
been
debt
eliminate
giving
how
it's
money
while
0
1
1
1
0
0
1
0
Obtain Raw Data
Feature Extraction
Predict
Evaluation
Supervised Learning
![Page 42: Overview - edX · 2015-06-29 · Distributed Computing Data Machine Learning ML is Applied Everywhere E.g., Personalized recommendations, Speech recognition, Face detection, Protein](https://reader034.vdocuments.mx/reader034/viewer/2022042108/5e88a05c3c7f457f6a39cd43/html5/thumbnails/42.jpg)
training set classifier
Classification Pipeline
Supervised Learning: Train classifier using training data• Common classifiers include Logistic Regression, SVMs, Decision
Trees, Random Forests, etc.
Training (especially at scale) often involves iterative computations, e.g., gradient descent
Obtain Raw Data
Feature Extraction
Predict
Evaluation
Supervised Learning
![Page 43: Overview - edX · 2015-06-29 · Distributed Computing Data Machine Learning ML is Applied Everywhere E.g., Personalized recommendations, Speech recognition, Face detection, Protein](https://reader034.vdocuments.mx/reader034/viewer/2022042108/5e88a05c3c7f457f6a39cd43/html5/thumbnails/43.jpg)
E.g., Logistic Regression
Goal: Find linear decision boundary• Parameters to learn are feature weights and offset • Nice probabilistic interpretation• Covered in more detail later in course
Obtain Raw Data
Feature Extraction
Predict
Evaluation
Supervised Learning
![Page 44: Overview - edX · 2015-06-29 · Distributed Computing Data Machine Learning ML is Applied Everywhere E.g., Personalized recommendations, Speech recognition, Face detection, Protein](https://reader034.vdocuments.mx/reader034/viewer/2022042108/5e88a05c3c7f457f6a39cd43/html5/thumbnails/44.jpg)
How can we evaluate the quality of our classifier?We want good predictions on unobserved data• ’Generalization’ abilityAccuracy on training data is overly optimistic since classifier has already learned from it• We might be ‘overfitting’
Classification Pipeline
training set classifier
Obtain Raw Data
Feature Extraction
Predict
Evaluation
Supervised Learning
![Page 45: Overview - edX · 2015-06-29 · Distributed Computing Data Machine Learning ML is Applied Everywhere E.g., Personalized recommendations, Speech recognition, Face detection, Protein](https://reader034.vdocuments.mx/reader034/viewer/2022042108/5e88a05c3c7f457f6a39cd43/html5/thumbnails/45.jpg)
Fitting training data does not guarantee generalization, e.g., lookup tableLeft: perfectly fits training samples, but it is complex / overfitting Right: misclassifies a few points, but simple / generalizesOccam’s razor
Overfitting and Generalization
Obtain Raw Data
Feature Extraction
Predict
Evaluation
Supervised Learning
![Page 46: Overview - edX · 2015-06-29 · Distributed Computing Data Machine Learning ML is Applied Everywhere E.g., Personalized recommendations, Speech recognition, Face detection, Protein](https://reader034.vdocuments.mx/reader034/viewer/2022042108/5e88a05c3c7f457f6a39cd43/html5/thumbnails/46.jpg)
How can we evaluate the quality of our classifier?Idea: Create test set to simulate unobserved data
Classification Pipeline
training set classifier
Obtain Raw Data
Feature Extraction
Predict
Evaluation
Supervised Learning
![Page 47: Overview - edX · 2015-06-29 · Distributed Computing Data Machine Learning ML is Applied Everywhere E.g., Personalized recommendations, Speech recognition, Face detection, Protein](https://reader034.vdocuments.mx/reader034/viewer/2022042108/5e88a05c3c7f457f6a39cd43/html5/thumbnails/47.jpg)
Evaluation: Split dataset into training / testing datasets• Train on training set (don’t expose test set to classifier)• Make predictions on test set (ignoring test labels)• Compare test predictions with underlying test labels
Classification Pipeline
training set classifier
full dataset
test set accuracy
Obtain Raw Data
Feature Extraction
Predict
Evaluation
Supervised Learning
![Page 48: Overview - edX · 2015-06-29 · Distributed Computing Data Machine Learning ML is Applied Everywhere E.g., Personalized recommendations, Speech recognition, Face detection, Protein](https://reader034.vdocuments.mx/reader034/viewer/2022042108/5e88a05c3c7f457f6a39cd43/html5/thumbnails/48.jpg)
Classification Pipeline
training set classifier
full dataset
test set accuracy
Obtain Raw Data
Feature Extraction
Predict
Evaluation
Supervised Learning
Evaluation: Split dataset into training / testing datasets • Various ways to compare predicted and true labels • Evaluation criterion is called a ‘loss’ function • Accuracy (or 0-1 loss) is common for classification
![Page 49: Overview - edX · 2015-06-29 · Distributed Computing Data Machine Learning ML is Applied Everywhere E.g., Personalized recommendations, Speech recognition, Face detection, Protein](https://reader034.vdocuments.mx/reader034/viewer/2022042108/5e88a05c3c7f457f6a39cd43/html5/thumbnails/49.jpg)
new entity
prediction
Classification Pipeline
Predict: Final classifier can then be used to make predictions on future observations, e.g., new emails we receive
training set classifier
full dataset
test set accuracy
Obtain Raw Data
Feature Extraction
Predict
Evaluation
Supervised Learning
![Page 50: Overview - edX · 2015-06-29 · Distributed Computing Data Machine Learning ML is Applied Everywhere E.g., Personalized recommendations, Speech recognition, Face detection, Protein](https://reader034.vdocuments.mx/reader034/viewer/2022042108/5e88a05c3c7f457f6a39cd43/html5/thumbnails/50.jpg)
Linear Algebra Review
![Page 51: Overview - edX · 2015-06-29 · Distributed Computing Data Machine Learning ML is Applied Everywhere E.g., Personalized recommendations, Speech recognition, Face detection, Protein](https://reader034.vdocuments.mx/reader034/viewer/2022042108/5e88a05c3c7f457f6a39cd43/html5/thumbnails/51.jpg)
Matrices
Notation:• Matrices are denoted by bold uppercase letters• Aij denotes the entry in ith row and jth column• If A is n × m, it has n rows an m columns• If A is n × m, then A
�
���
3.3 5.3 4.51.0 4.5 3.46.3 1.0 2.23.6 4.7 8.9
�
���
A11
A32
� Rn�m
A =A matrix is a 2-dimensional array4 × 3
![Page 52: Overview - edX · 2015-06-29 · Distributed Computing Data Machine Learning ML is Applied Everywhere E.g., Personalized recommendations, Speech recognition, Face detection, Protein](https://reader034.vdocuments.mx/reader034/viewer/2022042108/5e88a05c3c7f457f6a39cd43/html5/thumbnails/52.jpg)
Vectors
Notation:• Vectors are denoted by bold lowercase letters• ai denotes the ith entry• If a is m dimensional, then a
�
���
3.31.06.33.6
�
���
a2
a =A vector is a matrix with many rows and one column
� Rm
![Page 53: Overview - edX · 2015-06-29 · Distributed Computing Data Machine Learning ML is Applied Everywhere E.g., Personalized recommendations, Speech recognition, Face detection, Protein](https://reader034.vdocuments.mx/reader034/viewer/2022042108/5e88a05c3c7f457f6a39cd43/html5/thumbnails/53.jpg)
TransposeSwap the rows and columns of a matrixProperties of matrix transposes:• Aij = (A⊤)ji
• If A is n × m, then A⊤ is m × n
3 × 2 2 × 3
�
�341
�
� =��3 4 1
��
�1 46 13 5
�
� =��1 6 34 1 5
�
3 × 1 1 × 3
A12
(A⊤)21
![Page 54: Overview - edX · 2015-06-29 · Distributed Computing Data Machine Learning ML is Applied Everywhere E.g., Personalized recommendations, Speech recognition, Face detection, Protein](https://reader034.vdocuments.mx/reader034/viewer/2022042108/5e88a05c3c7f457f6a39cd43/html5/thumbnails/54.jpg)
Addition and SubtractionThese are element-wise operations
�3 56 1
�+
�4 58 12
�=
�3+ 4 5+ 56+ 8 1+ 12
�
=
�7 1014 13
�Addition:
Subtraction:�51
��
�43
�=
�5� 41� 3
�
=
�1
�2
�
![Page 55: Overview - edX · 2015-06-29 · Distributed Computing Data Machine Learning ML is Applied Everywhere E.g., Personalized recommendations, Speech recognition, Face detection, Protein](https://reader034.vdocuments.mx/reader034/viewer/2022042108/5e88a05c3c7f457f6a39cd43/html5/thumbnails/55.jpg)
Addition and SubtractionThe matrices must have the same dimensions
�3 5 46 1 2
�+
�4 5 18 12 9
�=
�7 10 514 13 11
�
�3 5 46 1 2
�+
�4 58 12
�=
�51
��
�43
�=
�1
�2
�
![Page 56: Overview - edX · 2015-06-29 · Distributed Computing Data Machine Learning ML is Applied Everywhere E.g., Personalized recommendations, Speech recognition, Face detection, Protein](https://reader034.vdocuments.mx/reader034/viewer/2022042108/5e88a05c3c7f457f6a39cd43/html5/thumbnails/56.jpg)
Matrix Scalar MultiplicationWe multiply each matrix element by the scalar value
3��3 5 46 1 2
�=
�9 15 1218 3 6
�
�0.5��38
�=
��1.5�4
�
![Page 57: Overview - edX · 2015-06-29 · Distributed Computing Data Machine Learning ML is Applied Everywhere E.g., Personalized recommendations, Speech recognition, Face detection, Protein](https://reader034.vdocuments.mx/reader034/viewer/2022042108/5e88a05c3c7f457f6a39cd43/html5/thumbnails/57.jpg)
Scalar ProductA function that maps two vectors to a scalar
Performs pairwise multiplication of vector elements
1� 4+ 4� 2+ 3� (�7) = �9
�
�143
�
� ·
�
�42
�7
�
� = �9
![Page 58: Overview - edX · 2015-06-29 · Distributed Computing Data Machine Learning ML is Applied Everywhere E.g., Personalized recommendations, Speech recognition, Face detection, Protein](https://reader034.vdocuments.mx/reader034/viewer/2022042108/5e88a05c3c7f457f6a39cd43/html5/thumbnails/58.jpg)
Scalar ProductA function that maps two vectors to a scalar
Performs pairwise multiplication of vector elements
1� 4+ 4� 2+ 3� (�7) = �9
�
�143
�
� ·
�
�42
�7
�
� = �9
![Page 59: Overview - edX · 2015-06-29 · Distributed Computing Data Machine Learning ML is Applied Everywhere E.g., Personalized recommendations, Speech recognition, Face detection, Protein](https://reader034.vdocuments.mx/reader034/viewer/2022042108/5e88a05c3c7f457f6a39cd43/html5/thumbnails/59.jpg)
Scalar ProductA function that maps two vectors to a scalar
Performs pairwise multiplication of vector elements
The two vectors must be the same dimension
Also known as dot product or inner product
1� 4+ 4� 2+ 3� (�7) = �9
�
�143
�
� ·
�
�42
�7
�
� = �9
![Page 60: Overview - edX · 2015-06-29 · Distributed Computing Data Machine Learning ML is Applied Everywhere E.g., Personalized recommendations, Speech recognition, Face detection, Protein](https://reader034.vdocuments.mx/reader034/viewer/2022042108/5e88a05c3c7f457f6a39cd43/html5/thumbnails/60.jpg)
Matrix-Vector MultiplicationInvolves repeated scalar products
�1 4 36 1 2
� �
�42
�7
�
� =
��912
�
1� 4+ 4� 2+ 3� (�7) = �9
![Page 61: Overview - edX · 2015-06-29 · Distributed Computing Data Machine Learning ML is Applied Everywhere E.g., Personalized recommendations, Speech recognition, Face detection, Protein](https://reader034.vdocuments.mx/reader034/viewer/2022042108/5e88a05c3c7f457f6a39cd43/html5/thumbnails/61.jpg)
Matrix-Vector MultiplicationInvolves repeated scalar products
�1 4 36 1 2
� �
�42
�7
�
� =
��912
�
1� 4+ 4� 2+ 3� (�7) = �9
6� 4+ 1� 2+ 2� (�7) = 12
![Page 62: Overview - edX · 2015-06-29 · Distributed Computing Data Machine Learning ML is Applied Everywhere E.g., Personalized recommendations, Speech recognition, Face detection, Protein](https://reader034.vdocuments.mx/reader034/viewer/2022042108/5e88a05c3c7f457f6a39cd43/html5/thumbnails/62.jpg)
Matrix-Vector Multiplication
� � �
�
�
� =
� �A w y
n × m m × 1 n × 1
ith row
yi equals scalar product between ith
row of A and w
To perform inner products, # columns in A must equal # rows of w
We repeat for each row of A, so if A has n rows, so does y
![Page 63: Overview - edX · 2015-06-29 · Distributed Computing Data Machine Learning ML is Applied Everywhere E.g., Personalized recommendations, Speech recognition, Face detection, Protein](https://reader034.vdocuments.mx/reader034/viewer/2022042108/5e88a05c3c7f457f6a39cd43/html5/thumbnails/63.jpg)
Scalar Product Revisited
Vectors assumed to be in column form (many rows, one column)
Transposed vectors are row vectors
Common notation for scalar product: x⊤w
� � �
�
�
� =
� �x⊤ w y
�
�� �� �
=��
scalar product
1 × m m × 1 scalar (1 × 1)
Special case of Matrix-Vector Multiplication
![Page 64: Overview - edX · 2015-06-29 · Distributed Computing Data Machine Learning ML is Applied Everywhere E.g., Personalized recommendations, Speech recognition, Face detection, Protein](https://reader034.vdocuments.mx/reader034/viewer/2022042108/5e88a05c3c7f457f6a39cd43/html5/thumbnails/64.jpg)
Matrix-Matrix MultiplicationAlso involves several scalar products
�9 3 54 1 2
� �
�1 23 �52 3
�
� =
�28 1811 9
�
9� 1+ 3� 3+ 5� 2 = 28
![Page 65: Overview - edX · 2015-06-29 · Distributed Computing Data Machine Learning ML is Applied Everywhere E.g., Personalized recommendations, Speech recognition, Face detection, Protein](https://reader034.vdocuments.mx/reader034/viewer/2022042108/5e88a05c3c7f457f6a39cd43/html5/thumbnails/65.jpg)
Matrix-Matrix MultiplicationAlso involves several scalar products
�9 3 54 1 2
� �
�1 23 �52 3
�
� =
�28 1811 9
�
9� 1+ 3� 3+ 5� 2 = 28
9� 2+ 3� (�5) + 5� 3 = 18
![Page 66: Overview - edX · 2015-06-29 · Distributed Computing Data Machine Learning ML is Applied Everywhere E.g., Personalized recommendations, Speech recognition, Face detection, Protein](https://reader034.vdocuments.mx/reader034/viewer/2022042108/5e88a05c3c7f457f6a39cd43/html5/thumbnails/66.jpg)
Matrix-Matrix MultiplicationAlso involves several scalar products
�9 3 54 1 2
� �
�1 23 �52 3
�
� =
�28 1811 9
�
9� 1+ 3� 3+ 5� 2 = 28
9� 2+ 3� (�5) + 5� 3 = 18
![Page 67: Overview - edX · 2015-06-29 · Distributed Computing Data Machine Learning ML is Applied Everywhere E.g., Personalized recommendations, Speech recognition, Face detection, Protein](https://reader034.vdocuments.mx/reader034/viewer/2022042108/5e88a05c3c7f457f6a39cd43/html5/thumbnails/67.jpg)
Matrix-Matrix MultiplicationA B C
m × p n × m n × p
� � �
�
�
� =
� �
To perform inner products, # columns in A must equal
# rows of B
Cij is scalar product of ith row of A and
jth column of B
We repeat for each row of A, so if A has n rows, so does C
ith row
jth col
We repeat for each column of B, so if B has p
columns, so does C
![Page 68: Overview - edX · 2015-06-29 · Distributed Computing Data Machine Learning ML is Applied Everywhere E.g., Personalized recommendations, Speech recognition, Face detection, Protein](https://reader034.vdocuments.mx/reader034/viewer/2022042108/5e88a05c3c7f457f6a39cd43/html5/thumbnails/68.jpg)
Matrix-Matrix MultiplicationA B C
m × p n × m n × p
� � �
�
�
� =
� �
Associative, i.e., (AB)C = A(BC)
Not commutative, i.e., AB ≠ BA
![Page 69: Overview - edX · 2015-06-29 · Distributed Computing Data Machine Learning ML is Applied Everywhere E.g., Personalized recommendations, Speech recognition, Face detection, Protein](https://reader034.vdocuments.mx/reader034/viewer/2022042108/5e88a05c3c7f457f6a39cd43/html5/thumbnails/69.jpg)
Outer Product
Cij is “inner product” of ith entry of x and jth entry of w
� � �
�
�
� =
� �x w⊤ C
�
�� �� �
=��
n × 1 1 × m n × m
� � �
�
�
� =
� �Special case of Matrix-Matrix
Multiplication involving two vectors
� � �
�
�
� =
� �
![Page 70: Overview - edX · 2015-06-29 · Distributed Computing Data Machine Learning ML is Applied Everywhere E.g., Personalized recommendations, Speech recognition, Face detection, Protein](https://reader034.vdocuments.mx/reader034/viewer/2022042108/5e88a05c3c7f457f6a39cd43/html5/thumbnails/70.jpg)
Identity Matrix
One is the scalar multiplication identity, i.e., c × 1 = c
In is the n × n identity matrix, i.e., InA = A and AIm = A for any n × m matrix AIdentity matrices are square, with ones on the diagonal entries
�9 3 54 1 2
� �
�1 0 00 1 00 0 1
�
� =
�9 3 54 1 2
�
![Page 71: Overview - edX · 2015-06-29 · Distributed Computing Data Machine Learning ML is Applied Everywhere E.g., Personalized recommendations, Speech recognition, Face detection, Protein](https://reader034.vdocuments.mx/reader034/viewer/2022042108/5e88a05c3c7f457f6a39cd43/html5/thumbnails/71.jpg)
Inverse Matrix
1/c is the scalar inverse, i.e., c × 1/c = 1
Multiplying a matrix by its inverse results in the identity matrix
• For an n × n matrix A, A-1 denotes its inverse (when it exists)
• AA-1 = A-1A = In
Only a square matrix (when n = m) can have an inverse
![Page 72: Overview - edX · 2015-06-29 · Distributed Computing Data Machine Learning ML is Applied Everywhere E.g., Personalized recommendations, Speech recognition, Face detection, Protein](https://reader034.vdocuments.mx/reader034/viewer/2022042108/5e88a05c3c7f457f6a39cd43/html5/thumbnails/72.jpg)
The magnitude / length of a scalar is its absolute value
Vector norms generalize this idea for vectors
The Euclidean norm for is denoted by
• sd
• Equals absolute value when m=1
• Related to scalar product:
Euclidean Norm for Vectors
�x�22 = x�x
�x�2
�x�2 =�x21 + x22 + . . . + x2m
x � Rm
![Page 73: Overview - edX · 2015-06-29 · Distributed Computing Data Machine Learning ML is Applied Everywhere E.g., Personalized recommendations, Speech recognition, Face detection, Protein](https://reader034.vdocuments.mx/reader034/viewer/2022042108/5e88a05c3c7f457f6a39cd43/html5/thumbnails/73.jpg)
Big O Notation for Time and Space Complexity
![Page 74: Overview - edX · 2015-06-29 · Distributed Computing Data Machine Learning ML is Applied Everywhere E.g., Personalized recommendations, Speech recognition, Face detection, Protein](https://reader034.vdocuments.mx/reader034/viewer/2022042108/5e88a05c3c7f457f6a39cd43/html5/thumbnails/74.jpg)
Big O NotationDescribes how algorithms respond to changes in input size • Both in terms of processing time and space requirements • We refer to complexity and Big O notation synonymously
Required space proportional to units of storage • Typically 8 bytes to store a floating point number
Required time proportional to number of ‘basic operations’ • Arithmetic operations (+, −, ×, /), logical tests (<, >, ==)
![Page 75: Overview - edX · 2015-06-29 · Distributed Computing Data Machine Learning ML is Applied Everywhere E.g., Personalized recommendations, Speech recognition, Face detection, Protein](https://reader034.vdocuments.mx/reader034/viewer/2022042108/5e88a05c3c7f457f6a39cd43/html5/thumbnails/75.jpg)
Big O NotationNotation: • Can describe an algorithm’s time or space complexity
Informal definition: does not grow faster than
Formal definition:
Ignores constants and lower-order terms • For large enough , these terms won’t matter • E.g.,
f(x) = O(g(x)) �� |f(x)| � C|g(x)| �x > N
f(x) = O(g(x)) �� |f(x)| � C|g(x)| � x > N
x2 + 3x � Cx2 � x > N
f g
x2 + 3x � Cx2 � x > N
![Page 76: Overview - edX · 2015-06-29 · Distributed Computing Data Machine Learning ML is Applied Everywhere E.g., Personalized recommendations, Speech recognition, Face detection, Protein](https://reader034.vdocuments.mx/reader034/viewer/2022042108/5e88a05c3c7f457f6a39cd43/html5/thumbnails/76.jpg)
E.g., O(1) Complexity
Constant time algorithms perform the same number of operations every time they’re called • E.g., performing a fixed number of arithmetic operations
Similarly, constant space algorithms require a fixed amount of storage every time they’re called • E.g., storing the results of a fixed number of arithmetic
operations
![Page 77: Overview - edX · 2015-06-29 · Distributed Computing Data Machine Learning ML is Applied Everywhere E.g., Personalized recommendations, Speech recognition, Face detection, Protein](https://reader034.vdocuments.mx/reader034/viewer/2022042108/5e88a05c3c7f457f6a39cd43/html5/thumbnails/77.jpg)
E.g., O(n) ComplexityLinear time algorithms perform a number of operations proportional to the number of inputs • E.g., adding two n-dimensional vectors requires O(n)
arithmetic operations
Similarly, linear space algorithms require storage proportional to the size of the inputs • E.g., adding two n-dimensional vectors results in a new n-
dimensional vector which requires O(n) storage
![Page 78: Overview - edX · 2015-06-29 · Distributed Computing Data Machine Learning ML is Applied Everywhere E.g., Personalized recommendations, Speech recognition, Face detection, Protein](https://reader034.vdocuments.mx/reader034/viewer/2022042108/5e88a05c3c7f457f6a39cd43/html5/thumbnails/78.jpg)
E.g., O(n2) ComplexityQuadratic time algorithms perform a number of operations proportional to the square of the number of inputs • E.g., outer product of two n-dimensional vectors requires
O(n2) multiplication operations (one per each entry of the resulting matrix)
Similarly, quadratic space algorithms require storage proportional to the square of the size of the inputs • E.g., outer product of two n-dimensional vectors requires
O(n2) storage (one per each entry of the resulting matrix)
![Page 79: Overview - edX · 2015-06-29 · Distributed Computing Data Machine Learning ML is Applied Everywhere E.g., Personalized recommendations, Speech recognition, Face detection, Protein](https://reader034.vdocuments.mx/reader034/viewer/2022042108/5e88a05c3c7f457f6a39cd43/html5/thumbnails/79.jpg)
Time and Space Complexity Can Differ
Inner product of two n-dimensional vectors • O(n) time complexity to multiply n pairs of numbers • O(1) space complexity to store result (which is a scalar)
Matrix inversion of an n × n matrix • O(n3) time complexity to perform inversion • O(n2) space complexity to store result
![Page 80: Overview - edX · 2015-06-29 · Distributed Computing Data Machine Learning ML is Applied Everywhere E.g., Personalized recommendations, Speech recognition, Face detection, Protein](https://reader034.vdocuments.mx/reader034/viewer/2022042108/5e88a05c3c7f457f6a39cd43/html5/thumbnails/80.jpg)
E.g., Matrix-Vector MultiplyGoal: multiply an n × m matrix with an m × 1 vector
Computing result takes O(nm) time • There are n entries in the resulting vector • Each entry computed via dot product between two m-
dimensional vectors (a row of input matrix and input vector)
Storing result takes O(n) space • The result is an n-dimensional vector
![Page 81: Overview - edX · 2015-06-29 · Distributed Computing Data Machine Learning ML is Applied Everywhere E.g., Personalized recommendations, Speech recognition, Face detection, Protein](https://reader034.vdocuments.mx/reader034/viewer/2022042108/5e88a05c3c7f457f6a39cd43/html5/thumbnails/81.jpg)
E.g., Matrix-Matrix MultiplyGoal: multiply an n × m matrix with an m × p matrix
Computing result takes O(npm) time • There are np entries in the resulting matrix • Each entry computed via dot product between two m-
dimensional vectors
Storing result takes O(np) space • The result is an n × p matrix