anomaly detection - new york machine learning

© 2014 MapR Technologies 1

© MapR Technologies, confidential

How to Find What You Didn’t Know to Look For

Anomaly Detection

October 14, 2014


Anomaly Detection: How To Find What You Didn’t Know to Look For

Ted Dunning, Chief Applications Architect MapR Technologies

Email [email protected] [email protected]

Twitter @Ted_Dunning

Ellen Friedman, Consultant and Commentator

Email [email protected]

Twitter @Ellen_Friedman

mailto:[email protected]




e-book available courtesy of MapR

http://bit.ly/1jQ9QuL

A New Look at Anomaly Detectionby Ted Dunning and Ellen Friedman © June 2014 (published by O’Reilly)




Practical Machine Learning series (O’Reilly)

• Machine learning is becoming mainstream• Need pragmatic approaches that take into account real world

business settings:– Time to value– Limited resources– Availability of data– Expertise and cost of team to develop and to maintain system

• Look for approaches with big benefits for the effort expended


Anomaly Detection


Who Needs Anomaly Detection?

Utility providers using smart meters



Feedback from manufacturing assembly lines



Monitoring data traffic on communication networks


What is Anomaly Detection?

• The goal is to discover rare events – especially those that shouldn’t have happened

• Find a problem before other people see it– especially before it causes a problem for customers

• Why is this a challenge?– I don’t know what an anomaly looks like (yet)


Spot the Anomaly


Spot the Anomaly

Looks pretty anomalous

to me


Spot the Anomaly

Will the real anomaly please stand up?


Basic idea:Find “normal” first


Steps in Anomaly Detection

• Build a model: Collect and process data for training a model• Use the machine learning model to determine what is the normal

pattern • Decide how far away from this normal pattern you’ll consider to

be anomalous• Use the AD model to detect anomalies in new data

– Methods such as clustering for discovery can be helpful


How hard is it to set an alert for anomalies?

Grey data is from normal events; x’s are anomalies.Where would you set the threshold?


Basic idea:Set adaptive thresholds


What Are We Really Doing

• We want action when something breaks (dies/falls over/otherwise gets in trouble)

• But action is expensive• So we don’t want too many false alarms• And we don’t want too many false negatives

• What’s the right threshold to set for alerts?– We need to trade off costs


A Second Look


A Second Look

99.9%-ile


New algorithm: t-digest


Online Summarizer

99.9%-ile

t

x > t ? Alarm !x

How Hard Can it Be?


Detecting Anomalies in Sporadic Events


Using t-Digest

• Apache Mahout uses t-digest as an on-line percentile estimator– very high accuracy for extreme tails– new in version Mahout v 0.9

• t-digest also available elsewhere– in streamlib (open source library on github)– standalone (github and Maven Central)

• What’s the big deal with anomaly detection?

• This looks like a solved problem


Already Done? Etsy Skyline?


What About This?


Model Delta Anomaly Detection

Online Summarizer

δ > t ?

99.9%-ile

t

Alarm !

Model

-

+ δ


The Real Inside Scoop

• The model-delta anomaly detector is really just a sum of random variables– the model we know about already– and a normally distributed error

• The output (delta) is (roughly) the log probability of the sum distribution (really δ2)

• Thinking about probability distributions is good

• But how do you handle AD in systems with sporadic events?


Spot the Anomaly

Anomaly?


Maybe not!


Where’s Waldo?

This is the real anomaly


Normal Isn’t Just Normal

• What we want is a model of what is normal

• What doesn’t fit the model is the anomaly

• For simple signals, the model can be simple …

• The real world is rarely so accommodating


We Do Windows


Windows on the World

• The set of windowed signals is a nice model of our original signal• Clustering can find the prototypes

– Fancier techniques available using sparse coding

• The result is a dictionary of shapes• New signals can be encoded by shifting, scaling and adding

shapes from the dictionary


Most Common Shapes (for EKG)


Reconstructed signal

Original signal

Reconstructed signal

Reconstructionerror

< 1 bit / sample


An Anomaly

Original technique for finding 1-d anomaly works against reconstruction error


Close-up of anomaly

Not what you want your heart to do.

And not what the model expects it to do.


A Different Kind of Anomaly



Online Summarizer

δ > t ?

99.9%-ile

t

Alarm !

Model

-

+ δ


The Real Inside Scoop

• The model-delta anomaly detector is really just a sum of random variables– the model we know about already– and a normally distributed error

• The output (delta) is (roughly) the log probability of the sum distribution (really δ2)

• Thinking about probability distributions is good


Anomalies among sporadic events


Sporadic Web Traffic to an e-Business Site

It’s important to know if traffic is stopped or delayed because of a problem…

But visits to site normally come at varying intervals.

How long after the last event should you begin to worry?


Sporadic Web Traffic to an e-Business Site

It’s important to know if traffic is stopped or delayed because of a problem…

But visits to site normally come at varying intervals.

And how do you let your CEO sleep through the night?


Basic idea:Time interval between events is how to

convert to something useful you can measure


Sporadic Events: Finding Normal and Anomalous Patterns

• Time between intervals is much more usable than absolute times

• Counts don’t link as directly to probability models

• Time interval is log ρ

• This is a big deal


Event Stream (timing)

• Events of various types arrive at irregular intervals– we can assume Poisson distribution

• The key question is whether frequency has changed relative to expected values– This shows up as a change in interval

• Want alert as soon as possible


Converting Event Times to Anomaly

99.9%-ile

99.99%-ile


But in the real world, event rates often change


Time Intervals Are Key to Modeling Sporadic Events


Model-Scaled Intervals Solve the Problem



Online Summarizer

δ > t ?

99.9%-ile

t

Alarm !

Model

-

+ δ

log p


Slipped Week: Simple Rate Predictor


Poisson Distribution

• Time between events is exponentially distributed

• This means that long delays are exponentially rare

• If we know λ we can select a good threshold– or we can pick a threshold empirically


Seasonality Poses a Challenge


Something more is needed …


We need a better rate predictor…


A New Rate Predictor for Sporadic Events


Improved Prediction with Adaptive Modeling


Anomaly Detection + Classification Useful Pair

• Use the AD model to detect anomalies in new data– Methods such as clustering for discovery can be helpful

• Once you have well-defined models in your system, you may also want to use classification to tag those

• Continue to use the AD model to find new anomalies


Recap (out of order)

• Anomaly detection is best done with a probability model• -log p is a good way to convert to anomaly measure• Adaptive quantile estimation (t-digest) works for auto-setting

thresholds


Recap

• Different systems require different models• Continuous time-series

– sparse coding to build signal model

• Events in time– rate model base on variable rate Poisson– segregated rate model

• Events with labels– language modeling– hidden Markov models


Why Use Anomaly Detection?


Keep in mind…

• Model normal, then find anomalies

• t-digest for adaptive threshold

• Probabilistic models for complex patterns

-


Keep in mind…

• Time intervals are key for sporadic events

• Complex time shift to predict rate with seasonality

• Sequence of events reveals phishing attack


e-book available courtesy of MapR


A New Look at Anomaly Detectionby Ted Dunning and Ellen Friedman © June 2014 (published by O’Reilly)




Thank you for coming today!


Sandbox

anomaly detection - new york machine learning

Technology

model delta anomaly

anomaly detectionhow

modeldelta anomaly detector

ad model

machine learning model

new look

onlinesummarizer t

onlinesummarizerx t