v like velocity, predicting in real-time with azure ml

54
V like Velocity, Predicting in Real-Time with Azure ML Barbara Fusinska @BasiaFusinska

Upload: barbara-fusinska

Post on 05-Apr-2017

120 views

Category:

Data & Analytics


2 download

TRANSCRIPT

Page 1: V like Velocity, Predicting in Real-Time with Azure ML

V like Velocity, Predicting in Real-Time with Azure MLBarbara Fusinska@BasiaFusinska

Page 2: V like Velocity, Predicting in Real-Time with Azure ML

About me

ProgrammerMachine Learning

Data Solutions Architect@BasiaFusinska

Page 3: V like Velocity, Predicting in Real-Time with Azure ML

Agenda• Aircraft Predictive maintenance

• Use Case description• Stream processing challenges• Machine Learning Model

• Solution Architecture• Azure Architecture• Stream processing• Applying intelligence to streams

Page 4: V like Velocity, Predicting in Real-Time with Azure ML

Aerospace Predictive Maintenance Engines Sensors Analysis

Page 5: V like Velocity, Predicting in Real-Time with Azure ML

Will the device fail in 2 weeks, 4 weeks or will

not fail during this time?

Use CaseWhen will the device

break?

Will the device fail in the next 2 weeks?

Page 6: V like Velocity, Predicting in Real-Time with Azure ML

VelocityHow is it different?

Page 7: V like Velocity, Predicting in Real-Time with Azure ML

Big Data 4Vs

• Untrusted• Uncleansed

• Speed of generation• Rate of analysis

• Unstructured• Semi-structured• Structured

• Click stream• Active/passive sensors• Logs• Events• Speech• Social media• Traditional

Volume Variety

VeracityVelocity

Page 8: V like Velocity, Predicting in Real-Time with Azure ML

Big Data & AzureIntelligence

Dashboards & Visualizations

Information Management

Big Data Stores Machine Learning and Analytics

CortanaEvent HubHDInsight (Hadoop and Spark)

Stream Analytics

Data Sources

Apps

Sensors and devices

Data Intelligence Action

People

Automated Systems

Apps

Web

Mobile

Bots

Bot Framework

SQL Data WarehouseData Catalog

Data Lake Analytics

Data Factory Machine LearningData Lake Store Blob Store

Cognitive Services

Power BI

Page 9: V like Velocity, Predicting in Real-Time with Azure ML

Batch vs Real-time

Page 10: V like Velocity, Predicting in Real-Time with Azure ML

Stream processing challenges• Data Ingestion• Stream Processing• Applying intelligence to the

stream• Aggregations• Data sink

Page 11: V like Velocity, Predicting in Real-Time with Azure ML

Machine Learning

Machine LearningModel

Model Training

PublishedMachine Learning

Model

Prediction

Training data

Publish model

Test stream

Scores

Page 12: V like Velocity, Predicting in Real-Time with Azure ML

Machine Learning workflowData

preparation Data split

Machine Learning algorithm

Trained model Score

Clean data

Training data Test data

Page 13: V like Velocity, Predicting in Real-Time with Azure ML

Azure Machine Learning ServiceData -> Predictive model -> Operational web API in minutes

Blobs and TablesHadoop (HDInsight)Relational DB (Azure SQL DB)

Data Clients

Model is now a web service that is callable

Monetize the API through our marketplace

API

Integrated development environment for Machine Learning

ML STUDIO

Page 14: V like Velocity, Predicting in Real-Time with Azure ML

Azure Machine Learning StudioIncome predictionDemo

Page 15: V like Velocity, Predicting in Real-Time with Azure ML

Will the device break in the next 2 weeks?

cycle 11 cycle 12 cycle 13 cycle 14 cycle 150

1

2

3

4

5

6

Sensor reads

s1 s2 s3

Device37

Cycle s1 ... s21 failed

1 518.67 23.419 0

2 518.67 23.4236 0

3 518.67 23.3442 0

...

134 518.67 23.1295 0

135 518.67 23.4085 1

(s, avg, stdev)

Page 16: V like Velocity, Predicting in Real-Time with Azure ML

https://gallery.cortanaintelligence.com/SolutionTemplate/Predictive-Maintenance-for-Aerospace-1https://gallery.cortanaintelligence.com/Collection/Predictive-Maintenance-Template-3

Page 17: V like Velocity, Predicting in Real-Time with Azure ML
Page 18: V like Velocity, Predicting in Real-Time with Azure ML
Page 19: V like Velocity, Predicting in Real-Time with Azure ML
Page 20: V like Velocity, Predicting in Real-Time with Azure ML

Aircraft Predictive Maintenan

ceUse Case

Page 21: V like Velocity, Predicting in Real-Time with Azure ML

Solution Architecture

Stream Analytics

Event Hub DocumentDB

Machine Learning Web AppML Model TrainingTraining data

Retrain

Page 22: V like Velocity, Predicting in Real-Time with Azure ML

Sensors data

simulationData generator

Page 23: V like Velocity, Predicting in Real-Time with Azure ML
Page 24: V like Velocity, Predicting in Real-Time with Azure ML

Event Hub configurationData Ingestion

Page 25: V like Velocity, Predicting in Real-Time with Azure ML
Page 26: V like Velocity, Predicting in Real-Time with Azure ML
Page 27: V like Velocity, Predicting in Real-Time with Azure ML
Page 28: V like Velocity, Predicting in Real-Time with Azure ML
Page 29: V like Velocity, Predicting in Real-Time with Azure ML
Page 30: V like Velocity, Predicting in Real-Time with Azure ML

DocumentDB configurationStoring Maintenance Predictions

Page 31: V like Velocity, Predicting in Real-Time with Azure ML
Page 32: V like Velocity, Predicting in Real-Time with Azure ML
Page 33: V like Velocity, Predicting in Real-Time with Azure ML
Page 34: V like Velocity, Predicting in Real-Time with Azure ML

Stream analytics configurationMaintenance prediction

Page 35: V like Velocity, Predicting in Real-Time with Azure ML
Page 36: V like Velocity, Predicting in Real-Time with Azure ML
Page 37: V like Velocity, Predicting in Real-Time with Azure ML
Page 38: V like Velocity, Predicting in Real-Time with Azure ML
Page 39: V like Velocity, Predicting in Real-Time with Azure ML
Page 40: V like Velocity, Predicting in Real-Time with Azure ML
Page 41: V like Velocity, Predicting in Real-Time with Azure ML

Stream Analytics – Source and SinkSELECT

DeviceId, result.ScoredLabels,cycle, seeting1, …, s1, …, s21,a1, …, a21, sd1, …, sd21

INTO maintenanceFROM predict

Page 42: V like Velocity, Predicting in Real-Time with Azure ML
Page 43: V like Velocity, Predicting in Real-Time with Azure ML

Aggregation in Stream Analytics

SELECT avg(s1) as a1FROM CallStreamGROUP BY id, TumblingWindow(minute, 1)

TIMESTAMP BY T

Page 44: V like Velocity, Predicting in Real-Time with Azure ML

Stream Analytics – AggregationsWITH aggregate AS (SELECT

id,avg(s1) AS a1, …, avg(s21) AS a21, stdev(s1) AS sd1, …, stdev(s21) AS sd21

FROM CallStream AS sGROUP BY id, TumblingWindow(minute, 1))

Page 45: V like Velocity, Predicting in Real-Time with Azure ML

Join in Stream AnalyticsSELECT c.s1, a.a1FROM CallStream cJOIN aggregate aON c.id = a.id

and DATEDIFF(minute, c, a)BETWEEN 0 and 1

Page 46: V like Velocity, Predicting in Real-Time with Azure ML

Stream Analytics – Join aggregationsWITH predict AS (SELECT

s.id, s.cycle, s.seeting1, …, s.s1, …, s.s21,a.a1, …, a.a21, a.sd1, …, a.sd21,predmain(s.cycle, s.seeting1, …, s.s1, …, s.s21,a.a1, …, a.a21, a.sd1, …, a.sd21) AS result

FROM CallStream AS sjoin aggregate AS aON s.id = a.idAND DATEDIFF(minute, s, a) BETWEEN 0 AND 1

)

Page 47: V like Velocity, Predicting in Real-Time with Azure ML

Monitoring DashboardReading Predictions from DocumentDB

Page 48: V like Velocity, Predicting in Real-Time with Azure ML
Page 49: V like Velocity, Predicting in Real-Time with Azure ML

Retraining scenarioDemo

Page 50: V like Velocity, Predicting in Real-Time with Azure ML
Page 51: V like Velocity, Predicting in Real-Time with Azure ML
Page 52: V like Velocity, Predicting in Real-Time with Azure ML

Scaling web services• Azure Classic Portal

• 20-200 concurrent requests• Azure ML optimization• Multiple endpoints for the

same web service

• New web services• Production web services• Pricing Plans• 1,000 – 50,000,000 request

per month

Page 53: V like Velocity, Predicting in Real-Time with Azure ML
Page 54: V like Velocity, Predicting in Real-Time with Azure ML

Keep in touch

BarbaraFusinska.com@BasiaFusinska