integrating matlab analytics into enterprise applications ...€¦ · messaging service •...
TRANSCRIPT
![Page 1: Integrating MATLAB Analytics into Enterprise Applications ...€¦ · Messaging Service • Reporting • Real ... • Training Models Connected Devices Continuous Data f(x) Stream](https://reader034.vdocuments.mx/reader034/viewer/2022043005/5f898b1d7b054432344f1875/html5/thumbnails/1.jpg)
1© 2015 The MathWorks, Inc.
Integrating MATLAB Analytics into Enterprise Applications
![Page 2: Integrating MATLAB Analytics into Enterprise Applications ...€¦ · Messaging Service • Reporting • Real ... • Training Models Connected Devices Continuous Data f(x) Stream](https://reader034.vdocuments.mx/reader034/viewer/2022043005/5f898b1d7b054432344f1875/html5/thumbnails/2.jpg)
2
Agenda
§ Example Problem
§ Access and Preprocess Data
§ Develop a Predictive Model
§ Integrate Analytics with Production Systems
§ Build a Dashboard to Visualize Results
![Page 3: Integrating MATLAB Analytics into Enterprise Applications ...€¦ · Messaging Service • Reporting • Real ... • Training Models Connected Devices Continuous Data f(x) Stream](https://reader034.vdocuments.mx/reader034/viewer/2022043005/5f898b1d7b054432344f1875/html5/thumbnails/3.jpg)
3
Example Problem – How’s my driving?
§ A group of MathWorks employees installed an OBD dongle in their car that monitors the on-board systems
§ Data is streamed to the cloud where it is aggregated and stored
§ I would like to use this data to score the driving habits of participants
![Page 4: Integrating MATLAB Analytics into Enterprise Applications ...€¦ · Messaging Service • Reporting • Real ... • Training Models Connected Devices Continuous Data f(x) Stream](https://reader034.vdocuments.mx/reader034/viewer/2022043005/5f898b1d7b054432344f1875/html5/thumbnails/4.jpg)
4
Fleet Analytics Architecture
MATLAB
Analytics Development
MATLAB Production Server
MATLABAnalytics
Business Decisions
MATLABCompiler
SDKAlgorithm
Developers
Storage Layer
End Users
API Gateway
AWS Lambda
Kafka Connector
Business Systems
Edge Devices
Production System
![Page 5: Integrating MATLAB Analytics into Enterprise Applications ...€¦ · Messaging Service • Reporting • Real ... • Training Models Connected Devices Continuous Data f(x) Stream](https://reader034.vdocuments.mx/reader034/viewer/2022043005/5f898b1d7b054432344f1875/html5/thumbnails/5.jpg)
5
Data Analytics Workflow
Integrate Analytics with Production Systems
Desktop Apps
Enterprise Scale Systems
Embedded Devices and Hardware
Files
Databases
Sensors
Access and Explore Data
Develop Predictive Models
Model Creation e.g. Machine Learning
Model Validation
Parameter Optimization
Preprocess Data
Working with Messy Data
Data Reduction/ Transformation
Feature Extraction
![Page 6: Integrating MATLAB Analytics into Enterprise Applications ...€¦ · Messaging Service • Reporting • Real ... • Training Models Connected Devices Continuous Data f(x) Stream](https://reader034.vdocuments.mx/reader034/viewer/2022043005/5f898b1d7b054432344f1875/html5/thumbnails/6.jpg)
6
Access and Preprocess Data
MATLAB
Analytics Development
Business Decisions
AlgorithmDevelopers
Storage Layer
End UsersBusiness Systems
Edge Devices
Production System
![Page 7: Integrating MATLAB Analytics into Enterprise Applications ...€¦ · Messaging Service • Reporting • Real ... • Training Models Connected Devices Continuous Data f(x) Stream](https://reader034.vdocuments.mx/reader034/viewer/2022043005/5f898b1d7b054432344f1875/html5/thumbnails/7.jpg)
7
The Data: Timestamped messages with JSON encoding
Key Value
{"vehicles_id": {"$oid":"55a3fd0069702d5b41000000"}
}
{ "time” : {"$date":"2015-07-13T18:01:35.000Z"},"kc” : 1975.0, "kff1225" : 100.65293, "kff125a" : 110.36619,…..
}
{ "vehicles_id":{"$oid":"55a3fe3569702d5c5c000020"}
}
{"time":{"$date":"2015-07-13T18:01:53.000Z"},"kc” : 2000.0, "kff1225" : 109.65293, "kff125a" : 115.36619,…..
}
{ "vehicles_id":"$oid":"55a4193569702d115b000001"}
}
{ "time":{"$date":"2015-07-12T19:04:04.000Z"}"kc":2200.0, "kff1225" : 112.65293, "kff125a" : 112.36619,…..
}
![Page 8: Integrating MATLAB Analytics into Enterprise Applications ...€¦ · Messaging Service • Reporting • Real ... • Training Models Connected Devices Continuous Data f(x) Stream](https://reader034.vdocuments.mx/reader034/viewer/2022043005/5f898b1d7b054432344f1875/html5/thumbnails/8.jpg)
8
Data Access and Preprocessing – Challenges
§ Data aggregation – Different sources (files, web, etc.)– Different types (images, text, audio, etc.)
§ Data clean up – Poorly formatted files – Irregularly sampled data – Redundant data, outliers, missing data etc.
§ Data specific processing – Signals: Smoothing, resampling,
denoising, Wavelet transforms, etc. – Images: Image registration, morphological
filtering, deblurring, etc.
§ Dealing with out of memory data (big data)
Challenges
Data preparation accounts for about 80% of the work of data scientists - Forbes
![Page 9: Integrating MATLAB Analytics into Enterprise Applications ...€¦ · Messaging Service • Reporting • Real ... • Training Models Connected Devices Continuous Data f(x) Stream](https://reader034.vdocuments.mx/reader034/viewer/2022043005/5f898b1d7b054432344f1875/html5/thumbnails/9.jpg)
10
Access a Sample of Data and Develop a Preprocessing Function
![Page 10: Integrating MATLAB Analytics into Enterprise Applications ...€¦ · Messaging Service • Reporting • Real ... • Training Models Connected Devices Continuous Data f(x) Stream](https://reader034.vdocuments.mx/reader034/viewer/2022043005/5f898b1d7b054432344f1875/html5/thumbnails/10.jpg)
11
Develop a Predictive Model
MATLAB
Analytics Development
Business Decisions
AlgorithmDevelopers
Storage Layer
End UsersBusiness Systems
Edge Devices
Production SystemMDCS
![Page 11: Integrating MATLAB Analytics into Enterprise Applications ...€¦ · Messaging Service • Reporting • Real ... • Training Models Connected Devices Continuous Data f(x) Stream](https://reader034.vdocuments.mx/reader034/viewer/2022043005/5f898b1d7b054432344f1875/html5/thumbnails/11.jpg)
12
Develop a Predictive Model in MATLAB
Label events
Represent signals
Train model
Validate model
Scale up
![Page 12: Integrating MATLAB Analytics into Enterprise Applications ...€¦ · Messaging Service • Reporting • Real ... • Training Models Connected Devices Continuous Data f(x) Stream](https://reader034.vdocuments.mx/reader034/viewer/2022043005/5f898b1d7b054432344f1875/html5/thumbnails/12.jpg)
13
Integrate Analytics with Production Systems
MATLAB
Analytics Development
MATLAB Production Server
MATLABAnalytics
Business Decisions
MATLABCompiler
SDKAlgorithm
Developers
Storage Layer
End Users
API Gateway
AWS Lambda
Kafka Connector
Business Systems
Edge Devices
Production System
![Page 13: Integrating MATLAB Analytics into Enterprise Applications ...€¦ · Messaging Service • Reporting • Real ... • Training Models Connected Devices Continuous Data f(x) Stream](https://reader034.vdocuments.mx/reader034/viewer/2022043005/5f898b1d7b054432344f1875/html5/thumbnails/13.jpg)
14
A quick Intro to Stream Processing§ Batch Processing applies computation to a finite sized historical data set
that was acquired in the past
§ Stream Processing applies computation to an unbounded data set that is produced continuously
Messaging Service
• Reporting• Real Time
Decision Support
Dashboards
Alerts
Storage
Historical Data
Storage
Files
Configure Resources Schedule and Run Job Output Data
Storage
Files• Reporting• Data Exploration• Training Models
Connected Devices
Continuous Data
f(x)
Stream Analytics
![Page 14: Integrating MATLAB Analytics into Enterprise Applications ...€¦ · Messaging Service • Reporting • Real ... • Training Models Connected Devices Continuous Data f(x) Stream](https://reader034.vdocuments.mx/reader034/viewer/2022043005/5f898b1d7b054432344f1875/html5/thumbnails/14.jpg)
15
Stream processing exploits the fact that recent data tends to be more valuable
MDCS, Compiler with Hadoop/Spark
Stream Processing with MATLAB Production Server
Edge Processing
with MATLAB
Coder
Time critical decisions Queries on historical data, Model training Real time decisions
Valu
e of
dat
a to
dec
isio
n m
akin
g
TimeHistorical
Reactive
Actionable
Prev
entiv
e /
Pred
ictiv
e
Real-Time Seconds Minutes Hours Days Months
KinesisEvent Hub
![Page 15: Integrating MATLAB Analytics into Enterprise Applications ...€¦ · Messaging Service • Reporting • Real ... • Training Models Connected Devices Continuous Data f(x) Stream](https://reader034.vdocuments.mx/reader034/viewer/2022043005/5f898b1d7b054432344f1875/html5/thumbnails/15.jpg)
16
Event Time
Vehicle RPM Torque Fuel Flow
… … … … …
… … … … …
… … … … …
… … … … …
… … … … …
… … … … …
… … … … …
… … … … …
… … … … …
… … … … …
… … … … …
… … … … …
MATLAB Function
State
State
18:01:10 55a3fd 1975 100 110
18:10:30 55a3fe 2000 109 115
18:05:20 55a3fd 1980 105 105
18:10:45 55a3fd 2100 110 100
18:30:10 55a419 2000 100 110
18:35:20 55a419 1960 103 105
18:20:40 55a3fe 1970 112 104
18:39:30 55a419 2100 105 110
18:30:00 55a3fe 1980 110 113
18:30:50 55a3fe 2000 100 110
MATLAB Function
State
MATLAB Function
State
Input TableTime window Vehicle Score
… … …
18:00:00 18:10:00 55a3fd …
55a3fe …
55a419 …
18:10:00 18:20:00 55a3fd …
55a3fe …
55a419 …
18:20:00 18:30:00 55a3fd …
55a3fe …
55a419 …
18:30:00 18:40:00 55a3fd …
55a3fe …
55a419 …
… … …
Output Table
5
7
3
9
4
5
8
Streaming data is treated as an unbounded table
![Page 16: Integrating MATLAB Analytics into Enterprise Applications ...€¦ · Messaging Service • Reporting • Real ... • Training Models Connected Devices Continuous Data f(x) Stream](https://reader034.vdocuments.mx/reader034/viewer/2022043005/5f898b1d7b054432344f1875/html5/thumbnails/16.jpg)
17
Databases
CloudStorage
Streaming
Visualization
Web
Custom App
Public Cloud Private Cloud
Introducing MATLAB Production Server
Platform
Data Business SystemAnalytics
MATLAB Production Server
RequestBroker
AzureBlob
Cosmos DB
Amazon Kinesis
Azure Event Hub
![Page 17: Integrating MATLAB Analytics into Enterprise Applications ...€¦ · Messaging Service • Reporting • Real ... • Training Models Connected Devices Continuous Data f(x) Stream](https://reader034.vdocuments.mx/reader034/viewer/2022043005/5f898b1d7b054432344f1875/html5/thumbnails/17.jpg)
18
Introducing MATLAB Production Server
§ Server software– Manages packaged MATLAB
programs and worker pool
§ MATLAB Runtime libraries– Single server can use runtimes
from different releases
§ RESTful JSON interface
§ Lightweight client libraries – C/C++, .NET, Python, and Java
MATLAB Production Server
MATLABRuntime
Request Broker &
Program ManagerApplications/
Database Servers RESTful
JSON
EnterpriseApplication
MPS ClientLibrary
![Page 18: Integrating MATLAB Analytics into Enterprise Applications ...€¦ · Messaging Service • Reporting • Real ... • Training Models Connected Devices Continuous Data f(x) Stream](https://reader034.vdocuments.mx/reader034/viewer/2022043005/5f898b1d7b054432344f1875/html5/thumbnails/18.jpg)
19
Introducing Apache Kafka
§ Kafka is a high through-put distributed messaging system
§ Originally developed at LinkedIn and open sourced in 2011
§ Kafka is architected as a massively scalable publish/subscribe message queue
§ Well suited for large scale streaming applications
ProducerProducerProducer
Consumer Consumer Consumer
TopicPartition
Partition
Partition
TopicPartition
Partition
Partition
Kafka Cluster
![Page 19: Integrating MATLAB Analytics into Enterprise Applications ...€¦ · Messaging Service • Reporting • Real ... • Training Models Connected Devices Continuous Data f(x) Stream](https://reader034.vdocuments.mx/reader034/viewer/2022043005/5f898b1d7b054432344f1875/html5/thumbnails/19.jpg)
20
Connecting MATLAB Production Server to Apache Kafka
MATLAB Production Server
RequestBroker
&ProgramManager
Consumer Process feeds Topic-1
Async Java Client
Topic-0Partition
Partition
Partition
Topic-1Partition
Partition
Partition
Kafka Cluster
Consumer Process feeds Topic-0
Async Java Client
§ Kafka client for MATLAB Production Server feeds topics to functions deployed on the server
§ Configurable batch of messages passed as a MATLAB Timetable
§ Each consumer process feeds one topic to a specified function
§ Drive everything from a simple config file– No programming outside of MATLAB!
![Page 20: Integrating MATLAB Analytics into Enterprise Applications ...€¦ · Messaging Service • Reporting • Real ... • Training Models Connected Devices Continuous Data f(x) Stream](https://reader034.vdocuments.mx/reader034/viewer/2022043005/5f898b1d7b054432344f1875/html5/thumbnails/20.jpg)
21
Develop, Test, and Deploy a Stream Processing Function
MATLAB
Analytics Development
Business Decisions
MATLABCompiler
SDKAlgorithm
Developers
Storage Layer
End Users
Kafka Connector
Business Systems
Edge Devices
Production System
![Page 21: Integrating MATLAB Analytics into Enterprise Applications ...€¦ · Messaging Service • Reporting • Real ... • Training Models Connected Devices Continuous Data f(x) Stream](https://reader034.vdocuments.mx/reader034/viewer/2022043005/5f898b1d7b054432344f1875/html5/thumbnails/21.jpg)
23
Develop a Stream Processing Function in MATLAB
DEMO
![Page 22: Integrating MATLAB Analytics into Enterprise Applications ...€¦ · Messaging Service • Reporting • Real ... • Training Models Connected Devices Continuous Data f(x) Stream](https://reader034.vdocuments.mx/reader034/viewer/2022043005/5f898b1d7b054432344f1875/html5/thumbnails/22.jpg)
25
Test Your Stream Processing Function on Live Data
![Page 23: Integrating MATLAB Analytics into Enterprise Applications ...€¦ · Messaging Service • Reporting • Real ... • Training Models Connected Devices Continuous Data f(x) Stream](https://reader034.vdocuments.mx/reader034/viewer/2022043005/5f898b1d7b054432344f1875/html5/thumbnails/23.jpg)
26
Complete Your Application
MATLAB
Analytics Development
MATLAB Production Server
MATLABAnalytics
Business Decisions
MATLABCompiler
SDKAlgorithm
Developers
Storage Layer
End UsersBusiness Systems
Edge Devices
Production System
![Page 24: Integrating MATLAB Analytics into Enterprise Applications ...€¦ · Messaging Service • Reporting • Real ... • Training Models Connected Devices Continuous Data f(x) Stream](https://reader034.vdocuments.mx/reader034/viewer/2022043005/5f898b1d7b054432344f1875/html5/thumbnails/24.jpg)
28
Complete Your Application
Reference: https://www.mathworks.com/help/database/ug/import-and-analyze-data-from-mongodb.html
![Page 25: Integrating MATLAB Analytics into Enterprise Applications ...€¦ · Messaging Service • Reporting • Real ... • Training Models Connected Devices Continuous Data f(x) Stream](https://reader034.vdocuments.mx/reader034/viewer/2022043005/5f898b1d7b054432344f1875/html5/thumbnails/25.jpg)
30
Go Live!
![Page 26: Integrating MATLAB Analytics into Enterprise Applications ...€¦ · Messaging Service • Reporting • Real ... • Training Models Connected Devices Continuous Data f(x) Stream](https://reader034.vdocuments.mx/reader034/viewer/2022043005/5f898b1d7b054432344f1875/html5/thumbnails/26.jpg)
31
Key Takeaways
§ MATLAB Connects directly to your data so you can quickly design and validate algorithms
§ MATLAB’s high-level language and apps enable fast design iterations
§ MATLAB Production Server enables easy integration of your MATLAB algorithms with enterprise production systems
§ This enables you to spend your time understanding the data and designing algorithms
![Page 27: Integrating MATLAB Analytics into Enterprise Applications ...€¦ · Messaging Service • Reporting • Real ... • Training Models Connected Devices Continuous Data f(x) Stream](https://reader034.vdocuments.mx/reader034/viewer/2022043005/5f898b1d7b054432344f1875/html5/thumbnails/27.jpg)
32
Resources to learn and get started
§ Data Analytics with MATLAB§ MATLAB Compiler SDK§ MATLAB Production Server§ Database Toolbox
– https://www.mathworks.com/help/database/ug/import-and-analyze-data-from-mongodb.html
![Page 28: Integrating MATLAB Analytics into Enterprise Applications ...€¦ · Messaging Service • Reporting • Real ... • Training Models Connected Devices Continuous Data f(x) Stream](https://reader034.vdocuments.mx/reader034/viewer/2022043005/5f898b1d7b054432344f1875/html5/thumbnails/28.jpg)
33© 2017 The MathWorks, Inc.
MATLAB and Simulink are registered trademarks of The MathWorks, Inc. See www.mathworks.com/trademarks for a list of additional trademarks.Other product or brand names may be trademarks or registered trademarks of their respective holders.”