tackling fleet test data with matlab · tackling fleet test data with matlab sebastian bomberg,...
TRANSCRIPT
![Page 1: Tackling Fleet Test Data with MATLAB · Tackling Fleet Test Data with MATLAB Sebastian Bomberg, Application Engineer. 4 A Fleet is a Collection of Resources that Generate Data You](https://reader034.vdocuments.mx/reader034/viewer/2022050417/5f8d7bda132e6161587b92b5/html5/thumbnails/1.jpg)
2 July | Europe
Tackling Fleet Test Data with MATLAB
Sebastian Bomberg, Application Engineer
![Page 2: Tackling Fleet Test Data with MATLAB · Tackling Fleet Test Data with MATLAB Sebastian Bomberg, Application Engineer. 4 A Fleet is a Collection of Resources that Generate Data You](https://reader034.vdocuments.mx/reader034/viewer/2022050417/5f8d7bda132e6161587b92b5/html5/thumbnails/2.jpg)
4
A Fleet is a Collection of Resources
that Generate Data You Want to Act on
Automotive
• Vehicles
• Engines
• Controllers
Manufacturing
• Pick & Place machines
• Welding robots
• Material handling systems
Agriculture
• Harvesters
• Tractors
• Mining
Infrastructure
• Charging stations
• Parking spaces
• Electronic toll collection
![Page 3: Tackling Fleet Test Data with MATLAB · Tackling Fleet Test Data with MATLAB Sebastian Bomberg, Application Engineer. 4 A Fleet is a Collection of Resources that Generate Data You](https://reader034.vdocuments.mx/reader034/viewer/2022050417/5f8d7bda132e6161587b92b5/html5/thumbnails/3.jpg)
5
Benefits of Fleet Data Analytics
▪ Faster time to insights (design, testing)
▪ Warranty / Transparency / Reputation
▪ New business opportunities in the Mobility Age
Understanding
Enables:
Better Design
New Business Opportunities
Good Reputation
![Page 4: Tackling Fleet Test Data with MATLAB · Tackling Fleet Test Data with MATLAB Sebastian Bomberg, Application Engineer. 4 A Fleet is a Collection of Resources that Generate Data You](https://reader034.vdocuments.mx/reader034/viewer/2022050417/5f8d7bda132e6161587b92b5/html5/thumbnails/4.jpg)
6
Fleet Analytics in Practice: Volkswagen Data Lab
Develop technology building block for tailoring
car features and services to individual
▪ Driver and Fleet Safety
▪ Driver Coaching
▪ Driver-Specific Insurance
Data sources
▪ Logged CAN bus data and travel record
Results
▪ Proof-of-concept model for “telematic fingerprint”
▪ Basis for the “pay-as-you-drive” concept
Source: “Connected Car – Fahrererkennung mit MATLAB“
Julia Fumbarev, Volkswagen Data Lab
MATLAB EXPO Germany, June 27, 2017, Munich Germany
![Page 5: Tackling Fleet Test Data with MATLAB · Tackling Fleet Test Data with MATLAB Sebastian Bomberg, Application Engineer. 4 A Fleet is a Collection of Resources that Generate Data You](https://reader034.vdocuments.mx/reader034/viewer/2022050417/5f8d7bda132e6161587b92b5/html5/thumbnails/5.jpg)
7
is a Leader in the Gartner Magic
Quadrant for 2020 Data Science
and Machine Learning Platforms
*Gartner Magic Quadrant for Data Science and Machine Learning Platforms, Peter Krensky, Erick Brethenoux, Jim Hare, Carlie Idoine, Alexander Linden, Svetlana Sicular, 11 February 2020 .
This graphic was published by Gartner, Inc. as part of a larger research document and should be evaluated in the context of the entire document. The Gartner document is available upon request from MathWorks.
Gartner does not endorse any vendor, product or service depicted in its research publications, and does not advise technology users to select only those vendors with the highest ratings or other designation. Gartner research
publications consist of the opinions of Gartner’s research organization and should not be construed as statements of fact. Gartner disclaims all warranties, express or implied, with respect to this research, including any
warranties of merchantability or fitness for a particular purpose.
![Page 6: Tackling Fleet Test Data with MATLAB · Tackling Fleet Test Data with MATLAB Sebastian Bomberg, Application Engineer. 4 A Fleet is a Collection of Resources that Generate Data You](https://reader034.vdocuments.mx/reader034/viewer/2022050417/5f8d7bda132e6161587b92b5/html5/thumbnails/6.jpg)
8
Data Analytics Workflow
Integrate Analytics with
Systems
Desktop Apps
Enterprise Scale
Systems
Embedded Devices
and Hardware
Files
Databases
Sensors
Access and Explore
Data
Develop Predictive
Models
Model Creation e.g.
Machine Learning
Model
Validation
Parameter
Optimization
Preprocess Data
Working with
Messy Data
Data Reduction/
Transformation
Feature
Extraction
![Page 7: Tackling Fleet Test Data with MATLAB · Tackling Fleet Test Data with MATLAB Sebastian Bomberg, Application Engineer. 4 A Fleet is a Collection of Resources that Generate Data You](https://reader034.vdocuments.mx/reader034/viewer/2022050417/5f8d7bda132e6161587b92b5/html5/thumbnails/7.jpg)
9
Data Analytics Workflow
Integrate Analytics with
Systems
Desktop Apps
Enterprise Scale
Systems
Embedded Devices
and Hardware
Files
Databases
Sensors
Access and Explore
Data
Develop Predictive
Models
Model Creation e.g.
Machine Learning
Model
Validation
Parameter
Optimization
Preprocess Data
Working with
Messy Data
Data Reduction/
Transformation
Feature
Extraction
![Page 8: Tackling Fleet Test Data with MATLAB · Tackling Fleet Test Data with MATLAB Sebastian Bomberg, Application Engineer. 4 A Fleet is a Collection of Resources that Generate Data You](https://reader034.vdocuments.mx/reader034/viewer/2022050417/5f8d7bda132e6161587b92b5/html5/thumbnails/8.jpg)
10
Code and Applications
Raw Files
File Types
Extract
MDF (.MF4)
.blf
.parquet
.mat
custom
binary
text
Extract, Transform, and Load (ETL) Workflow Considerations
Transform
Data Manipulation
- Unit Conversion
- Bit mapping
- Unpacking
Enforce Schema
Pre-Process & CleanWrite File Set
Write Mode
Load
.parquet
.mat
text
Many-to-fewer
One-to-one
![Page 9: Tackling Fleet Test Data with MATLAB · Tackling Fleet Test Data with MATLAB Sebastian Bomberg, Application Engineer. 4 A Fleet is a Collection of Resources that Generate Data You](https://reader034.vdocuments.mx/reader034/viewer/2022050417/5f8d7bda132e6161587b92b5/html5/thumbnails/9.jpg)
11
Databases
Images
MDF & BLF Files
Custom
Simulation
Access Data in Many Formats From Many Locations
▪ Type
▪ Structure
▪ Location
![Page 11: Tackling Fleet Test Data with MATLAB · Tackling Fleet Test Data with MATLAB Sebastian Bomberg, Application Engineer. 4 A Fleet is a Collection of Resources that Generate Data You](https://reader034.vdocuments.mx/reader034/viewer/2022050417/5f8d7bda132e6161587b92b5/html5/thumbnails/11.jpg)
13
Scale to Large Collections of Data
with DatastoreAvailable Datastores
General datastore
spreadsheetDatastore
tabularTextDatastore
fileDatastore
Database databaseDatastore
Image imageDatastore
denoisingImageDatastore
randomPatchExtractionDatastore
pixelLabelDatastore
augmentedImageDatastore
Audio audioDatastore
Predictive
Maintenance
fileEnsembleDatastore
simulationEnsembleDatastore
Simulink SimulationDatastore
Automotive mdfDatastore
Custom subclass matlab.io.Datastore
Transformed transform an existing datastore
![Page 12: Tackling Fleet Test Data with MATLAB · Tackling Fleet Test Data with MATLAB Sebastian Bomberg, Application Engineer. 4 A Fleet is a Collection of Resources that Generate Data You](https://reader034.vdocuments.mx/reader034/viewer/2022050417/5f8d7bda132e6161587b92b5/html5/thumbnails/12.jpg)
14
Data Analytics Workflow
Integrate Analytics with
Systems
Desktop Apps
Enterprise Scale
Systems
Embedded Devices
and Hardware
Files
Databases
Sensors
Access and Explore
Data
Develop Predictive
Models
Model Creation e.g.
Machine Learning
Model
Validation
Parameter
Optimization
Preprocess Data
Working with
Messy Data
Data Reduction/
Transformation
Feature
Extraction
![Page 13: Tackling Fleet Test Data with MATLAB · Tackling Fleet Test Data with MATLAB Sebastian Bomberg, Application Engineer. 4 A Fleet is a Collection of Resources that Generate Data You](https://reader034.vdocuments.mx/reader034/viewer/2022050417/5f8d7bda132e6161587b92b5/html5/thumbnails/13.jpg)
15
Data Analytics Workflow
Integrate Analytics with
Systems
Desktop Apps
Enterprise Scale
Systems
Embedded Devices
and Hardware
Files
Databases
Sensors
Access and Explore
Data
Develop Predictive
Models
Model Creation e.g.
Machine Learning
Model
Validation
Parameter
Optimization
Preprocess Data
Working with
Messy Data
Data Reduction/
Transformation
Feature
Extraction
![Page 14: Tackling Fleet Test Data with MATLAB · Tackling Fleet Test Data with MATLAB Sebastian Bomberg, Application Engineer. 4 A Fleet is a Collection of Resources that Generate Data You](https://reader034.vdocuments.mx/reader034/viewer/2022050417/5f8d7bda132e6161587b92b5/html5/thumbnails/14.jpg)
16
Preprocess and Explore Data in Few Lines of Code
![Page 15: Tackling Fleet Test Data with MATLAB · Tackling Fleet Test Data with MATLAB Sebastian Bomberg, Application Engineer. 4 A Fleet is a Collection of Resources that Generate Data You](https://reader034.vdocuments.mx/reader034/viewer/2022050417/5f8d7bda132e6161587b92b5/html5/thumbnails/15.jpg)
17
Use Dedicated Functions for Common Preprocessing Tasks
▪ Synchronize by time
▪ Find, fill, and remove missing
▪ Work with outliers
▪ Smooth noisy data
▪ Normalize, rescale data
![Page 16: Tackling Fleet Test Data with MATLAB · Tackling Fleet Test Data with MATLAB Sebastian Bomberg, Application Engineer. 4 A Fleet is a Collection of Resources that Generate Data You](https://reader034.vdocuments.mx/reader034/viewer/2022050417/5f8d7bda132e6161587b92b5/html5/thumbnails/16.jpg)
19
Explore Settings Quickly With Live Tasks
▪ Tasks are apps that can be
included in scripts
▪ Preprocessing tasks allow you to:
– Interactively explore parameters
and options
– Preview results based on those
parameters and options
▪ Automatically generate the
corresponding MATLAB code
▪ Save the task as part of the script
for subsequent use by others
![Page 17: Tackling Fleet Test Data with MATLAB · Tackling Fleet Test Data with MATLAB Sebastian Bomberg, Application Engineer. 4 A Fleet is a Collection of Resources that Generate Data You](https://reader034.vdocuments.mx/reader034/viewer/2022050417/5f8d7bda132e6161587b92b5/html5/thumbnails/17.jpg)
20
Data Analytics Workflow
Integrate Analytics with
Systems
Desktop Apps
Enterprise Scale
Systems
Embedded Devices
and Hardware
Files
Databases
Sensors
Access and Explore
Data
Develop Predictive
Models
Model Creation e.g.
Machine Learning
Model
Validation
Parameter
Optimization
Preprocess Data
Working with
Messy Data
Data Reduction/
Transformation
Feature
Extraction
![Page 18: Tackling Fleet Test Data with MATLAB · Tackling Fleet Test Data with MATLAB Sebastian Bomberg, Application Engineer. 4 A Fleet is a Collection of Resources that Generate Data You](https://reader034.vdocuments.mx/reader034/viewer/2022050417/5f8d7bda132e6161587b92b5/html5/thumbnails/18.jpg)
21
Data Analytics Workflow
Integrate Analytics with
Systems
Desktop Apps
Enterprise Scale
Systems
Embedded Devices
and Hardware
Files
Databases
Sensors
Access and Explore
Data
Develop Predictive
Models
Model Creation e.g.
Machine Learning
Model
Validation
Parameter
Optimization
Preprocess Data
Working with
Messy Data
Data Reduction/
Transformation
Feature
Extraction
![Page 19: Tackling Fleet Test Data with MATLAB · Tackling Fleet Test Data with MATLAB Sebastian Bomberg, Application Engineer. 4 A Fleet is a Collection of Resources that Generate Data You](https://reader034.vdocuments.mx/reader034/viewer/2022050417/5f8d7bda132e6161587b92b5/html5/thumbnails/19.jpg)
22
Automotive Vehicle Fleet – Intrinsic Hierarchy
Vehicles
Signals
Messages
Trips (files)
Time – Value pairs
![Page 20: Tackling Fleet Test Data with MATLAB · Tackling Fleet Test Data with MATLAB Sebastian Bomberg, Application Engineer. 4 A Fleet is a Collection of Resources that Generate Data You](https://reader034.vdocuments.mx/reader034/viewer/2022050417/5f8d7bda132e6161587b92b5/html5/thumbnails/20.jpg)
23
Fleet Analytics Questions Fall Into Two Broad Categories
For each (trip, day, serial #, customer,
etc) in the fleet data set, calculate
some Key Performance Indicator
(KPI*) given parameters XYZ".
Across All (data) in the fleet data set,
calculate descriptive statistics of
specific variables (min, max, median,
count, etc.) to summarize and
visualize (histograms).
Question
Type“Across All”“For Each”
![Page 21: Tackling Fleet Test Data with MATLAB · Tackling Fleet Test Data with MATLAB Sebastian Bomberg, Application Engineer. 4 A Fleet is a Collection of Resources that Generate Data You](https://reader034.vdocuments.mx/reader034/viewer/2022050417/5f8d7bda132e6161587b92b5/html5/thumbnails/21.jpg)
25
Scale Up to Big Data Without Big Changes
One file One hundred files
![Page 22: Tackling Fleet Test Data with MATLAB · Tackling Fleet Test Data with MATLAB Sebastian Bomberg, Application Engineer. 4 A Fleet is a Collection of Resources that Generate Data You](https://reader034.vdocuments.mx/reader034/viewer/2022050417/5f8d7bda132e6161587b92b5/html5/thumbnails/22.jpg)
26
Big Data File-Based Workflows
Question Type
File set location(C:\, NAS/NFS, S3, Blob, hdfs://…)
“For Each”
tds = transform(ds, @detectEvents);
TT = tall(tds);
EventsSummary = gather(TT);
function events = detectEvents(t)
<fill in analytics code here>
end
Datastoreds = parquetDatastore("hdfs://myData/Test")
![Page 23: Tackling Fleet Test Data with MATLAB · Tackling Fleet Test Data with MATLAB Sebastian Bomberg, Application Engineer. 4 A Fleet is a Collection of Resources that Generate Data You](https://reader034.vdocuments.mx/reader034/viewer/2022050417/5f8d7bda132e6161587b92b5/html5/thumbnails/23.jpg)
27
Compute Groupwise Metrics and Detect Events
▪ Explore range
▪ Grouped calculations
▪ Detect local minima
and maxima
▪ Detect abrupt changes in data with ischange
![Page 24: Tackling Fleet Test Data with MATLAB · Tackling Fleet Test Data with MATLAB Sebastian Bomberg, Application Engineer. 4 A Fleet is a Collection of Resources that Generate Data You](https://reader034.vdocuments.mx/reader034/viewer/2022050417/5f8d7bda132e6161587b92b5/html5/thumbnails/24.jpg)
28
Big Data File-Based Workflows
Question Type
File set location(C:\, NAS/NFS, S3, Blob, hdfs://…)
“Across All”
TT = tall(ds);
MedianValSigABC = gather(median(TT.SignalABC));
“For Each”
tds = transform(ds, @detectEvents);
TT = tall(tds);
EventsSummary = gather(TT);
function events = detectEvents(t)
<fill in analytics code here>
end
Datastoreds = parquetDatastore("hdfs://myData/Test")
![Page 25: Tackling Fleet Test Data with MATLAB · Tackling Fleet Test Data with MATLAB Sebastian Bomberg, Application Engineer. 4 A Fleet is a Collection of Resources that Generate Data You](https://reader034.vdocuments.mx/reader034/viewer/2022050417/5f8d7bda132e6161587b92b5/html5/thumbnails/25.jpg)
29
Perform “Across All” Calculations with Tall Tables
▪ Visualizations
▪ Data preprocessing
▪ Machine Learning
![Page 26: Tackling Fleet Test Data with MATLAB · Tackling Fleet Test Data with MATLAB Sebastian Bomberg, Application Engineer. 4 A Fleet is a Collection of Resources that Generate Data You](https://reader034.vdocuments.mx/reader034/viewer/2022050417/5f8d7bda132e6161587b92b5/html5/thumbnails/26.jpg)
31
Explore Fleet Data with Unsupervised Learning
![Page 27: Tackling Fleet Test Data with MATLAB · Tackling Fleet Test Data with MATLAB Sebastian Bomberg, Application Engineer. 4 A Fleet is a Collection of Resources that Generate Data You](https://reader034.vdocuments.mx/reader034/viewer/2022050417/5f8d7bda132e6161587b92b5/html5/thumbnails/27.jpg)
32
Unsupervised Learning for Operational Mode Clustering
![Page 28: Tackling Fleet Test Data with MATLAB · Tackling Fleet Test Data with MATLAB Sebastian Bomberg, Application Engineer. 4 A Fleet is a Collection of Resources that Generate Data You](https://reader034.vdocuments.mx/reader034/viewer/2022050417/5f8d7bda132e6161587b92b5/html5/thumbnails/28.jpg)
33
Data Analytics Workflow
Integrate Analytics with
Systems
Desktop Apps
Enterprise Scale
Systems
Embedded Devices
and Hardware
Files
Databases
Sensors
Access and Explore
Data
Develop Predictive
Models
Model Creation e.g.
Machine Learning
Model
Validation
Parameter
Optimization
Preprocess Data
Working with
Messy Data
Data Reduction/
Transformation
Feature
Extraction
![Page 29: Tackling Fleet Test Data with MATLAB · Tackling Fleet Test Data with MATLAB Sebastian Bomberg, Application Engineer. 4 A Fleet is a Collection of Resources that Generate Data You](https://reader034.vdocuments.mx/reader034/viewer/2022050417/5f8d7bda132e6161587b92b5/html5/thumbnails/29.jpg)
34
Data Analytics Workflow
Integrate Analytics with
Systems
Desktop Apps
Enterprise Scale
Systems
Embedded Devices
and Hardware
Files
Databases
Sensors
Access and Explore
Data
Develop Predictive
Models
Model Creation e.g.
Machine Learning
Model
Validation
Parameter
Optimization
Preprocess Data
Working with
Messy Data
Data Reduction/
Transformation
Feature
Extraction
![Page 30: Tackling Fleet Test Data with MATLAB · Tackling Fleet Test Data with MATLAB Sebastian Bomberg, Application Engineer. 4 A Fleet is a Collection of Resources that Generate Data You](https://reader034.vdocuments.mx/reader034/viewer/2022050417/5f8d7bda132e6161587b92b5/html5/thumbnails/30.jpg)
35
“Cold Storage” “Hot Storage”
Data
Historic data:
• Batch processing
• Large data on cluster
• Explore long term trends
• Build models
Streaming data:
• Near real-time
• Test and implement model
for new data
• Stream processing
Vehicle data, driver
profiles
Deploying Fleet Analytics
![Page 31: Tackling Fleet Test Data with MATLAB · Tackling Fleet Test Data with MATLAB Sebastian Bomberg, Application Engineer. 4 A Fleet is a Collection of Resources that Generate Data You](https://reader034.vdocuments.mx/reader034/viewer/2022050417/5f8d7bda132e6161587b92b5/html5/thumbnails/31.jpg)
36
Fleet Analytics Streaming Architecture
![Page 32: Tackling Fleet Test Data with MATLAB · Tackling Fleet Test Data with MATLAB Sebastian Bomberg, Application Engineer. 4 A Fleet is a Collection of Resources that Generate Data You](https://reader034.vdocuments.mx/reader034/viewer/2022050417/5f8d7bda132e6161587b92b5/html5/thumbnails/32.jpg)
37
Azure
Use Reference Architectures to Run MATLAB on Cloud Platforms
https://github.com/mathworks-ref-arch/
AWS
GitHub
![Page 33: Tackling Fleet Test Data with MATLAB · Tackling Fleet Test Data with MATLAB Sebastian Bomberg, Application Engineer. 4 A Fleet is a Collection of Resources that Generate Data You](https://reader034.vdocuments.mx/reader034/viewer/2022050417/5f8d7bda132e6161587b92b5/html5/thumbnails/33.jpg)
38
Data Analytics Workflow
Integrate Analytics with
Systems
Desktop Apps
Enterprise Scale
Systems
Embedded Devices
and Hardware
Files
Databases
Sensors
Access and Explore
Data
Develop Predictive
Models
Model Creation e.g.
Machine Learning
Model
Validation
Parameter
Optimization
Preprocess Data
Working with
Messy Data
Data Reduction/
Transformation
Feature
Extraction
![Page 34: Tackling Fleet Test Data with MATLAB · Tackling Fleet Test Data with MATLAB Sebastian Bomberg, Application Engineer. 4 A Fleet is a Collection of Resources that Generate Data You](https://reader034.vdocuments.mx/reader034/viewer/2022050417/5f8d7bda132e6161587b92b5/html5/thumbnails/34.jpg)
39
MATLAB Training Courses for Data Science
- Exploratory Data Analysis- Data Processing and Feature Engineering- Predictive Modeling and Machine Learning- Data Science Project
Training Courses
MATLAB Fundamentals (3 days)
MATLAB for Data Processing and
Visualization (1 day)
Processing Big Data with MATLAB (1 day)
Statistical Methods in MATLAB (2 days)
Machine Learning with MATLAB (2 days)
Signal Preprocessing and Feature
Extraction with MATLAB (1 day)
Deep Learning with MATLAB (2 days)
Accelerating and Parallelizing MATLAB
Code (2 days)
![Page 35: Tackling Fleet Test Data with MATLAB · Tackling Fleet Test Data with MATLAB Sebastian Bomberg, Application Engineer. 4 A Fleet is a Collection of Resources that Generate Data You](https://reader034.vdocuments.mx/reader034/viewer/2022050417/5f8d7bda132e6161587b92b5/html5/thumbnails/35.jpg)
40
MathWorks has the furthest completeness of vision in the
Leaders quadrant
We believe this recognition demonstrates our ability to:
▪ Empower your team, including those with limited AI or data science experience
▪ Provide complete workflows for data preparation, AI modeling, system design, and
production
▪ Deploy AI models on embedded devices, edge, enterprise systems, and the cloud
▪ Use Simulink to tackle integration challenges and reduce risk in designing AI-
driven systems
is a Leader in the Gartner Magic Quadrant for 2020
Data Science and Machine Learning Platforms
*Gartner Magic Quadrant for Data Science and Machine Learning Platforms, Peter Krensky, Erick Brethenoux, Jim Hare, Carlie Idoine, Alexander Linden, Svetlana Sicular, 11 February 2020 .
This graphic was published by Gartner, Inc. as part of a larger research document and should be evaluated in the context of the entire document. The Gartner document is available upon request from MathWorks.
Gartner does not endorse any vendor, product or service depicted in its research publications, and does not advise technology users to select only those vendors with the highest ratings or other designation. Gartner research
publications consist of the opinions of Gartner’s research organization and should not be construed as statements of fact. Gartner disclaims all warranties, express or implied, with respect to this research, including any
warranties of merchantability or fitness for a particular purpose.
![Page 36: Tackling Fleet Test Data with MATLAB · Tackling Fleet Test Data with MATLAB Sebastian Bomberg, Application Engineer. 4 A Fleet is a Collection of Resources that Generate Data You](https://reader034.vdocuments.mx/reader034/viewer/2022050417/5f8d7bda132e6161587b92b5/html5/thumbnails/36.jpg)
On which part of the data analytics
workflow do you spend most time?
Please contact us with questions
Sebastian [email protected]
b Data preprocessing
Data accessa
Data analysis and modelingc
Enterprise integrationd