wrangling data at the iot rodeo - qcon london the developers & data domain experts give them the...
TRANSCRIPT
![Page 1: Wrangling Data at the IOT Rodeo - QCon London the developers & data domain experts Give them the platforms and tools to be productive ... COAP anybody ? ... Free to download and use](https://reader031.vdocuments.mx/reader031/viewer/2022030420/5aa74a697f8b9a424f8c213f/html5/thumbnails/1.jpg)
Copyright © 2014 Splunk Inc.
Wrangling Data at the IOT Rodeo
Damien Dallimore
@damiendallimore
![Page 2: Wrangling Data at the IOT Rodeo - QCon London the developers & data domain experts Give them the platforms and tools to be productive ... COAP anybody ? ... Free to download and use](https://reader031.vdocuments.mx/reader031/viewer/2022030420/5aa74a697f8b9a424f8c213f/html5/thumbnails/2.jpg)
2
Developer Evangelist @ Splunk
3rd QCON
I’m a metaphorical data “cowboy” , not a real one
![Page 3: Wrangling Data at the IOT Rodeo - QCon London the developers & data domain experts Give them the platforms and tools to be productive ... COAP anybody ? ... Free to download and use](https://reader031.vdocuments.mx/reader031/viewer/2022030420/5aa74a697f8b9a424f8c213f/html5/thumbnails/3.jpg)
3
![Page 4: Wrangling Data at the IOT Rodeo - QCon London the developers & data domain experts Give them the platforms and tools to be productive ... COAP anybody ? ... Free to download and use](https://reader031.vdocuments.mx/reader031/viewer/2022030420/5aa74a697f8b9a424f8c213f/html5/thumbnails/4.jpg)
The cowboy metaphor
4
Data wrangling / lassoing (capturing)Data needs harnessing (bring under control for analysis)
Data might need a little grooming (clean, filter)Data might need branding (categorizing / labeling / enrichment)
Data corralling (correlation)Data stabling (securing)
Data needs to go to the rodeo (a platform)
Make data useful = be a data cowboy
![Page 5: Wrangling Data at the IOT Rodeo - QCon London the developers & data domain experts Give them the platforms and tools to be productive ... COAP anybody ? ... Free to download and use](https://reader031.vdocuments.mx/reader031/viewer/2022030420/5aa74a697f8b9a424f8c213f/html5/thumbnails/5.jpg)
MACHINE
IS EVERYWHEREDATA
© 2014 Splunk Inc. All rights reserved.
![Page 6: Wrangling Data at the IOT Rodeo - QCon London the developers & data domain experts Give them the platforms and tools to be productive ... COAP anybody ? ... Free to download and use](https://reader031.vdocuments.mx/reader031/viewer/2022030420/5aa74a697f8b9a424f8c213f/html5/thumbnails/6.jpg)
BIG DATA COMES FROM MACHINESVolume | Velocity | Variety | Variability
GPS,RFID,
Hypervisor,Web Servers,
Email, MessagingClickstreams, Mobile,
Telephony, IVR, Databases,Sensors, Telematics, Storage,
Servers, Security Devices, Desktops © 2014 Splunk Inc. All rights reserved.
![Page 7: Wrangling Data at the IOT Rodeo - QCon London the developers & data domain experts Give them the platforms and tools to be productive ... COAP anybody ? ... Free to download and use](https://reader031.vdocuments.mx/reader031/viewer/2022030420/5aa74a697f8b9a424f8c213f/html5/thumbnails/7.jpg)
The IOT Revolution (or rather Evolution)
7
![Page 8: Wrangling Data at the IOT Rodeo - QCon London the developers & data domain experts Give them the platforms and tools to be productive ... COAP anybody ? ... Free to download and use](https://reader031.vdocuments.mx/reader031/viewer/2022030420/5aa74a697f8b9a424f8c213f/html5/thumbnails/8.jpg)
8
Internet of DocumentsInternet of Commerce
Internet of PeopleInternet of APIs
Internet of MobileInternet of Things
![Page 9: Wrangling Data at the IOT Rodeo - QCon London the developers & data domain experts Give them the platforms and tools to be productive ... COAP anybody ? ... Free to download and use](https://reader031.vdocuments.mx/reader031/viewer/2022030420/5aa74a697f8b9a424f8c213f/html5/thumbnails/9.jpg)
9
![Page 10: Wrangling Data at the IOT Rodeo - QCon London the developers & data domain experts Give them the platforms and tools to be productive ... COAP anybody ? ... Free to download and use](https://reader031.vdocuments.mx/reader031/viewer/2022030420/5aa74a697f8b9a424f8c213f/html5/thumbnails/10.jpg)
10
What is this IOT data, is it these things ?
![Page 11: Wrangling Data at the IOT Rodeo - QCon London the developers & data domain experts Give them the platforms and tools to be productive ... COAP anybody ? ... Free to download and use](https://reader031.vdocuments.mx/reader031/viewer/2022030420/5aa74a697f8b9a424f8c213f/html5/thumbnails/11.jpg)
Operational Technology
Energy Oil & Gas Process Buildings MfgTransport-
ationMedical Devices
Telecom
Consumer Technology
Smart Home
Wearables Media
The landscape is much, much vaster
Industrial Data Producing Assets
![Page 12: Wrangling Data at the IOT Rodeo - QCon London the developers & data domain experts Give them the platforms and tools to be productive ... COAP anybody ? ... Free to download and use](https://reader031.vdocuments.mx/reader031/viewer/2022030420/5aa74a697f8b9a424f8c213f/html5/thumbnails/12.jpg)
Succeeding with IOT data
12
IOT data is already being generatedAnd we are already capturing this data
The key challenge will be in turning this into something genuinely useful. This is the opportunity.
Enable the developers & data domain expertsGive them the platforms and tools to be productive
This leads to ECOSYSTEM
![Page 13: Wrangling Data at the IOT Rodeo - QCon London the developers & data domain experts Give them the platforms and tools to be productive ... COAP anybody ? ... Free to download and use](https://reader031.vdocuments.mx/reader031/viewer/2022030420/5aa74a697f8b9a424f8c213f/html5/thumbnails/13.jpg)
How can Splunk help ?
![Page 14: Wrangling Data at the IOT Rodeo - QCon London the developers & data domain experts Give them the platforms and tools to be productive ... COAP anybody ? ... Free to download and use](https://reader031.vdocuments.mx/reader031/viewer/2022030420/5aa74a697f8b9a424f8c213f/html5/thumbnails/14.jpg)
Splunk can help you become an IOT data cowboy
14
Wrangle – Collect the dataHarness – Search over the data / Correlate
Show at the Rodeo – Visualize the data/Alerting
Provide a platform for Developers to build IOT Apps
![Page 15: Wrangling Data at the IOT Rodeo - QCon London the developers & data domain experts Give them the platforms and tools to be productive ... COAP anybody ? ... Free to download and use](https://reader031.vdocuments.mx/reader031/viewer/2022030420/5aa74a697f8b9a424f8c213f/html5/thumbnails/15.jpg)
Platform for machine data
15
Splunk storage Other Big Data stores
DeveloperPlatform
Data collectionand indexing
Report and
analyze
Custom dashboards
Monitor and alert
Ad hoc search
![Page 16: Wrangling Data at the IOT Rodeo - QCon London the developers & data domain experts Give them the platforms and tools to be productive ... COAP anybody ? ... Free to download and use](https://reader031.vdocuments.mx/reader031/viewer/2022030420/5aa74a697f8b9a424f8c213f/html5/thumbnails/16.jpg)
DeveloperPlatform
Report and
analyze
Custom dashboards
Monitor and alert
Ad hoc search
Platform for machine data
16
Splunk storage Other Big Data stores
Data collectionand indexing
Any amount, any location, any source.Schema at read time, not write time
Data in any format
No RDBMS
Very Extensible / Build Apps
Secure data / Lifecycle data
![Page 17: Wrangling Data at the IOT Rodeo - QCon London the developers & data domain experts Give them the platforms and tools to be productive ... COAP anybody ? ... Free to download and use](https://reader031.vdocuments.mx/reader031/viewer/2022030420/5aa74a697f8b9a424f8c213f/html5/thumbnails/17.jpg)
Wrangling
17
![Page 18: Wrangling Data at the IOT Rodeo - QCon London the developers & data domain experts Give them the platforms and tools to be productive ... COAP anybody ? ... Free to download and use](https://reader031.vdocuments.mx/reader031/viewer/2022030420/5aa74a697f8b9a424f8c213f/html5/thumbnails/18.jpg)
Wrangling
18
![Page 19: Wrangling Data at the IOT Rodeo - QCon London the developers & data domain experts Give them the platforms and tools to be productive ... COAP anybody ? ... Free to download and use](https://reader031.vdocuments.mx/reader031/viewer/2022030420/5aa74a697f8b9a424f8c213f/html5/thumbnails/19.jpg)
Wrangling
19
Amazon Kinesis Splunk Stream
MQTT Kafka
JMS AMQP
REST APIs SNMP
COAP anybody ? Any other sources ?
![Page 20: Wrangling Data at the IOT Rodeo - QCon London the developers & data domain experts Give them the platforms and tools to be productive ... COAP anybody ? ... Free to download and use](https://reader031.vdocuments.mx/reader031/viewer/2022030420/5aa74a697f8b9a424f8c213f/html5/thumbnails/20.jpg)
Kepware Industrial Data Forwarder for Splunk
20
Real-time streaming data collection from 150+ industrial protocolsProprietary and legacy data translation
![Page 21: Wrangling Data at the IOT Rodeo - QCon London the developers & data domain experts Give them the platforms and tools to be productive ... COAP anybody ? ... Free to download and use](https://reader031.vdocuments.mx/reader031/viewer/2022030420/5aa74a697f8b9a424f8c213f/html5/thumbnails/21.jpg)
ABBAllen-BradleyAnalog DevicesAromatAutomationDirectBeckhoffBristolContrexCutler-HammerFisherFujiGEHoneywellMettler-ToledoMitsubishiOmron
Opto 22PhilipsSattBusScanivalveSiemensSimaticSixnetSquareDTelemecaniqueThermo WestronicsToshibaToyopucTriconexWagoWeatherBugWeatherfordYokogawa
BACnet IPEnron ModbusModbus ASCII SerialModbus PlusModbus RTU SerialModbus TCP/IPODBCOPC DAOPC UAOPC XML-DA
Examples of Kepware Supported Commercial and Proprietary Protocols Examples of Open Protocols
![Page 22: Wrangling Data at the IOT Rodeo - QCon London the developers & data domain experts Give them the platforms and tools to be productive ... COAP anybody ? ... Free to download and use](https://reader031.vdocuments.mx/reader031/viewer/2022030420/5aa74a697f8b9a424f8c213f/html5/thumbnails/22.jpg)
Harnessing
05/27/2014T10:24:17GMT applicationId="safetyObs" eventType="safety" assetID="CV1002384-1045" employeeId="114635" jobSite="PLEC-2014-GC" observationId="184568-451124-256" observation="Control Valve handle extracted to manual position. No lockout/tagout or other tag visible. Process is running." observationCriticality="5" imageId="PLEC-2014-GC-184568-451124-256" imageUri="https://mybucket.s3.amazonaws.com/PLEC-2014-GC-184568-451124-256.png"
1543541, workorder, bsic, 78544, pipefitting, CV1002384, "install manual bleed bypass", 04/13/2014, 05/21/2014, 25663, complete
05/22/2014 03:17:31 asset_id="CV1002384-1045" process_id="batch transfer starting" alarm="control valve failed to open"05/22/2014 04:21:45 asset_id="CV1002384-1045" process_id="batch transfer starting" alarm="control valve failed to open"05/22/2014 06:35:39 asset_id="CV1002384-1045" process_id="batch transfer starting" alarm="control valve failed to open"05/22/2014 07:40:29 asset_id="CV1002384-1045" process_id="batch transfer starting" alarm="control valve failed to open"
![Page 23: Wrangling Data at the IOT Rodeo - QCon London the developers & data domain experts Give them the platforms and tools to be productive ... COAP anybody ? ... Free to download and use](https://reader031.vdocuments.mx/reader031/viewer/2022030420/5aa74a697f8b9a424f8c213f/html5/thumbnails/23.jpg)
Some data from a technician
© 2014 Splunk Inc. All rights reserved.
05/27/2014T10:24:17GMT applicationId="safetyObs" eventType="safety" assetID="CV1002384-1045" employeeId="114635" jobSite="PLEC-2014-GC" observationId="184568-451124-256" observation="Control Valve handle extracted to manual position. No lockout/tagout or other tag visible. Process is running." observationCriticality="5" imageId="PLEC-2014-GC-184568-451124-256" imageUri="https://mybucket.s3.amazonaws.com/PLEC-2014-GC-184568-451124-256.png"
1543541, workorder, bsic, 78544, pipefitting, CV1002384, "install manual bleed bypass", 04/13/2014, 05/21/2014, 25663, complete
05/22/2014 03:17:31 asset_id="CV1002384-1045" process_id="batch transfer starting" alarm="control valve failed to open"05/22/2014 04:21:45 asset_id="CV1002384-1045" process_id="batch transfer starting" alarm="control valve failed to open"05/22/2014 06:35:39 asset_id="CV1002384-1045" process_id="batch transfer starting" alarm="control valve failed to open"05/22/2014 07:40:29 asset_id="CV1002384-1045" process_id="batch transfer starting" alarm="control valve failed to open"
Safety Observation Application
![Page 24: Wrangling Data at the IOT Rodeo - QCon London the developers & data domain experts Give them the platforms and tools to be productive ... COAP anybody ? ... Free to download and use](https://reader031.vdocuments.mx/reader031/viewer/2022030420/5aa74a697f8b9a424f8c213f/html5/thumbnails/24.jpg)
Some data from a work order
05/27/2014T10:24:17GMT applicationId="safetyObs" eventType="safety" assetID="CV1002384-1045" employeeId="114635" jobSite="PLEC-2014-GC" observationId="184568-451124-256" observation="Control Valve handle extracted to manual position. No lockout/tagout or other tag visible. Process is running." observationCriticality="5" imageId="PLEC-2014-GC-184568-451124-256" imageUri="https://mybucket.s3.amazonaws.com/PLEC-2014-GC-184568-451124-256.png"
1543541, workorder, bsic, 78544, pipefitting, CV1002384, "install manual bleed bypass", 04/13/2014, 05/21/2014, 25663, complete
05/22/2014 03:17:31 asset_id="CV1002384-1045" process_id="batch transfer starting" alarm="control valve failed to open"05/22/2014 04:21:45 asset_id="CV1002384-1045" process_id="batch transfer starting" alarm="control valve failed to open"05/22/2014 06:35:39 asset_id="CV1002384-1045" process_id="batch transfer starting" alarm="control valve failed to open"05/22/2014 07:40:29 asset_id="CV1002384-1045" process_id="batch transfer starting" alarm="control valve failed to open"
CMMS (Work Order) Application
![Page 25: Wrangling Data at the IOT Rodeo - QCon London the developers & data domain experts Give them the platforms and tools to be productive ... COAP anybody ? ... Free to download and use](https://reader031.vdocuments.mx/reader031/viewer/2022030420/5aa74a697f8b9a424f8c213f/html5/thumbnails/25.jpg)
Some data from a “thing”
© 2014 Splunk Inc. All rights reserved.
05/27/2014T10:24:17GMT applicationId="safetyObs" eventType="safety" assetID="CV1002384-1045" employeeId="114635" jobSite="PLEC-2014-GC" observationId="184568-451124-256" observation="Control Valve handle extracted to manual position. No lockout/tagout or other tag visible. Process is running." observationCriticality="5" imageId="PLEC-2014-GC-184568-451124-256" imageUri="https://mybucket.s3.amazonaws.com/PLEC-2014-GC-184568-451124-256.png"
1543541, workorder, bsic, 78544, pipefitting, CV1002384, "install manual bleed bypass", 04/13/2014, 05/21/2014, 25663, complete
05/22/2014 03:17:31 asset_id="CV1002384-1045" process_id="batch transfer starting" alarm="control valve failed to open"05/22/2014 04:21:45 asset_id="CV1002384-1045" process_id="batch transfer starting" alarm="control valve failed to open"05/22/2014 06:35:39 asset_id="CV1002384-1045" process_id="batch transfer starting" alarm="control valve failed to open"05/22/2014 07:40:29 asset_id="CV1002384-1045" process_id="batch transfer starting" alarm="control valve failed to open"
SCADA Event and Alarm Logs
![Page 26: Wrangling Data at the IOT Rodeo - QCon London the developers & data domain experts Give them the platforms and tools to be productive ... COAP anybody ? ... Free to download and use](https://reader031.vdocuments.mx/reader031/viewer/2022030420/5aa74a697f8b9a424f8c213f/html5/thumbnails/26.jpg)
Correlate the data– Make New Discoveries
© 2014 Splunk Inc. All rights reserved.
05/27/2014T10:24:17GMT applicationId="safetyObs" eventType="safety" assetID="CV1002384-1045" employeeId="114635" jobSite="PLEC-2014-GC" observationId="184568-451124-256" observation="Control Valve handle extracted to manual position. No lockout/tagout or other tag visible. Process is running." observationCriticality="5" imageId="PLEC-2014-GC-184568-451124-256" imageUri="https://mybucket.s3.amazonaws.com/PLEC-2014-GC-184568-451124-256.png"
1543541, workorder, bsic, 78544, pipefitting, CV1002384, "install manual bleed bypass", 04/13/2014, 05/21/2014, 25663, complete
05/21/2014 03:17:31 asset_id="CV1002384-1045" process_id="batch transfer starting" alarm="control valve failed to open"05/21/2014 04:21:45 asset_id="CV1002384-1045" process_id="batch transfer starting" alarm="control valve failed to open"05/21/2014 06:35:39 asset_id="CV1002384-1045" process_id="batch transfer starting" alarm="control valve failed to open"05/21/2014 07:40:29 asset_id="CV1002384-1045" process_id="batch transfer starting" alarm="control valve failed to open"
Asset ID
Asset ID
Asset IDMTBF
CompletedTechnician
![Page 27: Wrangling Data at the IOT Rodeo - QCon London the developers & data domain experts Give them the platforms and tools to be productive ... COAP anybody ? ... Free to download and use](https://reader031.vdocuments.mx/reader031/viewer/2022030420/5aa74a697f8b9a424f8c213f/html5/thumbnails/27.jpg)
Extensive Platform (Rodeo) for Developers
27
REST API
Build Splunk Apps Extend and Integrate Splunk
Simple XML
JavaScript
HTML / CSS
Web Framework
JavaJavaScriptPython
RubyC#PHP
Data Models
Search Extensibility
Modular Inputs
SDKs
![Page 28: Wrangling Data at the IOT Rodeo - QCon London the developers & data domain experts Give them the platforms and tools to be productive ... COAP anybody ? ... Free to download and use](https://reader031.vdocuments.mx/reader031/viewer/2022030420/5aa74a697f8b9a424f8c213f/html5/thumbnails/28.jpg)
Splunk IOT Demos
28
![Page 29: Wrangling Data at the IOT Rodeo - QCon London the developers & data domain experts Give them the platforms and tools to be productive ... COAP anybody ? ... Free to download and use](https://reader031.vdocuments.mx/reader031/viewer/2022030420/5aa74a697f8b9a424f8c213f/html5/thumbnails/29.jpg)
Splunk options
29
Splunk> Enterprise : Free to download and use. Index 500 MB/day.
Splunk> Cloud : Premium, cloud hosted. Full Enterprise stack.100% uptime.
Splunk> Sandbox : Spin up a cloud instance in minutes. Load in data.
Hunk> : Splunk for data in Hadoop HDFS, MongoDB , other datastores (Neo4J)
![Page 30: Wrangling Data at the IOT Rodeo - QCon London the developers & data domain experts Give them the platforms and tools to be productive ... COAP anybody ? ... Free to download and use](https://reader031.vdocuments.mx/reader031/viewer/2022030420/5aa74a697f8b9a424f8c213f/html5/thumbnails/30.jpg)
splunk.com/iot
apps.splunk.com
dev.splunk.com
Be an IOT Data Cowboy