![Page 1: QCon London 2015 - Wrangling Data at the IOT Rodeo](https://reader030.vdocuments.mx/reader030/viewer/2022032714/55aa73371a28ab652e8b4871/html5/thumbnails/1.jpg)
Copyright © 2014 Splunk Inc.
Wrangling Data at the IOT Rodeo
Damien Dallimore
@damiendallimore
![Page 2: QCon London 2015 - Wrangling Data at the IOT Rodeo](https://reader030.vdocuments.mx/reader030/viewer/2022032714/55aa73371a28ab652e8b4871/html5/thumbnails/2.jpg)
2
Developer Evangelist @ Splunk
3rd QCON
I’m a metaphorical data “cowboy” , not a real one
![Page 3: QCon London 2015 - Wrangling Data at the IOT Rodeo](https://reader030.vdocuments.mx/reader030/viewer/2022032714/55aa73371a28ab652e8b4871/html5/thumbnails/3.jpg)
3
![Page 4: QCon London 2015 - Wrangling Data at the IOT Rodeo](https://reader030.vdocuments.mx/reader030/viewer/2022032714/55aa73371a28ab652e8b4871/html5/thumbnails/4.jpg)
The cowboy metaphor
4
Data wrangling / lassoing (capturing)Data needs harnessing (bring under control for analysis)
Data might need a little grooming (clean, filter)Data might need branding (categorizing / labeling / enrichment)
Data corralling (correlation)Data stabling (securing)
Data needs to go to the rodeo (a platform)
Make data useful = be a data cowboy
![Page 5: QCon London 2015 - Wrangling Data at the IOT Rodeo](https://reader030.vdocuments.mx/reader030/viewer/2022032714/55aa73371a28ab652e8b4871/html5/thumbnails/5.jpg)
MACHINE
IS EVERYWHEREDATA
© 2014 Splunk Inc. All rights reserved.
![Page 6: QCon London 2015 - Wrangling Data at the IOT Rodeo](https://reader030.vdocuments.mx/reader030/viewer/2022032714/55aa73371a28ab652e8b4871/html5/thumbnails/6.jpg)
BIG DATA COMES FROM MACHINESVolume | Velocity | Variety | Variability
GPS,RFID,
Hypervisor,Web Servers,
Email, MessagingClickstreams, Mobile,
Telephony, IVR, Databases,Sensors, Telematics, Storage,
Servers, Security Devices, Desktops © 2014 Splunk Inc. All rights reserved.
![Page 7: QCon London 2015 - Wrangling Data at the IOT Rodeo](https://reader030.vdocuments.mx/reader030/viewer/2022032714/55aa73371a28ab652e8b4871/html5/thumbnails/7.jpg)
The IOT Revolution (or rather Evolution)
7
![Page 8: QCon London 2015 - Wrangling Data at the IOT Rodeo](https://reader030.vdocuments.mx/reader030/viewer/2022032714/55aa73371a28ab652e8b4871/html5/thumbnails/8.jpg)
8
Internet of DocumentsInternet of Commerce
Internet of PeopleInternet of APIs
Internet of MobileInternet of Things
![Page 9: QCon London 2015 - Wrangling Data at the IOT Rodeo](https://reader030.vdocuments.mx/reader030/viewer/2022032714/55aa73371a28ab652e8b4871/html5/thumbnails/9.jpg)
9
![Page 10: QCon London 2015 - Wrangling Data at the IOT Rodeo](https://reader030.vdocuments.mx/reader030/viewer/2022032714/55aa73371a28ab652e8b4871/html5/thumbnails/10.jpg)
10
What is this IOT data, is it these things ?
![Page 11: QCon London 2015 - Wrangling Data at the IOT Rodeo](https://reader030.vdocuments.mx/reader030/viewer/2022032714/55aa73371a28ab652e8b4871/html5/thumbnails/11.jpg)
Operational Technology
Energy Oil & Gas Process Buildings MfgTransport-
ationMedical Devices
Telecom
Consumer Technology
Smart Home
Wearables Media
The landscape is much, much vaster
Industrial Data Producing Assets
![Page 12: QCon London 2015 - Wrangling Data at the IOT Rodeo](https://reader030.vdocuments.mx/reader030/viewer/2022032714/55aa73371a28ab652e8b4871/html5/thumbnails/12.jpg)
Succeeding with IOT data
12
IOT data is already being generatedAnd we are already capturing this data
The key challenge will be in turning this into something genuinely useful. This is the opportunity.
Enable the developers & data domain expertsGive them the platforms and tools to be productive
This leads to ECOSYSTEM
![Page 13: QCon London 2015 - Wrangling Data at the IOT Rodeo](https://reader030.vdocuments.mx/reader030/viewer/2022032714/55aa73371a28ab652e8b4871/html5/thumbnails/13.jpg)
How can Splunk help ?
![Page 14: QCon London 2015 - Wrangling Data at the IOT Rodeo](https://reader030.vdocuments.mx/reader030/viewer/2022032714/55aa73371a28ab652e8b4871/html5/thumbnails/14.jpg)
Splunk can help you become an IOT data cowboy
14
Wrangle – Collect the dataHarness – Search over the data / Correlate
Show at the Rodeo – Visualize the data/Alerting
Provide a platform for Developers to build IOT Apps
![Page 15: QCon London 2015 - Wrangling Data at the IOT Rodeo](https://reader030.vdocuments.mx/reader030/viewer/2022032714/55aa73371a28ab652e8b4871/html5/thumbnails/15.jpg)
Platform for machine data
15
Splunk storage Other Big Data stores
DeveloperPlatform
Data collectionand indexing
Report and
analyze
Custom dashboards
Monitor and alert
Ad hoc search
![Page 16: QCon London 2015 - Wrangling Data at the IOT Rodeo](https://reader030.vdocuments.mx/reader030/viewer/2022032714/55aa73371a28ab652e8b4871/html5/thumbnails/16.jpg)
DeveloperPlatform
Report and
analyze
Custom dashboards
Monitor and alert
Ad hoc search
Platform for machine data
16
Splunk storage Other Big Data stores
Data collectionand indexing
Any amount, any location, any source.Schema at read time, not write time
Data in any format
No RDBMS
Very Extensible / Build Apps
Secure data / Lifecycle data
![Page 17: QCon London 2015 - Wrangling Data at the IOT Rodeo](https://reader030.vdocuments.mx/reader030/viewer/2022032714/55aa73371a28ab652e8b4871/html5/thumbnails/17.jpg)
Wrangling
17
![Page 18: QCon London 2015 - Wrangling Data at the IOT Rodeo](https://reader030.vdocuments.mx/reader030/viewer/2022032714/55aa73371a28ab652e8b4871/html5/thumbnails/18.jpg)
Wrangling
18
![Page 19: QCon London 2015 - Wrangling Data at the IOT Rodeo](https://reader030.vdocuments.mx/reader030/viewer/2022032714/55aa73371a28ab652e8b4871/html5/thumbnails/19.jpg)
Wrangling
19
Amazon Kinesis Splunk Stream
MQTT Kafka
JMS AMQP
REST APIs SNMP
COAP anybody ? Any other sources ?
![Page 20: QCon London 2015 - Wrangling Data at the IOT Rodeo](https://reader030.vdocuments.mx/reader030/viewer/2022032714/55aa73371a28ab652e8b4871/html5/thumbnails/20.jpg)
Kepware Industrial Data Forwarder for Splunk
20
Real-time streaming data collection from 150+ industrial protocolsProprietary and legacy data translation
![Page 21: QCon London 2015 - Wrangling Data at the IOT Rodeo](https://reader030.vdocuments.mx/reader030/viewer/2022032714/55aa73371a28ab652e8b4871/html5/thumbnails/21.jpg)
ABBAllen-BradleyAnalog DevicesAromatAutomationDirectBeckhoffBristolContrexCutler-HammerFisherFujiGEHoneywellMettler-ToledoMitsubishiOmron
Opto 22PhilipsSattBusScanivalveSiemensSimaticSixnetSquareDTelemecaniqueThermo WestronicsToshibaToyopucTriconexWagoWeatherBugWeatherfordYokogawa
BACnet IPEnron ModbusModbus ASCII SerialModbus PlusModbus RTU SerialModbus TCP/IPODBCOPC DAOPC UAOPC XML-DA
Examples of Kepware Supported Commercial and Proprietary Protocols Examples of Open Protocols
![Page 22: QCon London 2015 - Wrangling Data at the IOT Rodeo](https://reader030.vdocuments.mx/reader030/viewer/2022032714/55aa73371a28ab652e8b4871/html5/thumbnails/22.jpg)
Harnessing
05/27/2014T10:24:17GMT applicationId="safetyObs" eventType="safety" assetID="CV1002384-1045" employeeId="114635" jobSite="PLEC-2014-GC" observationId="184568-451124-256" observation="Control Valve handle extracted to manual position. No lockout/tagout or other tag visible. Process is running." observationCriticality="5" imageId="PLEC-2014-GC-184568-451124-256" imageUri="https://mybucket.s3.amazonaws.com/PLEC-2014-GC-184568-451124-256.png"
1543541, workorder, bsic, 78544, pipefitting, CV1002384, "install manual bleed bypass", 04/13/2014, 05/21/2014, 25663, complete
05/22/2014 03:17:31 asset_id="CV1002384-1045" process_id="batch transfer starting" alarm="control valve failed to open"05/22/2014 04:21:45 asset_id="CV1002384-1045" process_id="batch transfer starting" alarm="control valve failed to open"05/22/2014 06:35:39 asset_id="CV1002384-1045" process_id="batch transfer starting" alarm="control valve failed to open"05/22/2014 07:40:29 asset_id="CV1002384-1045" process_id="batch transfer starting" alarm="control valve failed to open"
![Page 23: QCon London 2015 - Wrangling Data at the IOT Rodeo](https://reader030.vdocuments.mx/reader030/viewer/2022032714/55aa73371a28ab652e8b4871/html5/thumbnails/23.jpg)
Some data from a technician
© 2014 Splunk Inc. All rights reserved.
05/27/2014T10:24:17GMT applicationId="safetyObs" eventType="safety" assetID="CV1002384-1045" employeeId="114635" jobSite="PLEC-2014-GC" observationId="184568-451124-256" observation="Control Valve handle extracted to manual position. No lockout/tagout or other tag visible. Process is running." observationCriticality="5" imageId="PLEC-2014-GC-184568-451124-256" imageUri="https://mybucket.s3.amazonaws.com/PLEC-2014-GC-184568-451124-256.png"
1543541, workorder, bsic, 78544, pipefitting, CV1002384, "install manual bleed bypass", 04/13/2014, 05/21/2014, 25663, complete
05/22/2014 03:17:31 asset_id="CV1002384-1045" process_id="batch transfer starting" alarm="control valve failed to open"05/22/2014 04:21:45 asset_id="CV1002384-1045" process_id="batch transfer starting" alarm="control valve failed to open"05/22/2014 06:35:39 asset_id="CV1002384-1045" process_id="batch transfer starting" alarm="control valve failed to open"05/22/2014 07:40:29 asset_id="CV1002384-1045" process_id="batch transfer starting" alarm="control valve failed to open"
Safety Observation Application
![Page 24: QCon London 2015 - Wrangling Data at the IOT Rodeo](https://reader030.vdocuments.mx/reader030/viewer/2022032714/55aa73371a28ab652e8b4871/html5/thumbnails/24.jpg)
Some data from a work order
05/27/2014T10:24:17GMT applicationId="safetyObs" eventType="safety" assetID="CV1002384-1045" employeeId="114635" jobSite="PLEC-2014-GC" observationId="184568-451124-256" observation="Control Valve handle extracted to manual position. No lockout/tagout or other tag visible. Process is running." observationCriticality="5" imageId="PLEC-2014-GC-184568-451124-256" imageUri="https://mybucket.s3.amazonaws.com/PLEC-2014-GC-184568-451124-256.png"
1543541, workorder, bsic, 78544, pipefitting, CV1002384, "install manual bleed bypass", 04/13/2014, 05/21/2014, 25663, complete
05/22/2014 03:17:31 asset_id="CV1002384-1045" process_id="batch transfer starting" alarm="control valve failed to open"05/22/2014 04:21:45 asset_id="CV1002384-1045" process_id="batch transfer starting" alarm="control valve failed to open"05/22/2014 06:35:39 asset_id="CV1002384-1045" process_id="batch transfer starting" alarm="control valve failed to open"05/22/2014 07:40:29 asset_id="CV1002384-1045" process_id="batch transfer starting" alarm="control valve failed to open"
CMMS (Work Order) Application
![Page 25: QCon London 2015 - Wrangling Data at the IOT Rodeo](https://reader030.vdocuments.mx/reader030/viewer/2022032714/55aa73371a28ab652e8b4871/html5/thumbnails/25.jpg)
Some data from a “thing”
© 2014 Splunk Inc. All rights reserved.
05/27/2014T10:24:17GMT applicationId="safetyObs" eventType="safety" assetID="CV1002384-1045" employeeId="114635" jobSite="PLEC-2014-GC" observationId="184568-451124-256" observation="Control Valve handle extracted to manual position. No lockout/tagout or other tag visible. Process is running." observationCriticality="5" imageId="PLEC-2014-GC-184568-451124-256" imageUri="https://mybucket.s3.amazonaws.com/PLEC-2014-GC-184568-451124-256.png"
1543541, workorder, bsic, 78544, pipefitting, CV1002384, "install manual bleed bypass", 04/13/2014, 05/21/2014, 25663, complete
05/22/2014 03:17:31 asset_id="CV1002384-1045" process_id="batch transfer starting" alarm="control valve failed to open"05/22/2014 04:21:45 asset_id="CV1002384-1045" process_id="batch transfer starting" alarm="control valve failed to open"05/22/2014 06:35:39 asset_id="CV1002384-1045" process_id="batch transfer starting" alarm="control valve failed to open"05/22/2014 07:40:29 asset_id="CV1002384-1045" process_id="batch transfer starting" alarm="control valve failed to open"
SCADA Event and Alarm Logs
![Page 26: QCon London 2015 - Wrangling Data at the IOT Rodeo](https://reader030.vdocuments.mx/reader030/viewer/2022032714/55aa73371a28ab652e8b4871/html5/thumbnails/26.jpg)
Correlate the data– Make New Discoveries
© 2014 Splunk Inc. All rights reserved.
05/27/2014T10:24:17GMT applicationId="safetyObs" eventType="safety" assetID="CV1002384-1045" employeeId="114635" jobSite="PLEC-2014-GC" observationId="184568-451124-256" observation="Control Valve handle extracted to manual position. No lockout/tagout or other tag visible. Process is running." observationCriticality="5" imageId="PLEC-2014-GC-184568-451124-256" imageUri="https://mybucket.s3.amazonaws.com/PLEC-2014-GC-184568-451124-256.png"
1543541, workorder, bsic, 78544, pipefitting, CV1002384, "install manual bleed bypass", 04/13/2014, 05/21/2014, 25663, complete
05/21/2014 03:17:31 asset_id="CV1002384-1045" process_id="batch transfer starting" alarm="control valve failed to open"05/21/2014 04:21:45 asset_id="CV1002384-1045" process_id="batch transfer starting" alarm="control valve failed to open"05/21/2014 06:35:39 asset_id="CV1002384-1045" process_id="batch transfer starting" alarm="control valve failed to open"05/21/2014 07:40:29 asset_id="CV1002384-1045" process_id="batch transfer starting" alarm="control valve failed to open"
Asset ID
Asset ID
Asset IDMTBF
CompletedTechnician
![Page 27: QCon London 2015 - Wrangling Data at the IOT Rodeo](https://reader030.vdocuments.mx/reader030/viewer/2022032714/55aa73371a28ab652e8b4871/html5/thumbnails/27.jpg)
Extensive Platform (Rodeo) for Developers
27
REST API
Build Splunk Apps Extend and Integrate Splunk
Simple XML
JavaScript
HTML / CSS
Web Framework
JavaJavaScriptPython
RubyC#PHP
Data Models
Search Extensibility
Modular Inputs
SDKs
![Page 28: QCon London 2015 - Wrangling Data at the IOT Rodeo](https://reader030.vdocuments.mx/reader030/viewer/2022032714/55aa73371a28ab652e8b4871/html5/thumbnails/28.jpg)
Splunk IOT Demos
28
![Page 29: QCon London 2015 - Wrangling Data at the IOT Rodeo](https://reader030.vdocuments.mx/reader030/viewer/2022032714/55aa73371a28ab652e8b4871/html5/thumbnails/29.jpg)
Splunk options
29
Splunk> Enterprise : Free to download and use. Index 500 MB/day.
Splunk> Cloud : Premium, cloud hosted. Full Enterprise stack.100% uptime.
Splunk> Sandbox : Spin up a cloud instance in minutes. Load in data.
Hunk> : Splunk for data in Hadoop HDFS, MongoDB , other datastores (Neo4J)
![Page 30: QCon London 2015 - Wrangling Data at the IOT Rodeo](https://reader030.vdocuments.mx/reader030/viewer/2022032714/55aa73371a28ab652e8b4871/html5/thumbnails/30.jpg)
splunk.com/iot
apps.splunk.com
dev.splunk.com
Be an IOT Data Cowboy