wrangling data at the iot rodeo - qcon london the developers & data domain experts give them the...

30
Copyright © 2014 Splunk Inc. Wrangling Data at the IOT Rodeo Damien Dallimore [email protected] @damiendallimore

Upload: tranphuc

Post on 13-Mar-2018

215 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Wrangling Data at the IOT Rodeo - QCon London the developers & data domain experts Give them the platforms and tools to be productive ... COAP anybody ? ... Free to download and use

Copyright © 2014 Splunk Inc.

Wrangling Data at the IOT Rodeo

Damien Dallimore

[email protected]

@damiendallimore

Page 2: Wrangling Data at the IOT Rodeo - QCon London the developers & data domain experts Give them the platforms and tools to be productive ... COAP anybody ? ... Free to download and use

2

Developer Evangelist @ Splunk

3rd QCON

I’m a metaphorical data “cowboy” , not a real one

Page 3: Wrangling Data at the IOT Rodeo - QCon London the developers & data domain experts Give them the platforms and tools to be productive ... COAP anybody ? ... Free to download and use

3

Page 4: Wrangling Data at the IOT Rodeo - QCon London the developers & data domain experts Give them the platforms and tools to be productive ... COAP anybody ? ... Free to download and use

The cowboy metaphor

4

Data wrangling / lassoing (capturing)Data needs harnessing (bring under control for analysis)

Data might need a little grooming (clean, filter)Data might need branding (categorizing / labeling / enrichment)

Data corralling (correlation)Data stabling (securing)

Data needs to go to the rodeo (a platform)

Make data useful = be a data cowboy

Page 5: Wrangling Data at the IOT Rodeo - QCon London the developers & data domain experts Give them the platforms and tools to be productive ... COAP anybody ? ... Free to download and use

MACHINE

IS EVERYWHEREDATA

© 2014 Splunk Inc. All rights reserved.

Page 6: Wrangling Data at the IOT Rodeo - QCon London the developers & data domain experts Give them the platforms and tools to be productive ... COAP anybody ? ... Free to download and use

BIG DATA COMES FROM MACHINESVolume | Velocity | Variety | Variability

GPS,RFID,

Hypervisor,Web Servers,

Email, MessagingClickstreams, Mobile,

Telephony, IVR, Databases,Sensors, Telematics, Storage,

Servers, Security Devices, Desktops © 2014 Splunk Inc. All rights reserved.

Page 7: Wrangling Data at the IOT Rodeo - QCon London the developers & data domain experts Give them the platforms and tools to be productive ... COAP anybody ? ... Free to download and use

The IOT Revolution (or rather Evolution)

7

Page 8: Wrangling Data at the IOT Rodeo - QCon London the developers & data domain experts Give them the platforms and tools to be productive ... COAP anybody ? ... Free to download and use

8

Internet of DocumentsInternet of Commerce

Internet of PeopleInternet of APIs

Internet of MobileInternet of Things

Page 9: Wrangling Data at the IOT Rodeo - QCon London the developers & data domain experts Give them the platforms and tools to be productive ... COAP anybody ? ... Free to download and use

9

Page 10: Wrangling Data at the IOT Rodeo - QCon London the developers & data domain experts Give them the platforms and tools to be productive ... COAP anybody ? ... Free to download and use

10

What is this IOT data, is it these things ?

Page 11: Wrangling Data at the IOT Rodeo - QCon London the developers & data domain experts Give them the platforms and tools to be productive ... COAP anybody ? ... Free to download and use

Operational Technology

Energy Oil & Gas Process Buildings MfgTransport-

ationMedical Devices

Telecom

Consumer Technology

Smart Home

Wearables Media

The landscape is much, much vaster

Industrial Data Producing Assets

Page 12: Wrangling Data at the IOT Rodeo - QCon London the developers & data domain experts Give them the platforms and tools to be productive ... COAP anybody ? ... Free to download and use

Succeeding with IOT data

12

IOT data is already being generatedAnd we are already capturing this data

The key challenge will be in turning this into something genuinely useful. This is the opportunity.

Enable the developers & data domain expertsGive them the platforms and tools to be productive

This leads to ECOSYSTEM

Page 13: Wrangling Data at the IOT Rodeo - QCon London the developers & data domain experts Give them the platforms and tools to be productive ... COAP anybody ? ... Free to download and use

How can Splunk help ?

Page 14: Wrangling Data at the IOT Rodeo - QCon London the developers & data domain experts Give them the platforms and tools to be productive ... COAP anybody ? ... Free to download and use

Splunk can help you become an IOT data cowboy

14

Wrangle – Collect the dataHarness – Search over the data / Correlate

Show at the Rodeo – Visualize the data/Alerting

Provide a platform for Developers to build IOT Apps

Page 15: Wrangling Data at the IOT Rodeo - QCon London the developers & data domain experts Give them the platforms and tools to be productive ... COAP anybody ? ... Free to download and use

Platform for machine data

15

Splunk storage Other Big Data stores

DeveloperPlatform

Data collectionand indexing

Report and

analyze

Custom dashboards

Monitor and alert

Ad hoc search

Page 16: Wrangling Data at the IOT Rodeo - QCon London the developers & data domain experts Give them the platforms and tools to be productive ... COAP anybody ? ... Free to download and use

DeveloperPlatform

Report and

analyze

Custom dashboards

Monitor and alert

Ad hoc search

Platform for machine data

16

Splunk storage Other Big Data stores

Data collectionand indexing

Any amount, any location, any source.Schema at read time, not write time

Data in any format

No RDBMS

Very Extensible / Build Apps

Secure data / Lifecycle data

Page 17: Wrangling Data at the IOT Rodeo - QCon London the developers & data domain experts Give them the platforms and tools to be productive ... COAP anybody ? ... Free to download and use

Wrangling

17

Page 18: Wrangling Data at the IOT Rodeo - QCon London the developers & data domain experts Give them the platforms and tools to be productive ... COAP anybody ? ... Free to download and use

Wrangling

18

Page 19: Wrangling Data at the IOT Rodeo - QCon London the developers & data domain experts Give them the platforms and tools to be productive ... COAP anybody ? ... Free to download and use

Wrangling

19

Amazon Kinesis Splunk Stream

MQTT Kafka

JMS AMQP

REST APIs SNMP

COAP anybody ? Any other sources ?

Page 20: Wrangling Data at the IOT Rodeo - QCon London the developers & data domain experts Give them the platforms and tools to be productive ... COAP anybody ? ... Free to download and use

Kepware Industrial Data Forwarder for Splunk

20

Real-time streaming data collection from 150+ industrial protocolsProprietary and legacy data translation

Page 21: Wrangling Data at the IOT Rodeo - QCon London the developers & data domain experts Give them the platforms and tools to be productive ... COAP anybody ? ... Free to download and use

ABBAllen-BradleyAnalog DevicesAromatAutomationDirectBeckhoffBristolContrexCutler-HammerFisherFujiGEHoneywellMettler-ToledoMitsubishiOmron

Opto 22PhilipsSattBusScanivalveSiemensSimaticSixnetSquareDTelemecaniqueThermo WestronicsToshibaToyopucTriconexWagoWeatherBugWeatherfordYokogawa

BACnet IPEnron ModbusModbus ASCII SerialModbus PlusModbus RTU SerialModbus TCP/IPODBCOPC DAOPC UAOPC XML-DA

Examples of Kepware Supported Commercial and Proprietary Protocols Examples of Open Protocols

Page 22: Wrangling Data at the IOT Rodeo - QCon London the developers & data domain experts Give them the platforms and tools to be productive ... COAP anybody ? ... Free to download and use

Harnessing

05/27/2014T10:24:17GMT applicationId="safetyObs" eventType="safety" assetID="CV1002384-1045" employeeId="114635" jobSite="PLEC-2014-GC" observationId="184568-451124-256" observation="Control Valve handle extracted to manual position. No lockout/tagout or other tag visible. Process is running." observationCriticality="5" imageId="PLEC-2014-GC-184568-451124-256" imageUri="https://mybucket.s3.amazonaws.com/PLEC-2014-GC-184568-451124-256.png"

1543541, workorder, bsic, 78544, pipefitting, CV1002384, "install manual bleed bypass", 04/13/2014, 05/21/2014, 25663, complete

05/22/2014 03:17:31 asset_id="CV1002384-1045" process_id="batch transfer starting" alarm="control valve failed to open"05/22/2014 04:21:45 asset_id="CV1002384-1045" process_id="batch transfer starting" alarm="control valve failed to open"05/22/2014 06:35:39 asset_id="CV1002384-1045" process_id="batch transfer starting" alarm="control valve failed to open"05/22/2014 07:40:29 asset_id="CV1002384-1045" process_id="batch transfer starting" alarm="control valve failed to open"

Page 23: Wrangling Data at the IOT Rodeo - QCon London the developers & data domain experts Give them the platforms and tools to be productive ... COAP anybody ? ... Free to download and use

Some data from a technician

© 2014 Splunk Inc. All rights reserved.

05/27/2014T10:24:17GMT applicationId="safetyObs" eventType="safety" assetID="CV1002384-1045" employeeId="114635" jobSite="PLEC-2014-GC" observationId="184568-451124-256" observation="Control Valve handle extracted to manual position. No lockout/tagout or other tag visible. Process is running." observationCriticality="5" imageId="PLEC-2014-GC-184568-451124-256" imageUri="https://mybucket.s3.amazonaws.com/PLEC-2014-GC-184568-451124-256.png"

1543541, workorder, bsic, 78544, pipefitting, CV1002384, "install manual bleed bypass", 04/13/2014, 05/21/2014, 25663, complete

05/22/2014 03:17:31 asset_id="CV1002384-1045" process_id="batch transfer starting" alarm="control valve failed to open"05/22/2014 04:21:45 asset_id="CV1002384-1045" process_id="batch transfer starting" alarm="control valve failed to open"05/22/2014 06:35:39 asset_id="CV1002384-1045" process_id="batch transfer starting" alarm="control valve failed to open"05/22/2014 07:40:29 asset_id="CV1002384-1045" process_id="batch transfer starting" alarm="control valve failed to open"

Safety Observation Application

Page 24: Wrangling Data at the IOT Rodeo - QCon London the developers & data domain experts Give them the platforms and tools to be productive ... COAP anybody ? ... Free to download and use

Some data from a work order

05/27/2014T10:24:17GMT applicationId="safetyObs" eventType="safety" assetID="CV1002384-1045" employeeId="114635" jobSite="PLEC-2014-GC" observationId="184568-451124-256" observation="Control Valve handle extracted to manual position. No lockout/tagout or other tag visible. Process is running." observationCriticality="5" imageId="PLEC-2014-GC-184568-451124-256" imageUri="https://mybucket.s3.amazonaws.com/PLEC-2014-GC-184568-451124-256.png"

1543541, workorder, bsic, 78544, pipefitting, CV1002384, "install manual bleed bypass", 04/13/2014, 05/21/2014, 25663, complete

05/22/2014 03:17:31 asset_id="CV1002384-1045" process_id="batch transfer starting" alarm="control valve failed to open"05/22/2014 04:21:45 asset_id="CV1002384-1045" process_id="batch transfer starting" alarm="control valve failed to open"05/22/2014 06:35:39 asset_id="CV1002384-1045" process_id="batch transfer starting" alarm="control valve failed to open"05/22/2014 07:40:29 asset_id="CV1002384-1045" process_id="batch transfer starting" alarm="control valve failed to open"

CMMS (Work Order) Application

Page 25: Wrangling Data at the IOT Rodeo - QCon London the developers & data domain experts Give them the platforms and tools to be productive ... COAP anybody ? ... Free to download and use

Some data from a “thing”

© 2014 Splunk Inc. All rights reserved.

05/27/2014T10:24:17GMT applicationId="safetyObs" eventType="safety" assetID="CV1002384-1045" employeeId="114635" jobSite="PLEC-2014-GC" observationId="184568-451124-256" observation="Control Valve handle extracted to manual position. No lockout/tagout or other tag visible. Process is running." observationCriticality="5" imageId="PLEC-2014-GC-184568-451124-256" imageUri="https://mybucket.s3.amazonaws.com/PLEC-2014-GC-184568-451124-256.png"

1543541, workorder, bsic, 78544, pipefitting, CV1002384, "install manual bleed bypass", 04/13/2014, 05/21/2014, 25663, complete

05/22/2014 03:17:31 asset_id="CV1002384-1045" process_id="batch transfer starting" alarm="control valve failed to open"05/22/2014 04:21:45 asset_id="CV1002384-1045" process_id="batch transfer starting" alarm="control valve failed to open"05/22/2014 06:35:39 asset_id="CV1002384-1045" process_id="batch transfer starting" alarm="control valve failed to open"05/22/2014 07:40:29 asset_id="CV1002384-1045" process_id="batch transfer starting" alarm="control valve failed to open"

SCADA Event and Alarm Logs

Page 26: Wrangling Data at the IOT Rodeo - QCon London the developers & data domain experts Give them the platforms and tools to be productive ... COAP anybody ? ... Free to download and use

Correlate the data– Make New Discoveries

© 2014 Splunk Inc. All rights reserved.

05/27/2014T10:24:17GMT applicationId="safetyObs" eventType="safety" assetID="CV1002384-1045" employeeId="114635" jobSite="PLEC-2014-GC" observationId="184568-451124-256" observation="Control Valve handle extracted to manual position. No lockout/tagout or other tag visible. Process is running." observationCriticality="5" imageId="PLEC-2014-GC-184568-451124-256" imageUri="https://mybucket.s3.amazonaws.com/PLEC-2014-GC-184568-451124-256.png"

1543541, workorder, bsic, 78544, pipefitting, CV1002384, "install manual bleed bypass", 04/13/2014, 05/21/2014, 25663, complete

05/21/2014 03:17:31 asset_id="CV1002384-1045" process_id="batch transfer starting" alarm="control valve failed to open"05/21/2014 04:21:45 asset_id="CV1002384-1045" process_id="batch transfer starting" alarm="control valve failed to open"05/21/2014 06:35:39 asset_id="CV1002384-1045" process_id="batch transfer starting" alarm="control valve failed to open"05/21/2014 07:40:29 asset_id="CV1002384-1045" process_id="batch transfer starting" alarm="control valve failed to open"

Asset ID

Asset ID

Asset IDMTBF

CompletedTechnician

Page 27: Wrangling Data at the IOT Rodeo - QCon London the developers & data domain experts Give them the platforms and tools to be productive ... COAP anybody ? ... Free to download and use

Extensive Platform (Rodeo) for Developers

27

REST API

Build Splunk Apps Extend and Integrate Splunk

Simple XML

JavaScript

HTML / CSS

Web Framework

JavaJavaScriptPython

RubyC#PHP

Data Models

Search Extensibility

Modular Inputs

SDKs

Page 28: Wrangling Data at the IOT Rodeo - QCon London the developers & data domain experts Give them the platforms and tools to be productive ... COAP anybody ? ... Free to download and use

Splunk IOT Demos

28

Page 29: Wrangling Data at the IOT Rodeo - QCon London the developers & data domain experts Give them the platforms and tools to be productive ... COAP anybody ? ... Free to download and use

Splunk options

29

Splunk> Enterprise : Free to download and use. Index 500 MB/day.

Splunk> Cloud : Premium, cloud hosted. Full Enterprise stack.100% uptime.

Splunk> Sandbox : Spin up a cloud instance in minutes. Load in data.

Hunk> : Splunk for data in Hadoop HDFS, MongoDB , other datastores (Neo4J)

Page 30: Wrangling Data at the IOT Rodeo - QCon London the developers & data domain experts Give them the platforms and tools to be productive ... COAP anybody ? ... Free to download and use

splunk.com/iot

apps.splunk.com

dev.splunk.com

Be an IOT Data Cowboy