Bigdataeverywhere 2016 Alex Bates, CTO, Mtell

Operationalizing Apache Spark for the IoT

image purchased from Alamy, NY for use in commercial presentation

Smart Machine

https://vimeo.com/145543808

The Collection of:

Sensors

Sensor Networks

Smart Machines

Computer Power

Analytics

PeopleOne element of the Industrial Internet of Things

Solving problems that were previously unsolvable

The Internet of ThingsThe Internet of Things

Network of physical objects or “things” embedded with electronics, software, sensors, and network connectivity, which enables these objects to collect and exchange data.

4BILLION

Connected People

25+MILLION

Apps

25+BILLION

Devices

50TRILLIONGBs of DATA

Use Cases

remote monitoring

SME’s manage

many machines

condition monitoring

detect early when

problems are small

share learning

learn on one

transfer to many

maintenance

sensorstreams

leverages existing infrastructure: plant historian sensor data streams, and EAM system

connect import combine analyze present

prescriptiveaction

Information Flow

Library

Benchmark Statistics

Equipment Metadata

Raw Data

Failure Signatures, PM Repository, Failure Code Hierarchy

Failure Rates,MTBF/MTTF/MTBR/MTBM,Average Repair Cost,Performance Data

Equipment Type,ISO 14224 Structure,Sensor Templates

Sensor Data,Maintenance Work Orders,Operational Events,Crowdsourced Info

Levels of Data

global equipment knowledge base

Machine Genome Project

WAN

Evolution to Tier 2

IoT and Big Data

Comet 67P vs Los Angeles

Scale

Data Points/Yr

Single Rig

315 B

100 rigs

31.5 T

Scale

Data Points/Yr

Mtell and OpenTSDB

Optimized for Time Series Data AccessQuerying sensor data by date range plus other filters

MapR OpenTSDB Optimization

• Blob Ingestion – 100x faster– Instead of inserting each point, buffer

data in memory and insert a blob

containing batch

– Move blob maker upstream of insertion

into the storage tier

– Less writes to disk (once / blob instead

of once/point) and reduced data size

(blob compresses raw data)

– 10 node MapR cluster achieved 100

million points/sec ingestion (10 million

points/sec/node)

What has happened?

What will happen?

What should we do?

DESCRIPTIVE

PREDICTIVE

PRESCRIPTIVE

Analytics in Maintenance

Explanation from Gartner

Types of Agents

Learns precise specific failure signature & performs live monitoring, providing

early warnings of recurrences

Failure Agent

Learns baseline normal & performs live monitoring to expose abnormal operations – updates as conditions

change

Anomaly Agent

Finds unrecorded failure patterns in training data & excludes suspect

data from baseline normal conditions

Hidden Failure Agent

anomaly agentknows all learned patterns

matching all normal operating states

failure agent 001knows precise signature of

patterns leading to bearing failure

failure agent 002knows precise signature of

patterns leading to drive coupling failure

failure agents 003+many other agents each

assigned to detect exact failure patterns

Many Agents Per Asset

…each one holds the

precise multi-

dimensional/temporal data

pattern of a machine in a

specific operating mode

capture worker

experiences & actual

measured sensor data

Mtell uses agents for

individual machines

created by you

…in minutes

one single job

…constantly monitor for

that exact pattern

& sound off “alarm bells”

Agents – Retained Knowledge

Find Degradation Earlier

Multi-variate

Temporal & multi-variate

Detect complex failure patterns that cannot be detected by humans, or other technologies, or seen in any single variable trend

Platform for IoT Analytics

Equipment Sets

& Taxonomy

make & model

operating context

population analysis

Equipment

asset hierarchy

sync from EAM

Sensor Mapping

data

streams

Live Agents

rules

maintenance scheduling

machine learning

M2M

population learning

transfer learning

Performance

usage, states

wear, fatigue

efficacy

benchmarking

advancedanalytics

analyst

automation

population

learning

deep

learning

fleet bench-

marking

reservoirsignature

library

sensor

data store

cloud

sync

fault/eff.

signatures

signature

search

operatingcentermgmt.

incident

response

immersive

visualization

adaptive

feedback

knowledge

capture

intelligentsignal

processing

audit

automation

instrument

reliability

derived

signal mgt.

interpola-

tion

globalequipmenttaxonomy

eqmt model

catalog

Industry

op. context

fleet sensor

templates

sensor

groups

Platform Functions

Transfer Learning Signature Library

Template

Signature

Pump 02

Pump 01

Library of Known

Failure Signatures

Time-Series

Sensor Data

A

B

C

A

B

C

Mtell and Spark

RDD – Resilient Distributed DataSet

OpenTSDB

Spark RDD

RDD – Resilient Distributed DataSet

• Read-only collection partitioned across a set of machines

• Can be rebuilt if partition lost

• Enables spark to outperform Hadoop 10x on iterative machine learning jobs

Query data via HTTP

Over time builds RDD Data-frame; distributed across nodes

Aim: query database only once

Distributed data storage system

Stores high-precision data points

Scales almost linearly

But lacks analytics

Mtell REST API

Any request

Any client

Flexible/scalable link to any Python-based machine learning libraries

Human friendly

Spark Integration

Automated & Self-ImprovingPreviously Learned Normal

Automated & Self-ImprovingKnown Failure Signature

prescriptive maintenance

well in advancefixing a small problem

before it’s a big one

Automated & Self-ImprovingLearn New Operating State

Automated & Self-ImprovingLearn New Failure Signature

search deeper & improve

7-day anomaly alert 30+ day failure signature alert

Predictive / Prescriptive Analytics

Maintenance costsdecrease dramatically

Machineslast longer

Net outputincreases dramatically

Critical Assets stopbreaking down

Alex BatesABates@Mtell.com