IoT REALIZED CONNECTED CAR Ian Huston Senior Data Scientist @ianhuston

https://github.com/pivotal/IoT-ConnectedCar

B A C K T O T H E BEGINNING

REALLY COOL IoT PROJECT

COVERED BY AN

NDA

CONNECTED

CAR

PIECES ARE THE

SAME

P R E D I C T T H E DESTINATION

PREDICT THE RANGE

TCP SOCKET OVER

BLUETOOTH HTTP POST

OVER CELLULAR

HOW DOES THIS

WORK?

START IN THE

CAR

ON BOARD DIAGNOSTICS

OBD II

PHONE PROVIDES CONNECTIVITY

127.0.0.1:9000

{ ! "vehicle_speed":103, ! "obd_standards":2, ! "intake_manifold_pressure":"", ! "accelerator_throttle_pos_e":14, ! "engine_load":89, ! "maf_airflow":33, ! "latitude":"32.897554", ! "vin":"1HGCM82633A004352", ! "bearing":"343.922580", ! "catalyst_temp":779, ! "relative_throttle_pos":12, ! "fuel_level_input":89, ! "fuel_system_status":[2,0], ! "accelerator_throttle_pos_d":29, ! "acceleration":"0.953", ! "throttle_position":21, ! "barometric_pressure":97, ! "control_module_voltage":13, ! "longitude":"-96.810236", ! "distance_with_mil_on":0, ! "coolant_temp":94, ! "intake_air_temp":34, ! "rpm":1593, ! "short_term_fuel":-2, ! "time_since_engine_start":4054, ! "absolute_throttle_pos_b":38, ! "long_term_fuel":3 !} !

Pivotal λ-Architecture

Spring XD

Pivotal HD

Pivotal GemFire

Speed Layer Serving Layer

Batch Layer

Pivotal CF ®

What is Cloud Foundry? http://cloudfoundry.org

Open Source

Multi-Cloud Platform

Simple App Deployment, Scaling & Availability

Cloud Applications Haiku Here is my source code Run it on the cloud for me I do not care how.

-  Onsi Fakhouri @onsijoe

Same applications running across different cloud providers

Private Public Hosted

Multi-Cloud

Pivotal Cloud Foundry for Open Cloud Computing

$ cf push

Pivotal λ-Architecture

Spring XD

Pivotal HD

Pivotal GemFire

Speed Layer Serving Layer

Batch Layer

Pivotal CF ®

O N T H E SERVER

SPRING XD EXTREME DATA

Spring XD Unified, distributed, and extensible open-source system for data ingestion, real time analytics, batch processing, and data export. o  Data Ingestion and Pipeline Processing

o  Kafka, RabbitMQ, MQTT, JMS, HTTP, GPDB, HAWQ

o  Partition, Filter, Transform, Split, Aggregate

o  Real Time Analytics and Complex Event Processing o  Spark Streaming, RxJava, JPMML Scoring

o  Redis, GemFire, Cassandra, etc..

o  Rapid Dashboarding

o  Batch Workflow Orchestration + ETL o  Map Reduce, HDFS, PIG, Hive, GPDB, HAWQ, Spark

o  RDBMS, FILE, FTP, Log, Mongo, Splunk Spring XD

SPRING

Ingestion

Orchestration

Extraction

Real-time Analytics

D I S T R I B U T E D RUNTIME

STREAMING BATCH &

http | filter | acmeenrich | shell | hdfs!

// Ingestion Stream Definitions !

IoT-HTTP.shell > typeconversiontransformer | !gemfire-server !

// Tap Definitions !

job create pi --definition "sparkapp --name=PI \ ! --master=local[8] \ ! --appJar=path/lib/spark-examples-1.4.0-hadoop2.6.0.jar \ ! --mainClass=org.apache.spark.examples.SparkPi \ ! --programArgs=100" –deploy !!job launch pi !

// Spark Example Job !

TCP SOCKET OVER

BLUETOOTH HTTP POST

OVER CELLULAR

Spring XD

HTTP transformers hdfs

gemfire tap

python

hdfs

GemFire

REST API

PySpark

Spring XD

HTTP transformers hdfs

gemfire tap

python

hdfs

GemFire

REST API

PySpark

Pivotal λ-Architecture

Spring XD

Pivotal HD

Pivotal GemFire

Speed Layer Serving Layer

Batch Layer

Pivotal CF ®

PivotalBig Data Suite

Advanced Analytics

Pivotal HAWQ

Pivotal Greenplum Database

Pivotal GemFire

RabbitMQ

Redis

Apps at ScaleData Processing

Big Data Suite Services on Pivotal Cloud Foundry

Spring XD

Spark

Pivotal HD

Pivotal Big Data Suite

Pivotal HAWQ RabbitMQRedisSpring XD Pivotal HD

Pivotal HD

Spring XD

Realtime Evaluation Batch Training

Data Persistence

REALTIME DATA SCIENCE

1 PREDICT JOURNEY

HOW DOES I T

WORK?

STORE SENSOR

DATA

R E C O R D E D

JOURNEYS

OFFLINE BATCH TRAINING

J O U R N E Y CLUSTERS

INITIAL PREDICTION

DRIVING HOME TO WORK

DRIVING WORK TO HOME

ONLINE PREDICTION

2 PREDICT RANGE

… !"Predictions": { ! "ClusterPredictions": { ! "0": { ! "EndLocation": [ ! 32.98525175453122, ! -96.70940837440399 ! ], ! "MPG_Journey": 25.60900810315203, ! "Probability": 0.63736 ! }, !… !

HORIZONTAL SCALABILITY

100,000’s OF CLIENTS

MILLIONS OF MESSAGES

PER MINUTE OVER

100 GB

IoT

ISSUES

COMPATIBILITY

CONNECTIVITY

SECURITY

LOOKING INTO THE

FUTURE

A D D I T I O N A L USE CASES

F L E E T MANAGEMENT

P R E D I C T I V E MAINTENANCE

A C C I D E N T ASSISTANCE

Ian Huston @ianhuston Michael Natusch @cumulyst