build your own real time data processing platform in the cloud for

31
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Philipp Behre, Solutions Architect Jan Metzner, EMEA Solutions Architect Mobile/IoT Berlin, April 12 th 2016 berlin Build your own Real Time Data Processing Platform in the Cloud for Connecting Millions of Things @pbehre @janmetzner

Upload: dangque

Post on 13-Feb-2017

223 views

Category:

Documents


1 download

TRANSCRIPT

Page 1: Build your own Real Time Data Processing Platform in the Cloud for

© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Philipp Behre, Solutions Architect Jan Metzner, EMEA Solutions Architect Mobile/IoT

Berlin, April 12th 2016

berlin

Build your own Real Time Data Processing Platform in the Cloud for

Connecting Millions of Things

@pbehre @janmetzner

Page 2: Build your own Real Time Data Processing Platform in the Cloud for

What to expect from this session

•  How to connect Devices/Phones and People through the Cloud •  Analyze and react on Sensordata •  React on Data and Control Devices out of the Cloud Next Session: How to build out of that a smart system that does the decision on its own

Page 3: Build your own Real Time Data Processing Platform in the Cloud for

The Person has the context to decide

Page 4: Build your own Real Time Data Processing Platform in the Cloud for

The Person has the context to decide

Analyze & decide

Page 5: Build your own Real Time Data Processing Platform in the Cloud for

The Cloud make decisions with smart situational awareness

Monitor& have the final say

Enable smart decisions & act

Page 6: Build your own Real Time Data Processing Platform in the Cloud for

One Example: Water Pipe

Page 7: Build your own Real Time Data Processing Platform in the Cloud for

Water Pipe – Simplified Data Flow Sensors

Valve

Page 8: Build your own Real Time Data Processing Platform in the Cloud for

IoT Architectures build out of AWS Services

Page 9: Build your own Real Time Data Processing Platform in the Cloud for

Primitives for IoT

AWS Lambda

Amazon EMR

Amazon S3

AWS IoT

Amazon Kinesis

Amazon Machine Learning

Amazon Redshift

Amazon QuickSight

Amazon Cognito

Amazon Elasticsearch Service

Amazon DynamoDB

Page 10: Build your own Real Time Data Processing Platform in the Cloud for

Connect – Secure – Integrate

Page 11: Build your own Real Time Data Processing Platform in the Cloud for

IoT – Connect – Secure - Integrate

AWS Lambda

Amazon EMR

Amazon S3

AWS IoT

Amazon Kinesis

Amazon Machine Learning

Amazon Redshift

Amazon QuickSight

Amazon Cognito

Amazon Elasticsearch Service

Amazon DynamoDB

Page 12: Build your own Real Time Data Processing Platform in the Cloud for

DEVICE SDK Set of client libraries to

connect, authenticate and exchange messages

DEVICE GATEWAY Communicate with devices via

MQTT and HTTP

AUTHENTICATION Secure with mutual

authentication and encryption

RULES ENGINE Transform messages based on rules and

route to AWS Services

AWS Services - - - - -

3P Services

SHADOW Persistent thing state

during intermittent connections

APPLICATIONS

AWS IoT API

REGISTRY Identity and Management of

your things

AWS IoT: How it Works

Page 13: Build your own Real Time Data Processing Platform in the Cloud for

AWS IoT Security: Authentication and Authorization

AUTHENTICATION Secure with mutual

authentication and encryption

AUTHENTICATION AUTHORIZATION

Secure with mutual authentication and encryption

Page 14: Build your own Real Time Data Processing Platform in the Cloud for

Security and Authentication/ Authorization

Secure Communications with Things - Single API call to CreateKeysAndCertificate() - Client Generated CreateCertificateFromCSR(CSR) Fine-grained Authorization for:

Thing Management Pub/Sub Data Access AWS Service Access

AUTHENTICATION AUTHORIZATION

Secure with mutual authentication and encryption

Page 15: Build your own Real Time Data Processing Platform in the Cloud for

One Service, Two Protocols

MQTT + Mutual Auth TLS AWS Auth + HTTPS

Server Auth TLS + Cert TLS + Cert

Client Auth TLS + Cert AWS API Keys

Confidentiality TLS TLS

Protocol MQTT HTTP

Identification AWS ARNs AWS ARNs

Authorization AWS Policy AWS Policy

NDA

Page 16: Build your own Real Time Data Processing Platform in the Cloud for

AWS IoT Rules Engine

RULES ENGINE Transform messages based on rules and

route to AWS Services

Page 17: Build your own Real Time Data Processing Platform in the Cloud for

AWS IoT Rules Engine

Complex Evaluations Respond to the fleet, not just a single unit. Dozens of functions() available

Multiple / Simultaneous Actions Sometimes a situation requires you to take many actions

Simple & Familiar Syntax -  SQL Statement to define topic filter -  Optional WHERE clause

-  Advanced JSON support

Page 18: Build your own Real Time Data Processing Platform in the Cloud for

AWS IoT Rules Engine Actions

RULES ENGINE Transform messages based on rules and

route to AWS Services

AWS Services - - - - -

3P Services

AWS Services - - - - -

3P Services

Page 19: Build your own Real Time Data Processing Platform in the Cloud for

1. AWS Services (Direct Integration)

Rules Engine

Actions

AWS IoT Rules Engine

3. External Endpoints (via Lambda and SNS)

2. Rest of AWS (via Kinesis, Lambda, S3, and more)

SQS Redshift

RDS

EC2

ElasticSearch Kinesis

S3 Firehose

SNS Lambda DynamoDB

Page 20: Build your own Real Time Data Processing Platform in the Cloud for

AWS IoT Thing Shadow

THING SHADOW Persistent thing state

during intermittent connections

SHADOW Persistent thing state

during intermittent connections

APPLICATIONS

Page 21: Build your own Real Time Data Processing Platform in the Cloud for

AWS IoT Thing Shadow

Shadow

Thing

Report its current state to one or multiple shadows Retrieve its desired state from shadow

Mobile App

Set the desired state of a device Get the last reported state of the device Delete the shadow

Shadow

Shadow reports delta, desired and reported states along with metadata and version

Page 22: Build your own Real Time Data Processing Platform in the Cloud for

Let’s get our hands dirty - Demo

Page 23: Build your own Real Time Data Processing Platform in the Cloud for

Instantly React – but not ‘smart’

Page 24: Build your own Real Time Data Processing Platform in the Cloud for

Simple Store and Analyze

Page 25: Build your own Real Time Data Processing Platform in the Cloud for

IoT – Simple Store and Analyze

AWS Lambda

Amazon EMR

Amazon S3

AWS IoT

Amazon Kinesis

Amazon Machine Learning

Amazon Redshift

Amazon QuickSight

Amazon Cognito

Amazon Elasticsearch Service

Amazon DynamoDB

Page 26: Build your own Real Time Data Processing Platform in the Cloud for

The Architecture of our Example

Page 27: Build your own Real Time Data Processing Platform in the Cloud for

Your custom IoT Architecture

AWS IoT

ES

DynamoDB

Kinesis Lambda

Machine Learning

Firehose S3

Devices

RT Dashboard Cognito

QuickSight Redshift

Page 28: Build your own Real Time Data Processing Platform in the Cloud for

Pipe Example – Demo “not smart”

Page 29: Build your own Real Time Data Processing Platform in the Cloud for

How to get ‘smart’ à next Session

Page 30: Build your own Real Time Data Processing Platform in the Cloud for

Resources

AWS IoT Landing Page: http://aws.amazon.com/iot AWS Mobile Landing Page: http://aws.amazon.com/mobile YouTube Channels/Playlist: •  AWS re:Invent 2015 Mobile/IoT Sessions:

http://bit.ly/22ik1V1 •  AWS re:Invent 2015 Big Data / Analytics Sessions: •  http://bit.ly/1S2 •  AWS Webinar Channel: http://bit.ly/1QVI2IY

Page 31: Build your own Real Time Data Processing Platform in the Cloud for

@pbehre @janmetzner