designing and implementing your iot solutions with open source
TRANSCRIPT
© 2016 MapR Technologies 1© 2016 MapR Technologies
Designing & Implementing your IOT Solutions with Open Source
Sunil Patil Solutions Architect, MapRSridhar Reddy Director, Solutions Engineering, MapR
2016
© 2016 MapR Technologies 2
Agenda• Use Case• Demo• Layers
– Sensor – Controller– Protocol– Messaging– Computation– Persistence– Dashboard
© 2016 MapR Technologies 3© 2015 MapR Technologies
Use Case – Temperature Monitoring
© 2016 MapR Technologies 4
Use Case – Temperature monitoring dashboard
© 2016 MapR Technologies 5
Do I need big data for IOT ?Sensor reads temperature once every second
• Number of readings per year3600*24*365 = 31,536,000 (31 million)
• Every reading is 100 bytes 31,536,000 * 100 = 3,153,600,000 (3.1 GB)• Every store has 50 aisel(4 sensors) and 6000+ stores
200 * 6000 = 1,200,000 (1.2 million sensors)3 * 200 * 6000 = 3,600,000 (3.6 PetaBytes)
© 2016 MapR Technologies 6© 2015 MapR Technologies
Demo
© 2016 MapR Technologies 7
Demo – Data Flow
© 2016 MapR Technologies 8
Demo – Data Flow
© 2016 MapR Technologies 9
IOT Big Data Reference Architecture
© 2016 MapR Technologies 10© 2015 MapR Technologies
Sensors/Actuators
© 2016 MapR Technologies 11
Why do we need Sensor/Actuator Layer ?Sensor/actuator will be responsible for1. Interacting with physical world
1. Sensor – Measures or detects physical property
1. Thermistor2. Temperature Oscillators
2. Actuator – Move or control mechanism or system
1. LED2. Servo Motor
© 2016 MapR Technologies 12
Temperature Sensors
DS18B20 DHT22
© 2016 MapR Technologies 13
Sensor Pin Layout
DS18B20 DHT22
© 2016 MapR Technologies 14
Actuator Pin Layout
© 2016 MapR Technologies 15
Home Automation
© 2016 MapR Technologies 16
IOT Home AutomationYou can build home automation1. Temp Monitoring – For AC2. Auto turning of lights on and off3. Sprinkler System4. Primitive home security5. Gas detection
© 2016 MapR Technologies 17© 2015 MapR Technologies
Device
© 2016 MapR Technologies 18
Why do we need Device Layer ?The controller layer is responsible for1. Reading data from sensor2. Transform sensor data into
message3. Communicate with IOT Gateway
using suitable protocol4. Listening to message from IOT Gateway5. Activating with actuators based on message from Gateway
© 2016 MapR Technologies 19
Arduino
Arduino Uno NodeMCU V2 –ESP8266
© 2016 MapR Technologies 20
Arduino Application
© 2016 MapR Technologies 21
Install Sensor Library
© 2016 MapR Technologies 22
Temperature sensor – Sensor Library
© 2016 MapR Technologies 23
Temperature sensor – Start Sensor
© 2016 MapR Technologies 24
Arduino – Read temperature
© 2016 MapR Technologies 25
Upload code to Arduino
© 2016 MapR Technologies 26
Raspberry Pi
© 2016 MapR Technologies 27
Raspberry pi - Logic for reading temperature
© 2016 MapR Technologies 28
Raspberry Pi – Development Life Cycle
© 2016 MapR Technologies 29
IOT Device comparison
Arduino Raspberry Pi
Architecture Micro controller Micro Processor
Operating System No Yes
Power Can go to sleep Continuous 5V
Network No Yes
Sensor Analog + digital Digital
Development Environment
Arduino IDE – C code Java, Python,… etc
© 2016 MapR Technologies 30© 2015 MapR Technologies
Protocols
© 2016 MapR Technologies 31
Why do we need Protocol Layer ?The protocol layer is responsible for 1. Communication between IOT Device and IT infrastructure2. The IOT Devices operate in constrained environment
1. Limited processing power2. Limited amount of memory (Less than 100K)3. Network usage
1. Less Reliable network2. Optimized usage of network
4. Operate on battery1. Sleep mode
© 2016 MapR Technologies 32
MQTT – Message Queue Telemetry TransportMQTT is publish/subscribe messaging protocol designed for lightweight M2M communication.• Open: MQTT is OASIS standard created by IBM • Lightweight Client: Client libraries less than 30 KB• Optimized Network usage: Small packet size 2 bytes (header)
– TCP Based. MQTT-SN is UDP based– Easy to parse packets
• Reliable: Three QoS patterns to avoid packet loss on client disconnect– At-most-once – does not survive failures, but is never duplicated– At-least-once – survives connection loss and other failures, but can be duplicated– Exactly-once – survives connection loss and other failures, and is never
duplicated
© 2016 MapR Technologies 33
MQTT vs. HTTPS 1. Performance is 80 times better2. Battery usage 10 times less
http://stephendnicholas.com/archives/1217
© 2016 MapR Technologies 34
MQTT
Device 1
Device 2
Device 3
MQTTBroker
Subscriber 1
Subscriber 2
/store/fremont/aisel1
/store/fremont/aisel2
/store/sanjose/aisel1
/store/fremont/#
/store/#
© 2016 MapR Technologies 35
MQTT – Server & Broker
Subscriber 1
Subscriber 2
/store/fremont/aisel1
/store/fremont/aisel2
/store/sanjose/aisel1
/store/fremont/#
/store/#
© 2016 MapR Technologies 36
MQTT – Include MQTT client library
© 2016 MapR Technologies 37
MQTT – Setup MQTT Connection
© 2016 MapR Technologies 38
Arduino – Send Receive Message
© 2016 MapR Technologies 39
Arduino – Memory usage
© 2016 MapR Technologies 40
CoAP – Constrained Application ProtocolThe CoAP is request/response protocol similar to HTTP + REST• Open: CoAP is IETF standard - RFC 7252 IETF• Lightweight Client: Client libraries less than 30 KB• Optimized Network usage
– Based on UDP (6LowPAN), SMS in addition to TCP– Compact 4 bytes header, compressed– CoAP uses familiar HTTP stuff like Method (Get, Put, Post, Put, Delete), status
codes, URIs, content types/mapping– Can be used to carry different type of data. It uses content type to identify type
of data being carried– HTTP Proxy can be used to map COAP to HTTP
• Reliable: The HTTP proxy takes care of reliability
© 2016 MapR Technologies 41
Arduino – Read Temp. + Publish MQTT Message
http://slideplayer.com/slide/9365664/
© 2016 MapR Technologies 42
COAP
Device1
Device 2
Device 3
HTTPProxy
Subscriber 1
Subscriber 2
/store/fremont/aisel1
/store/fremont/aisel2
/store/sanjose/aisel1 coap://192.168.1.2:5683:/store/sanjose/aisel1
http://192.168.1.2:5683:/store/fremont/store1
© 2016 MapR Technologies 43
Arduino – Read Temp. + Publish COAP Message
© 2016 MapR Technologies 44© 2015 MapR Technologies
Messaging
© 2016 MapR Technologies 45
Why do we need Messaging Layer ?The Messaging layer is responsible for 1. Highly available MQTT
1. Scale MQTT layer horizontally2. Provide high availability
2. Decouple IOT layer from the computation layer
3. Allow multiple consumers to process the IOT data to meet different business requirements
© 2016 MapR Technologies 46
MQTT load balancing – No Kafka
Device
Device
Device
Device
MQTT
MQTT
MQTT
Spark
Spark
Spark
© 2016 MapR Technologies 47
MQTT load balancing - Kafka
Device
Device
Device
Device
MQTT
MQTT
MQTT
Spark
Spark
Spark
Kafka
© 2016 MapR Technologies 48
MQTT Kafka Bridge
MQTT Kafka
Bridge 1
Bridge 2
© 2016 MapR Technologies 49© 2015 MapR Technologies
Computation
© 2016 MapR Technologies 50
Why do we need computation layer ?The computation layer is responsible for 1. Process huge amount of data in linear
scalable manner1. Read streaming data 2. Transform/Enrich/Analyze data3. Store the incoming data into persistence storage
2. Analyze IOT data1. Business Rules2. Predictive model
© 2016 MapR Technologies 51
Spark Streaming
© 2016 MapR Technologies 52
Business rules
© 2016 MapR Technologies 53© 2015 MapR Technologies
Storage
© 2016 MapR Technologies 54
Why do we need Storage Layer ?The storage layer is responsible for 1. Providing persistent store for IOT
data2. Store data in time series format
1. Efficient storage of data2. Ingest data at high speed3. Retrieve/Query data
© 2016 MapR Technologies 55
Time Series Database• Data Point
(time, temperature)
• Metricssensor.temp.1s
• TagssensorId=2, location=bed1
• Metrics + Tags = Time Series
© 2016 MapR Technologies 56
InfluxDB
© 2016 MapR Technologies 57© 2015 MapR Technologies
Dashboard
© 2016 MapR Technologies 58
Why do we need Dashboard Layer ?The dashboard layer is responsible for 1. Interact with TSDB2. Visualize TSDB data in easy to
understand format
© 2016 MapR Technologies 59
Configure InfluxDB connection
© 2016 MapR Technologies 60
Temperature Variation Graph
© 2016 MapR Technologies 61
Temperature Variation Graph
© 2016 MapR Technologies 62
Resources • Streaming Processing for Real-Time Analytics and Dashboard• MapR Streams• MapR DB• Eclipse IOT • COAP Tutorial• GitHub For source code
© 2016 MapR Technologies 64
Arduino - Logic for reading temperature