iot architecture - are traditional architectures good enough?
TRANSCRIPT
BASEL BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENEVA HAMBURG COPENHAGEN LAUSANNE MUNICH STUTTGART VIENNA ZURICH
Internet of Things (IoT) Are traditional architectures good enough?
Guido Schmutz
Guido Schmutz
Working for Trivadis for more than 18 yearsOracle ACE Director for Fusion Middleware and SOACo-Author of different booksConsultant, Trainer, Software Architect for Java, SOA & Big Data / Fast DataMember of Trivadis Architecture BoardTechnology Manager @ Trivadis
More than 25 years of software development experience
Contact: [email protected]: http://guidoschmutz.wordpress.comSlideshare: http://www.slideshare.net/gschmutzTwitter: gschmutz
Our company.
Trivadis is a market leader in IT consulting, system integration, solution engineeringand the provision of IT services focusing on and and Open Source technologiesin Switzerland, Germany, Austria and Denmark. We offer our services in the followingstrategic business fields:
Trivadis Services takes over the interacting operation of your IT systems.
O P E R A T I O N
COPENHAGEN
MUNICH
LAUSANNEBERN
ZURICHBRUGG
GENEVA
HAMBURG
DÜSSELDORF
FRANKFURT
STUTTGART
FREIBURG
BASEL
VIENNA
With over 600 specialists and IT experts in your region.
14 Trivadis branches and more than600 employees
200 Service Level Agreements
Over 4,000 training participants
Research and development budget:CHF 5.0 million
Financially self-supporting andsustainably profitable
Experience from more than 1,900 projects per year at over 800customers
Agenda
1. Introduction
2. Are Traditional Solution architectures good enough?3. IoT Architecture & Components
4. Demo Use Case – Truck Movements
5. Oracle IoT Cloud Service6. Summary
Introduction
Internet of Things Wave
Internet of Things (IoT): Enabling communication between devices, people & processes to exchange useful information & knowledge that create value for humans
Term was first proposed by Kevin Ashton in 1999
The concept of IoT first became popular at the Auto-ID center and MIT
IoT can also be named M2M
Internet Of Things – Sensorsare/will be everywhereThere are more devices tapping into the internet than people on earth
How do we prepare our systems/architecture for the future?
Source:CiscoSource:TheEconomist
Reasons why IoT opportunity is occurring now ?
Affordable hardware
• Costs of actuators & sensors have been cut in half over last 10 years
Smaller, but more powerful hardware
• Form factors of hardware have shrunk to millimeter or even nanometer levels
Ubiquitous & cheap mobility
• Cost for mobile devices, bandwidth and data processing has declined as much as 97% over last 10 years
Availability of supporting tools
• Big data tools & cloud based infrastructure have become widely available and fairly sophisticated
Mass market awareness
• IoT has surpassed a critical tipping point• Vision of a connected world has reached such a
followership that companies have initiated IoTdevelopments
• Commitment is irreversible
The Sensing-as-a-Service Model
Are Traditional Solution architectures good enough?
A little story of a “real-life” customer situation
Traditional system interact with its clients and does its workImplemented using legacy technologies (i.e. PL/SQL)
New requirement:• Offer notification service to notify
customer when goods are shipped• Subscription and inform over different
channels• Existing technology doesn’t fit
delivery
LogisticSystem
Oracle
MobileApps
Sensor ship
sort
12
Rich (Web)Client Apps
DB
schedule
Logic(PL/SQL)
delivery
A little story of a “real-life” customer situation
Events are “owned” by traditional application (as well as the channels they are transported over)
Implement notification as a new Java-based application/system
But we need the events ! => so let’s integrate
delivery
LogisticSystem
Oracle
MobileApps
Sensor ship
sort
13
Rich (Web)Client Apps
DB
schedule
Notification
Logic(PL/SQL)
Logic(Java)delivery
SMS
…
A little story of a “real-life” customer situation
integrate in order to get the information! Oracle Service Bus was already there
Rule Engine implemented in Java and invoked from OSB message flowNotification system informed via queueHigher Latency introduced (good enough in this case)
delivery
LogisticSystem
Oracle
OracleServiceBus
MobileApps
Sensor AQship
sort
14
Rich (Web)Client Apps
DB
schedule
Filter
Notification
Logic(PL/SQL)
JMS
RuleEngine(Java)
Logic(Java)delivery
shipdelivery
delivery true SMS
…
A little story of a “real-life” customer situation
Treat events as first-class citizens
Events belong to the “enterprise” and not an individual system => Catalog of Events similar to Catalog of Services/APIs !!
Event (stream) processing can be introduced and by that latency reduced!
delivery
LogisticSystem
Oracle
OracleServiceBus
MobileApps
Sensor AQship
sort
15
Rich (Web)Client Apps
DB
schedule
Filter
Notification
Logic(PL/SQL)
JMS
RuleEngine(Java)
Logic(Java)delivery
shipdelivery
delivery true SMS
…
Treat Events as Events and make them globally available
delivery
LogisticSystem
Oracle
OracleServiceBus
MobileApps
Sensorship
sort
16
Rich (Web)Client Apps
DB
schedule
Filter
Notification
Logic(PL/SQL)
JMS
RuleEngine(Java)
Logic(Java)
delivery
ship
delivery true SMS
…
EventBus/Hub
Stream/EventProcessing
Traditional Data Processing - Challenges
• Introduces too much “decision latency”
• Responses are delivered “after the fact”
• Maximum value of the identified situation is lost
• Decision are made on old and stale data
• “Data a Rest”
The New Era: Stream Data Processing / Fast Data
• Events are analyzed and processed in real-time as the arrive
• Decisions are timely, contextual and based on fresh data
• Decision latency is eliminated
• “Data in motion”
IoT Architecture
Key Challenges for building an IoT application
1. Connect: How to collect data from intelligent devices?
• Abstract complexity associated with device connectivity• Standardize integration of devices with enterprise
2. Analyze: How to analyze IoT data?
• Reduce noise and detect business event at real-time• Enable historical big-data analysis
3. Integrate: How to integrate IoT data & events with enterprise infrastructure?
• Make enterprise processes IoT friendly• Allow enterprise & mobile applications to control devices
Today) Existing Service-/API-based Architecture
21
MobileApps
DB
Rich(Web)ClientApps
DB
APIG
atew
ay
EnterpriseS
ervic
eBus(ESB)/DataIntegratio
n
EnterpriseApplications
WS
External CloudServiceProviders
BPMandSOAPlatform
Event
BusinessLogic/Rules
Business Intelligence
Services
WS
Event
Processes
Visualization
Analytics
DB
=oneway=request/response
REST SOAP
REST SOAP
SOAP
SOAP
any
SQL
SOAPREST
APIG
atew
ay
ServiceBu
sOracleDa
taIntegrator
SOASuite
BPMSuite
BusinessActivityMonitoring
IoT 1a) Reuse Service-/API-based Architecture
IoT Device
22
MobileApps
DB
Rich(Web)ClientApps
DB
APIG
atew
ay
EnterpriseS
ervic
eBus(ESB)/DataIntegratio
n
EnterpriseApplications
WS
External CloudServiceProviders
BPMandSOAPlatform
Event
BusinessLogic/Rules
Business Intelligence
Services
WS
Event
Processes
Visualization
Analytics
DB
=oneway=request/response
SOAP
REST
REST HTTP
JMS
SOAP
any
SQL
SOAP REST
WebSocket
REST SOAP
REST SOAP
APIG
atew
ay
ServiceBu
sOracleDa
taIntegrator
SOASuite
BPMSuite
BusinessActivityMonitoring
JMS
HTTP
IoT 1b) Reuse existing Service-/API-based Architecture
IoT Device
23
MobileApps
DB
Rich(Web)ClientApps
DB
APIG
atew
ay
EnterpriseS
ervic
eBus(ESB)/DataIntegratio
n
EnterpriseApplications
WS
External CloudServiceProviders
BPMandSOAPlatform
Event
BusinessLogic/Rules
Business Intelligence
Services
WS
Event
Processes
Visualization
Analytics
DB
WebSocket
REST SOAP
REST SOAP REST SOAP
RESTHTTP
REST
APIG
atew
ay
ServiceBu
sOracleDa
taIntegrator
SOASuite
BPMSuite
JMS
BusinessActivityMonitoring
REST HTTP
JMS
=oneway=request/response
IoT 1c) Reuse existing Service-/API-based Architecture
24
MobileApps
DB
Rich(Web)ClientApps
DB
EnterpriseS
ervic
eBus(ESB)/DataIntegratio
n
APIG
atew
ay
IoT Device Gateway
IoT SmartDevice
EnterpriseApplications
WS
External CloudServiceProviders
BPMandSOAPlatform
Event
BusinessLogic/Rules
Business Intelligence
Services
WS
Event
Processes
Visualization
Analytics
DB
REST SOAP
REST SOAP
SOASuite
BPMSuite
APIG
atew
ay
ServiceBu
sOracleDa
taIntegrator
BusinessActivityMonitoring
REST HTTP
JMS MQTT Kafka
REST HTTP
JMS MQTT Kafka
=oneway=request/response
Event Hub - Apache Kafka
Distributed publish-subscribe messaging system
Designed for processing of real time activity stream data (logs, metrics collections, social media streams, …)
Topic Semantic
Does not use JMS
Initially developed at LinkedIn, now part of Apache
Kafka Cluster
Consumer Consumer Consumer
Producer Producer Producer
Service Bus as a consumer of Kafka
ServiceBus12c
CloudApps
BusinessService Cl
oud
ProxyServiceKa
fka
Cloud
API
MobileApps Pipeline
Routing
KafkaSensor/IoT
WebApps
BusinessService RE
ST
BusinessService W
SDL
BackendAppsRE
ST
BackendAppsW
SDLProxy
ServiceKafka Pipeline
Routing
Database
DB CDC
StreamProcessing
IoT 2) Stream Processing / Analytics in Backend
27
MobileApps
DB
Rich (Web)Client Apps
DB
(ESB)/DataIntegratio
n
APIG
atew
ay
IoT Device Gateway
IoT SmartDevice
EventH
ub
StreamProcessingInfrastructure
ESP/CEP
DB
DB
EventH
ub
EnterpriseApplications
WS
External CloudServiceProviders
BPMandSOAPlatform
Event
BusinessLogic/Rules
Business Intelligence
Services
WS
Event
Processes
Visualization
Analytics
DB
=oneway=request/response
REST SOAP
REST SOAP
BusinessActivityMonitoring
SOASuite
BPMSuite
APIG
atew
ay
ServiceBu
sOracleDa
taIntegrator
StreamAnalytics
Oracle Stream Analytics as consumer of Kafka/MQTT
OracleStreamAnalytics
StreamAnalytics
Kafka
Kafka
MobileAppsKafka
Sensor/IoT
WebApps
Machine Data
DB CDC
Kafka
MQTT
Distribute runtime portion of IoT Analytics to the edge
Some IoT Analytics applications need to be distributed, so that processing can take place in devices, gateways, smart routers, servers at the site where sensor data is generated
Three main reasons:
• The factory, vehicle, home or other edge location needs to stay in operation, even when the corporate data center or cloud is down or not reachable
• Wide-are communication is generally to slow
• Transmitting all sensor data to corporate or cloud data center may be impractical or impossible if the volume of data is high or if reliable, high-bandwidth networks are unavailable
IoT 3) Stream Processing / Analytics at Edge
EnterpriseApplications
WS
External CloudServiceProviders
BPMandSOAPlatform
Event
BusinessLogic/Rules
Business Intelligence
Services
WS
Event
Processes
Visualization
Analytics
DB
30
MobileApps
DB
Rich (Web)Client Apps
DB
(ESB)/DataIntegratio
n
APIG
atew
ay
IoT DeviceGateway
IoT SmartDevice
EventH
ub
StreamProcessingInfrastructure
ESP/CEP
DB
DB
EventH
ubESP/CEP
=oneway=request/response
DB
REST SOAP
REST SOAP
StreamAnalyticsEdgeAnalytics
BusinessActivityMonitoring
SOASuite
BPMSuite
ServiceBu
sOracleDa
taIntegrator
APIG
atew
ay
IoT 4) Raw Data Storage and Batch Analytics
EnterpriseApplications
WS
External CloudServiceProviders
BPMandSOAPlatform
Event
BusinessLogic/Rules
Business Intelligence
Services
WS
Event
Processes
Visualization
Analytics
DB
31
MobileApps
DB
Rich (Web)Client Apps
DB
(ESB)/DataIntegratio
n
APIG
atew
ay
=oneway=request/response
IoT Device
IoT SmartDevice
EventH
ub
StreamProcessingInfrastructure
ESP/CEP
DB
DB
EventH
ub
BigDataInfrastructureHDFS
BatchProcessing
DB
REST SOAP
REST SOAP
Gateway
ESP/CEP
DB
EdgeAnalytics
StreamAnalytics
BusinessActivityMonitoring
SOASuite
BPMSuite
ServiceBu
sOracleDa
taIntegrator
BigDataAppliance
APIG
atew
ay
IoT 4a) Raw Event Replay Feature
EnterpriseApplications
WS
External CloudServiceProviders
BPMandSOAPlatform
Event
BusinessLogic/Rules
Business Intelligence
Services
WS
Event
Processes
Visualization
Analytics
DB
32
MobileApps
DB
Rich (Web)Client Apps
DB
(ESB)/DataIntegratio
n
APIG
atew
ay
=oneway=request/response
IoT Device
IoT SmartDevice
EventH
ub
StreamProcessingInfrastructure
ESP/CEP
DB
DB
EventH
ub
BigDataInfrastructureHDFS
Replay
REST SOAP
REST SOAP
Gateway
ESP/CEP
DB
EdgeAnalytics
BusinessActivityMonitoring
SOASuite
BPMSuite
ServiceBu
sOracleDa
taIntegrator
StreamAnalytics
BigDataAppliance
APIG
atew
ay
IoT 5) Data Mining and Model evaluation
EnterpriseApplications
WS
External CloudServiceProviders
BPMandSOAPlatform
Event
BusinessLogic/Rules
Business Intelligence
Services
WS
Event
Processes
Visualization
Analytics
DB
33
MobileApps
DB
Rich (Web)Client Apps
DB
(ESB)/DataIntegratio
n
APIG
atew
ay
=oneway=request/response
IoT Device
IoT SmartDevice
EventH
ub
StreamProcessingInfrastructure
ESP/CEP
DB
DB
EventH
ub
BigDataInfrastructureHDFS
BatchProcessing
DB
REST SOAP
REST SOAP
Gateway
ESP/CEP
DB
EdgeAnalytics
BusinessActivityMonitoring
SOASuite
BPMSuite
ServiceBu
sOracleDa
taIntegrator
StreamAnalytics
BigDataAppliance
APIG
atew
ay
IoT Services
IoT Logical Reference Architecture
IoTDevice
Sensor
Actuator
IoTGateway
Storage
UIApp
StreamingAnalytics
EnterpriseApplications
BPMandSOAPlatformStreaming
Analytics
Storage
EndpointManagement
EventHub
ServiceBus
EventHub Event
Hub
ServiceBus
BigData/BI
Storage
Services Processes
UIApp
Storage
BulkAnalytics UI
BulkAnalytics UI
Storage
StreamingAnalytics
ServiceBusAP
I
RESTSOAPHTTPKAFKA
MQTTCoAPXMPPDDSAMQPKAFKAWIFI
BLEZigBeeWIFIWired
IoT Services
IoT Logical Reference Architecture – Oracleon premises
IoTDevice
Sensor
Actuator
IoTGateway
Storage
UIApp
StreamingAnalytics
EnterpriseApplications
BPMandSOAPlatformStreaming
Analytics
Storage
EndpointManagement
EventHub
ServiceBus
EventHub Event
Hub
ServiceBus
BigData/BI
Storage
Services Processes
UIApp
Storage
BulkAnalytics UI
BulkAnalytics UI
Storage
StreamingAnalytics
ServiceBusAP
I
RESTSOAPHTTPKAFKA
MQTTCoAPXMPPDDSAMQPKAFKAWIFI
BLEZigBeeWIFIWired
EdgeAnalytics
BusinessActivityMonitoring
SOASuite
BPMSuiteServiceBus
OracleDataIntegrator
StreamAnalytics
BigDataAppliance
StreamAnalytics
ServiceBus
APIG
atew
ay
IoT ServicesIoT
Device
Sensor
Actuator
IoTGateway
Storage
UIApp
StreamingAnalytics
EnterpriseApplications
BPMandSOAPlatformStreaming
Analytics
Storage
EndpointManagement
EventHub
ServiceBus
EventHub Event
Hub
ServiceBus
BigData/BI
Storage
Services Processes
UIApp
Storage
BulkAnalytics UI
BulkAnalytics UI
Storage
StreamingAnalytics
ServiceBusAP
I
RESTSOAPHTTPKAFKA
MQTTCoAPXMPPDDSAMQPKAFKAWIFI
BLEZigBeeWIFIWired
IoT Logical Architecture – Open Source
Demo Use Case – Truck Movements
Demo Use Case – Truck Movements
Truck DataIngestion Geo-Fencing
2016-06-0214:39:56.605|98|27|MarkLochbihler|803014426|Wichita toLittle RockRoute 2|Normal|38.65|-90.21|5187297736652502631
{"timestamp": "2016-06-0214:39:56.991", "truckId":99,"driverId":31,"driverName":"Rommel Garcia", "routeId":1565885487, "routeName":"Springfield toKCViaHanibal","eventType":"Normal", "latitude":37.16,"longitude": "-94.46","correlationId":5187297736652502631}
RecklessDrivingDetector
NEAR
ENTER
TruckDriver
DashboardMovement MovementJSON
RecklessDriver
Oracle IoT Cloud Service
Oracle IoT Cloud Service within Oracle Cloud Platform
DocumentsCloud
ProcessCloud
SitesCloud
SocialCloud
CONTENTANDCOLLABORATION
BusinessIntelligence Cloud
BigDataDiscoveryCloud
BigDataPreparation
Cloud
InternetofThingsCloud
BUSINESSANALYTICS
NoSQLDatabaseCloud
BigDataSQLCloud
ExadataCloud
BigDataCloud
DatabaseBackupCloud
DatabaseCloud
BigDataPreparation
Cloud
DATAMANAGEMENT
NodeCloud
DeveloperCloud
MobileCloud
JavaCloud
ApplicationBuilderCloud
APPLICATIONDEVELOPMENT
Integration Cloud
SOACloud
GoldenGateCloud
InternetofThingsCloud
INTEGRATION
ApplicationPerformance Monitoring
Cloud
ITAnalyticsCloud
LogAnalyticsCloud
ENTERPRISEMANAGEMENT
MobileCloud
MOBILEIntegration Cloud
Internet ofThings Cloud
SitesCloud
DocumentsCloud
Oracle IoT Cloud Service
DeviceVirtualization
High SpeedMessaging
StreamProcessing
EndpointManagement
EventStore
IoTCloudService
EnterpriseConnectivity
Integration CloudService
BI&Big DataCloudService
OracleCloudServices
MobileCloudService
3rd partyapps
IndustryVertical Apps
Enterprise AppsCloudorOnPremise
Manufacturing
Transportation
ServiceMgmt
AssetMgmt
Firewall
OracleIoTCSGateways/w
3rd partygateways/wwith OracleIoT
Client Library
IoTCloudServiceClientLibraries &Gateway
IndirectlyconnectedDevices
Directlyconnecteddevice
Anatomy of an IoT Application in Oracle IoT CS
IoTApplication
Devices Messages Analytics Integrations Configurations
Real-time Analytics Applications Workflow
Devices Data/Alerts
Devices Data/Alerts
Devices Data/Alerts
Stream
Stream
......
Exploration
Patterns
Stream Processing Engine
Publish
Stream
StreamExplorerEditor StreamProcessingEngine
Stream Processing Applications
AnalyzedDataIoT CloudService
Summary
Summary
Treat events as events! Infrastructures for handling lots of events are available!
IoT tends to make Big Data / Fast Data infrastructures necessary
Know your use case/requirements to choose the right architecture!• Can my existing backend landscape handle the new IoT load?
• Do I have to handle huge amount of events in “real-time”?
• Do I need to filter/aggregate data before invoking existing backend systems?
• Do I want to do Advanced Analytics (predictive analytics) where historical information is necessary?
• What is the network bandwidth between device/gateway and cloud/backend?
• Centralized or Decentralized IoT solution?
J
Guido SchmutzTechnology Manager