oracle stream analytics - simplifying stream processing

BASEL BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENEVA HAMBURG COPENHAGEN LAUSANNE MUNICH STUTTGART VIENNA ZURICH

Oracle Stream AnalyticsSimplifying Stream Processing

29.9.2016 – DOAG 2016 Big Data DaysGuido Schmutz

Guido Schmutz

Working for Trivadis for more than 19 yearsOracle ACE Director for Fusion Middleware and SOACo-Author of different booksConsultant, Trainer, Software Architect for Java, SOA & Big Data / Fast DataMember of Trivadis Architecture BoardTechnology Manager @ Trivadis

More than 25 years of software development experience

Contact: guido.schmutz@trivadis.comBlog: http://guidoschmutz.wordpress.comSlideshare: http://www.slideshare.net/gschmutzTwitter: gschmutz

Oracle Stream Analytics - Simplifying Stream Processing2

Unser Unternehmen.

Trivadis ist führend bei der IT-Beratung, der Systemintegration, dem Solution Engineering und der Erbringung von IT-Services mit Fokussierung auf -und -Technologien in der Schweiz, Deutschland, Österreich und Dänemark. Trivadis erbringt ihre Leistungen aus den strategischen Geschäftsfeldern:

Trivadis Services übernimmt den korrespondierenden Betrieb Ihrer IT Systeme.

B E T R I E B

KOPENHAGEN

MÜNCHEN

LAUSANNEBERN

ZÜRICHBRUGG

HAMBURG

DÜSSELDORF

FRANKFURT

STUTTGART

FREIBURG

Mit über 600 IT- und Fachexperten bei Ihnen vor Ort.

14 Trivadis Niederlassungen mitüber 600 Mitarbeitenden.

Über 200 Service Level Agreements.

Mehr als 4'000 Trainingsteilnehmer.

Forschungs- und Entwicklungsbudget: CHF 5.0 Mio.

Finanziell unabhängig undnachhaltig profitabel.

Erfahrung aus mehr als 1'900 Projekten pro Jahr bei über 800 Kunden.

Agenda

1. Introduction to Streaming Analytics2. Oracle Stream Analytics3. Demo

Introduction to Streaming Analytics

Traditional Data Processing - Challenges

• Introduces too much “decision latency”

• Responses are delivered “after the fact”

• Maximum value of the identified situation is lost

• Decision are made on old and stale data

• “Data a Rest”

The New Era: Streaming Data Analytics / Fast Data

• Events are analyzed and processed in real-time as the arrive

• Decisions are timely, contextual and based on fresh data

• Decision latency is eliminated

• “Data in motion”

Event / Stream Processing Architecture

DataIngestion

Batchcompute

DataSources

Channel

DataConsumer

Reports

Service

AnalyticTools

AlertingTools

Content

Logfiles

Social

Sensor

Machine

(Analytical)Real-TimeDataProcessing

Stream/EventProcessing

ResultStore

Messaging

ResultStore

Oracle Stream Analytics - Simplifying Stream Processing

=DatainMotion =DataatRest9

“Lambda Architecture” for Big Data

DataIngestion

(Analytical)BatchDataProcessing

Batchcompute

ResultStoreDataSources

Channel

DataConsumer

Reports

Service

AnalyticTools

AlertingTools

Content

Social

Logfiles

Sensor

Machine

(Analytical)Real-TimeDataProcessing

Stream/EventProcessing

Batchcompute

Messaging

ResultStore

QueryEngine

ResultStore

ComputedInformation

RawData(Reservoir)

Oracle Stream Analytics - Simplifying Stream Processing

=DatainMotion =DataatRest

PullingIngestion

When to Stream / When not?

ConstantlowMilliseconds&under

Lowmillisecondstoseconds,delayincaseoffailures

10sofsecondsofmore,Re-runincaseoffailures

Real-Time Near-Real-Time Batch

“No free lunch”

ConstantlowMilliseconds&under

Lowmillisecondstoseconds,delayincaseoffailures

10sofsecondsofmore,Re-runincaseoffailures

Real-Time Near-Real-Time Batch

“Difficult”architectures,lowerlatency “Easierarchitectures”,higherlatency

Why Event / Stream Processing?

Visualize Business in real-time• Dashboards can help people to visualize, monitor and make sense of massive amount of

incoming data in real-time

Detect Urgent Situations• Based on simple or complex analytical patterns of urgent business events• Urgent because they happen in real-time

Automate immediate actions• Run in the background quietly until detecting an urgent situation (risk or opportunity)• Alerts can go to humans through email, text or push notifications or to other applications trough

message queues or service call

Oracle Stream Analytics

History of Oracle Stream Analytics

OracleComplexEventProcessing(OCEP)

OracleEventProcessing(OEP)

OracleStreamExplorer(SX)

OracleEventProcessingforJavaEmbedded

OracleStreamAnalytics(OSA)

OracleEdgeAnalytics(OAE)

BEAWeblogic EventServerOracleCQL

OracleIoT CloudService

• Filtering• Correlation• Aggregation• Pattern

matching

Devices / Gateways

Services

Computing Edge Enterprise

“Sea of data”

Macro-eventHigh-valueActionableIn-context

EDGEAnalytics

StreamAnalytics

• High Volume• Continuous Streaming• Extreme Low Latency• Disparate Sources• Temporal Processing• Pattern Matching• Machine Learning

Oracle Stream Analytics: From Noise to Value

• HighVolume• ContinuousStreaming• Sub-MillisecondLatency• DisparateSources• Time-WindowProcessing• PatternMatching

• HighAvailability/Scalability• CoherenceIntegration• Geospatial,Geofencing• BigDataIntegration

• BusinessEventVisualization

• Action!

Oracle Stream Analytics Platform

What it does• Compelling, friendly and visually stunning real time

streaming analytics user experience for Business users to dynamically create and implement Instant Insight solutions

Key Features• Analyze simulated or live data feeds to determine event

patterns, correlation, aggregation & filtering• Pattern library for industry specific solutions• Streams, References, Maps & Explorations

Benefits• Accelerated delivery time• Hides all challenges & complexities of underlying real-time

event-driven infrastructure

Oracle Stream Analytics – Self-Service Stream Processing!

Understanding of CQL Filtering, Correlation, Pattern: NOT NEEDED

Understanding of IT Deployment and Management: NOT NEEDED

Understanding of Development, Java, Best Practices: NOT NEEDED

Understanding of the Event Driven Platform: NOT NEEDED

Oracle Stream Analytics – Terminology

Explorer: The Application User Interface Catalog: The repository for browsing resources

Stream: incoming flow of events that you want to analyze (CSV, Kafka, JMS, Rest, MQTT, …)

Exploration: application that correlates events from streams and data sources, using filters, groupings, summaries, ranges, and more

Shape: A blueprint of an event in a stream or data in a data source. How the business data is represented in the selected stream

Map: collection of geo-fences

Reference: A connection to static data that is joined to a stream to enrich it and/or to be used in business logic and output

Pattern: A pre-built Exploration that addresses a particular business scenario in a focused and simplified User Interface

Connection: collection of metadata required to connect to an external system

Targets: defines an interface with a downstream system

Business accessibility to Geo-Streaming Analytics

Real Time Streaming Solutions face an increasing need to track "assets of interest" and initiate actions based on encroachment of boundary proximity to fixed and moving objects and other geographic, temporal, or event conditions.

Geo-Fence,Fence,Polygon

Geo-StreamingOracle Stream Analytics - Simplifying Stream Processing25

“Addvaluetoyourrealtimestreamingdatadiscoveryandanalyticsbyapplyingandincludingmathematical,statisticalanalysistotheliveoutputstream”

“Thesestreaming“Excelspreadsheets”reallydocometolife”

Expression Builder enabling calculations

Concept of Connections and their reuse in Streams

Decision Table for Nested IF-THEN-ELSE Rules

Topology View and Navigation

Relationship between Streams (Sources), References and Explorations

Oracle Stream Analytics Demo Use Case: Truck Movements

Truck DataIngestion Geo-Fencing

2016-06-0214:39:56.605|98|27|MarkLochbihler|803014426|WichitatoLittle RockRoute 2|Normal|38.65|-90.21|5187297736652502631

{"timestamp":"2016-06-0214:39:56.991","truckId":99,"driverId":31,"driverName":"RommelGarcia","routeId":1565885487,"routeName":"SpringfieldtoKCViaHanibal","eventType":"Normal","latitude":37.16,"longitude":"-94.46","correlationId":5187297736652502631}

RecklessDrivingDetector

TruckDriver

DashboardMovement MovementJSON

RecklessDriver

Continuous Ingestion in Stream Processing

DBSource

BigData

StreamProcessing

IoT Sensor

EventHub

IoT GW

ectCDC

DBSource

Log CDC Native

IoT Sensor

DataflowGW

MQTTGW

DataflowGW

Dataflow

TopicRE

ST33FileSourceLog

Social

Native

Apache Kafka – High-volume messaging system

Distributed publish-subscribe messaging system

Designed for processing of high-volume, real time activity stream data (logs, metrics collections, social media streams, …)

Topic Semantic

does not implement JMS standard!

Initially developed at LinkedIn, now part of Apache

Kafka Cluster

Consumer Consumer Consumer

Producer Producer Producer

Demo: Oracle Stream Analytics

Summary

Native Stream Processing => OEP server

Ingestion

Event Source

IndividualEvent

PPPPPPPPPPPP

Micro-Batch Stream Processing => Spark Streaming

Ingestion

Event Source

PPPPPP

Summary

Oracle Stream Analytics leverages the capabilities found in Oracle Event Processing (OEP)

Empowering Business users to gain insight into real-time information and take appropriate actions when needed => makes stream processing accessible

Makes Stream/Event Processing less technical => “Excel spread sheet” on Streams

Part of Oracle IoT Cloud Service

Support Spark Streaming as a deployment platform for Streaming ML

Interesting road map: Rule Engine, Machine Learning, Extensible Patterns

Oracle Stream Analytics on Docker

Oracle Stream Analytics 12.2.1 Documentation

Oracle Stream Analytics 12.2.1 Download

Guido SchmutzTechnology Manager

guido.schmutz@trivadis.com

oracle stream analytics - simplifying stream processing

Technology

stream analytics service in azure

simplifying advanced analytics to empower business managers

processing data with stream analytics jobs

imcsummit 2015 - day 2 general session - simplifying big...

real-time big data stream analytics

azure stream analytics deep dive - tetranoodle.com€¦ ·...

simplifying big data analytics: unifying batch and … ·...

event hub & azure stream analytics

azure stream analytics - github pages€¦ · analytics on...

realtime stream analytics with momentum

azure stream analytics by nico jacobs

oracle® fusion middleware oracle stream analytics...1.5...

engineering the communication stream with meaningful...

event stream analytics in oracle bam 12.1.3

stream analytics in the enterprise

simplifying big data analytics with apache spark

pentaho google hangout - simplifying analytics architecture...

event hubs and stream analytics

stream 2 - drive business performance with information...

jss 2015 - microsoft stream analytics