john plummer technical specialist data platform microsoft ltd streaminsight complex event processing...
TRANSCRIPT
John PlummerTechnical Specialist Data PlatformMicrosoft Ltd
StreamInsightComplex Event Processing (CEP) Platform
3
What is CEP?
3
Complex Event Processing (CEP) is the continuous and incremental processing of event streams from multiple sources based on declarative query and pattern specifications with near-zero latency.
Database Applications Event-driven Applications
Query Paradigm Ad-hoc queries or requests Continuous standing queries
Latency Seconds, hours, days Milliseconds or less
Data Rate Hundreds of events/sec Tens of thousands of events/sec or more
request
response
Eventoutput streaminput
stream
4
Example CEP Scenarios
4
Dat
a St
ream
Stream Data Store & Archive
Event Processing Engine
Dat
a St
ream
Asset Specs & Parameters
Power, Utilities:• Energy consumption• Outages• Smart grids• 100,000 events/sec
Visual trend-line and KPI monitoringBatch & product managementAutomated anomaly detectionReal-time customer segmentation Algorithmic tradingProactive condition-based maintenance
Web Analytics:• Click-stream data• Online customer
behavior• Page layout• 100,000 events /sec
Manufacturing:• Sensor on plant floor• React through device
controllers• Aggregated data • 10,000 events/sec
• Threshold queries• Event correlation from
multiple sources• Pattern queries
Lookup
Asset Instrumentation for Data Acquisition, Subscriptions to Data Feeds
Financial Services:• Stock & news feeds• Algorithmic trading• Patterns over time• Super-low latency• 100,000 events /sec
5
“Sweet Spot” for Complex Event Processing
Relational Database Applications
Financial Trading Applications
Aggregate Data Rate (Events/sec)
0 10 100 1000 10000 100000 higher
Months
Days
Hours
Minutes
Seconds
100 ms
< 1 ms
Operational Analytics Applications (e.g., Logistics)
Manufacturing ApplicationsMonitoring Applications
CEP Target Scenarios
Data Warehousing Applications Web Analytics Applications
6
Complex Event Processing
"Intelligence isn't just about knowing what is happening. It's about looking at the patterns in real time. If business people want to truly optimize their resources, they must examine patterns in real time.”
- Mike Gualtieri - Forrester
SITUATION TODAYSQL SERVER 2008 R2 STREAMINSIGHT
Pressure to handle large volumes of complex data in real time
The size and frequency of data make it challenging to store for data mining and analysisGrowing need to monitor, analyze and act on the data in motion
Process large volume of events across multiple data streams in less than a secondManage your business through historical data mining and continuous insights.
Built in support for different types of event handling and rich query semantics
7
MSFT CEP Engine
Reference data
APPLICATIONInput A
dapte
rs
EVENT SOURCES
Devices, Sensors
Web servers
Event stores & Databases
Stock tickers & News feeds
Outp
ut A
dapte
rs
EVENT TARGETS
Event stores & Databases
Pagers & Monitoring
devices
KPI Dashboards, SharePoint UI
Trading stations
Platform from Microsoft for continuous and incremental processing of event streams from multiple sources based on declarative query and pattern specifications with low latency
StreamInsight
Bloomberg.com
8
Events expose different temporal characteristicsPoint in time eventsInterval events with fixed durationInterval events with initially unknown duration (Edge)
Rich payloads capture all properties of an event
t1 t4t3t2 t5Time
Payl
oad/
val
ue
ab
c d e
Events
9
Event TypesEvents in Microsoft’s CEP platform use the .NET type systemEvents are structured and can have multiple fieldsFields are typed using the .NET framework types CEP engine provisioned timestamp fields capture all the different temporal event characteristicsEvent sources populate time stamp fields
Timestamps/Metadata
LongpumpID
StringType
StringLocation
Doubleflow
Doublepressure
… … … … … …
10
Event Streams & AdaptersA stream is a possibly infinite sequence of events
Insertions of new eventsChanges to event durationsExamples: time series sensor data, log entries
Stream characteristics:Event/data arrival patterns
Steady rate with end-of-stream indication Examples: files, tablesIntermittent, random, or in burstsExample: retail scanners, web, weather telemetryOrder of arrival of events does not match the order of theirapplication timestamps
Input & Output Design Time for dealing with Event Streams10
11
Core CEP Query EngineHosts “standing queries”
Operators consume and produce streams
Queries are composable
Query results are computed incrementally
11
• Query instance management:• Submit, start, stop
• Runtime statistics
• Takeaway: CEP engine does the heavy lifting for you when processing temporal event data
CEP Engine Output Adapters
Input Adapters Event
Standing Queries
Event
Event
Event
Event
Event
Event
Event
Event
12
Developer Experience with the CEP Platform
CEP development environmentBuilds on top of .NETMicrosoft Visual Studio as IDEApplications written in C#Queries written in LINQSupports various deployment models for CEP applications
Embedded applicationsServer applications
12
LINQ:
var queryFilter = from c in TestEventStreamwhere c.Field1 > 1select c;
13
StreamInsight Demo…
14
StreamInsight – More Information
StreamInsight - http://www.microsoft.com/sqlserver/2008/en/us/R2-complex-event.aspx
Team Blog - http://blogs.msdn.com/streaminsight/
Forum - http://social.msdn.microsoft.com/Forums/en-US/streaminsight/threads
© 2008 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED
OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.