SQL Server 2008 R2 StreamInsight Overview

NameTitle

Microsoft Corporation

Information Platform Vision

Empowered ITPervasive Insight

Dynamic

DevelopmentMISSION CRITICAL PLATFORM

Desktop & Mobile CloudServer & Datacenter

Scalable relational database platform

Consistent, familiar model & tools

Self-managed, highly available cloud services

MPP support for 10s to 100s TB DW

Highly scalable appliances

Seamless integration with Microsoft BI

A Comprehensive Platform

Managed Self-Service BI

Multi-server management

Virtualization & Live Migration

Enterprise-level security, scalability

High-scale, complex event processing

Data consistency across heterogeneous systems

MPP support for 10s-100s TB DW

Highly scalable appliancesSeamless BI Integration

Scalable relational database platformConsistent, familiar model & tools

Self-managed, highly available cloud service

TRUSTED, SCALABLE PLATFORM

IT & DEVELOPER EFFICIENCY

MANAGED SELF-SERVICE BI

Multi-server management

Virtualization & Live Migration

Accelerated development& deployment

Self-service analytics

Self-service reporting

Streamlined collaboration& management

COMPREHENSIVE PLATFORM FOR IT VALUE

What is CEP?

6

Complex Event Processing (CEP) is the continuous and incremental processing of event streams from multiple sources based on declarative query and pattern specifications with near-zero latency.

Database Applications

Event-driven Applications

Query Paradigm

Ad-hoc queries or requests

Continuous standing queries

Latency Seconds, hours, days Milliseconds or less

Data Rate Hundreds of events/sec Tens of thousands of events/sec or more

request

response

Eventoutput streaminput

stream

Relational Database Applications

Financial trading Applications

Scenarios for Event Processing

7

Aggregate Data Rate (Events/sec.)

Latency

0 10 100 1000 10000 100000 ~1million

Months

Days

hours

Minutes

Seconds

100 ms

< 1ms

Operational Analytics Applications, e.g., Logistics, etc.

Manufacturing ApplicationsMonitoring Applications

CEP Target Scenarios

Data Warehousing Applications

Web Analytics Applications

Example CEP Scenarios

8

Dat

a St

ream

Stream Data Store & Archive

Event Processing Engine

Dat

a St

ream

Asset Specs & Parameters

Power, Utilities:• Energy consumption• Outages• Smart grids• 100,000 events/sec

Visual trend-line and KPI monitoringBatch & product managementAutomated anomaly detectionReal-time customer segmentation Algorithmic tradingProactive condition-based maintenance

Web Analytics:• Click-stream data• Online customer

behavior• Page layout• 100,000 events /sec

Manufacturing:• Sensor on plant floor• React through device

controllers• Aggregated data • 10,000 events/sec

• Threshold queries• Event correlation from

multiple sources• Pattern queries

Lookup

Asset Instrumentation for Data Acquisition, Subscriptions to Data Feeds

Financial Services:• Stock & news feeds• Algorithmic trading• Patterns over time• Super-low latency• 100,000 events /sec

Industry trends• Data acquisition

costs are negligible

• Raw storage costs are small and continue to decrease

• Processing costs are non-negligible

• Data loading costs continue to be significant

Manage business via KPI-triggered

actions

Mine historical dataDevise new KPIs

Monitor KPIsRecord raw

data (history)

Virtuous Cycle: Monitor, Manage, Mine

9

CEP advantage• Process data

incrementally, i.e., while it is in flight

• Avoid loading while still doing the processing you want

• Seamless querying for monitoring, managing and mining

Microsoft’s CEP Solution

10

Data Sources, Operations, Assets, Feeds, Sensors, Devices

Monitor &

Record

Operational Data Store & Archive

CEP Engine

f(x) g(y)

CEP Enginef(x) f'(x)

g(y) h(x,y)

History

Deploy

Resultsf'(x) h(x,y)

Manage&

Benefit

Mine&

Design

Input Data Streams

Input Data Streams

OutputData Streams

CEP Deployment Alternatives

Data Sources

Aggregation & Correlation

11

CEP

CEP

CEP

CEP for lightweight processing and filtering

CEP for aggregation and correlation of in-flight events

CEP for complex analytics including historical data

Event processing engines are deployed at multiple places on different scales• At the edge – close to the data source• In the mid-tier – consolidate related data sources, • In the data center – historical archive, mining, large scale correlation

Devices

SensorsWeb servers

Feeds

CEP

CEP CEP

CEP

CEP CEP

CEP CEP CEP

Complex Analytics &Mining

MSFT CEP Engine

Reference data

APPLICATION

Input A

dapte

rs

EVENT SOURCES

Devices, Sensors

Web servers

Event stores & Databases

Stock tickers & News feeds

Outp

ut A

dapte

rs

EVENT TARGETS

Event stores & Databases

Pagers & Monitoring

devices

KPI Dashboards, SharePoint UI

Trading stations

Platform from Microsoft for continuous and incremental processing of event streams from multiple sources based on declarative query and pattern

specifications with low latency

KEY SOLUTION REQUIREMENTS

• Event driven• Low latency processing (< milliseconds)• High volume (>100,000 events /second)• Correlated events from multiple sources• Time interval oriented standing queries

Overview: CEP Platform from Microsoft

Bloomberg.com

PLATFORM FOR KEY MARKET OPPORTUNITIES• Manufacturing: Process monitoring i.e. scheduling, control, inventory • Financial: algorithmic trading, compliance monitoring, fraud detection• Web: Click stream analysis ,ad-delivery model• Utility: Grid monitoring • Healthcare: Patient Monitoring• Networking: traffic monitoring ,security intrusion detection• Telecom: Network monitoring• Oil and Gas: Geological data analysis, process control

Events• Events expose different temporal characteristics

−Point in time events−Interval events with fixed duration−Interval events with initially unknown duration• Rich payloads capture all properties of an event

t1 t4t3t2 t5Time

Payl

oad/

val

ue

ab

c de

Event Types

• Events in Microsoft’s CEP platform use the .NET type system• Events are structured and can have multiple fields• Fields are typed using the .NET framework types • CEP engine provisioned timestamp fields capture all the

different temporal event characteristics• Event sources populate time stamp fields

Timestamps/Metadata

LongpumpID

StringType

StringLocation

Doubleflow

Doublepressure

… … … … … …

Event Streams

• A stream is a possibly infinite sequence of events−Insertions of new events−Changes to event durations−Examples: time series sensor data, log entries

• Stream characteristics:−Event/data arrival patterns

‐ Steady rate with end-of-stream indication Examples: files, tables

‐ Intermittent, random, or in burstsExample: retail scanners, web, weather telemetry

−Out of order events‐ Order of arrival of events does not match the order of their

application timestamps

• Takeaway: CEP engine does the heavy lifting for you when dealing with out-of-order data

15

Event Stream Adapters

• Design time specification of an adapter‐ for a particular event type and particular source/sink‐ methods to handle event and stream behavior, and‐ properties to indicate adapter features to the engine• Adapter features

16

Event properties Adapter template for point in time and other event types

Stream properties Event arrival properties for out-of-order, arrival rate etc.

Payload specification .NET scalar types for payload fields

Typed vs. generic Adapter templates for single type vs. generic adapters

Support for optimization

Predicate/project pushdown property and supporting code

Problem Feature

Core CEP Query Engine

• Hosts “standing queries”− Operators consume and

produce streams

− Queries are composable

− Query results are computed incrementally

17

• Query instance management:− Submit, start, stop

− Runtime statistics

• Takeaway: CEP engine does the heavy lifting for you when processing temporal event data

CEP Engine

Output Adapters

Input Adapters

Event

Standing Queries

Event

Event

Event

Event

Event

Event

Event

Event

Typical CEP Queries

• Typical CEP queries require combination of functionality− Complex type describes event properties− Calculations introduce additional event properties− Grouping by one or more event properties− Aggregation for each event group over a pre-defined

period of time, typically a window− Multiple event groups monitored by the same query− Correlate event streams − Check for absence of activity with a data source− Enrich events with reference data− Collection of assets may change over time

• We want to make writing and maintaining those queries easy or even effortless

CEP Query Features

• Operators over streams− Calculations (PROJECT)− Correlation of streams from different data sources (JOIN)− Check for absence of activity with a data source (EXISTS)− Selection of events from streams (FILTER)− Stream partitioning (GROUP & APPLY)− Aggregation (SUM, COUNT, …) − Ranking and heavy hitters (TOP-K)− Temporal operations: hopping window, sliding window

• Extensibility – to add new domain-specific operators

• Queries are written over specific event types− They can be evaluated on all data sources with the same

event type

• Support for streaming data, reference data (lookup), and historical data (replay)

LINQ Query Examples

LINQ Example – GROUP&APPLY, WINDOW:

from e3 in MyStream3group e3 by e3.i into SubStreamsfrom s4 in SubStreamsfrom e4 in s4.SlidingWindow(FiveMinutes,ThreeSeconds)select new {

pl = new MyNewPayload(e4.i, e4.f)};

LINQ Example – JOIN, PROJECT, FILTER:

from e1 in MyStream1join e2 in MyStream2 e1.ID equals e2.IDwhere e1.f2 = “foo”select new { e1.f1, e2.f4 };

Join

FilterProject

Grouping

Window

Extensibility

• Built-in operators do not cover all functionality−Need for domain-specific extensions−Integrate with functionality available in existing libraries• Support for extensions in the CEP platform:

−User-defined operators, functions, aggregates−Code written in .NET, deployed as .NET assembly−Query operators can refer to functionality of the assembly−LINQ queries can easily refer to user-defined functionality• Temporal snap-shot operator framework

−Interface to implement user-defined operators−Manages operator state and snapshot changes−Framework does the heavy lifting to deal with intricate

temporal behavior such as out-of-order events

Developer Experience with the CEP Platform• Goals

− Easy to use and highly productive

− Most comprehensive toolset− Supports various deployment

models for CEP applications‐ Embedded applications‐ Server applications

− Compelling story for domain-specific extensions

• CEP development environment− Builds on top of .NET− Microsoft Visual Studio as IDE− Applications written in C#− Queries written in LINQ

22

LINQ:

var queryFilter = from c in TestEventStreamwhere c.Field1 > 1select c;

Managing CEP Data Sources & Queries• Typical CEP Application

‒ Multiple different data sources and adapters‒ Several queries• Intellectual property is in the queries• Protect investment in queries over time

‒ Re-use existing queries as newer version of an asset, device or data source become available

‒ Re-use existing queries for new data source types‒ Seamlessly switch between live data feeds and historical

data

Query Binding

• Binding a query with data sources yields a query instance• Configure multiple query instances for a query• Similar to data independence: the same query can be bound to

different data sources as long as they deliver the required event type

Data Sources Analytics & Queries

QT1

CEP EngineO

utput Adapters

Input Adapters

Event

Standing Queries

Event

Event

Event

Event

Event

AT2

AT1

Q1

Q1’

AT3Q1’’

CEP Query Configuration

• Query binding:−Coupling adapters with a query−Event types required by the

query need to be matched by types delivered by adapters

• Query can be re-used for all data sources of the same type• No changes to query

necessary for re-use

QueryTemplate

Output Adapter

Join

Proj.

Type Type

Type

InputAdapter

Type

Type

InputAdapter

Type

Query

Recap: CEP Platform from Microsoft

26

CEP Engine

Output Adapters

Input Adapters

Event

Standing Queries

Event sources Event targets

`

Event

Event

Event

Event

Event

Event

C_ID

C_NAME

C_ZIP

Event

Event

Static reference data

CEP Application Development

Development experience with .NET, C#, LINQ and Visual Studio 2008

CEP platform from Microsoft to build event-driven applications

Event-driven applications are fundamentally different from

traditional database applications: queries are

continuous, consume and produce streams, and compute

results incrementally

Flexible adapter SDK with high performance to

connect to different event sources and sinks

The CEP platform does the heavy lifting for you to deal

with temporal characteristics of event stream data

CEP Deployment Scenarios

CEP Engine

Reference data

Custom CEP Application

Scenario 1: Custom CEP Application Dev Scenario 2: Embed CEP in Application

ISV Application with CEP Engine

CEP Engine

Reference data

Scenario 4: Operational Intelligence w/ CEP

MadisonETL Pipeline with CEP engine

CEP EngineCEP CEP

CEP CEP

KPIs KPI mining

Scenario 3: CEP Enabled Device

Device with Embedded CEP Engine

CEP Engine

.NET, C#, LINQ

CEP Platform Roadmap

• Focus on −Custom development platform for CEP applications−CEP platform for Microsoft partners

• TAP (Technology Adopter Program) will launch in July 2009• Product availability targeted for 2010

timeJuly 2009

CTP2TAP program

2010

General Availability

CTP3TAP continues

Late 2009

A not-for-profit, user-run association to enable the SQL Server community to connect, share, and learn

Get ongoing SQL Server learning from MVPs, top experts, and real-world users:

• Training webcasts, technical education, worldwide event info, and moreSign up for your FREE PASS membership today: www.sqlpass.org

• Go to local PASS Chapter meetings Visit the PASS Chapter booth to learn about upcoming local events, or visit www.sqlpass.org/PASSChapters

• No local Chapter? Start a Chapter with support and resources from PASS

www.sqlpass.org/getchapterinfo

• Attend PASS Summit 2010World's largest and best conference for SQL Server and BI technical education www.sqlpass.org/na2010 Follow / Join us online: Twitter @sqlpass #sqlpass, LinkedIn group, Facebook

group

© 2009 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.

The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after

the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.