microsoft streaminsight

Click here to load reader

Post on 24-Feb-2016

74 views

Category:

Documents

0 download

Embed Size (px)

DESCRIPTION

Microsoft StreamInsight. Allan Mitchell Konesans Limited. Who am I?. Allan Mitchell Joint author on “First to Market” SSIS Book by Wrox Websites www.SQLIS.com www.SQLDTS.com www.konesans.com StreamInsight and SQL Azure Advisory Boards Microsoft SQL Server MVP Twitter : allanSQLIS - PowerPoint PPT Presentation

TRANSCRIPT

Extending SSIS with Custom Tasks

Microsoft StreamInsight

Allan MitchellKonesans Limited

4/8/2011 3:42 PM 2007 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

1Who am I?Allan Mitchell Joint author on First to Market SSIS Book by WroxWebsites www.SQLIS.com www.SQLDTS.comwww.konesans.comStreamInsight and SQL Azure Advisory BoardsMicrosoft SQL Server MVPTwitter: allanSQLISeMail: [email protected]

AgendaWhat is Stream ProcessingUnderstanding Streaming DataDifferences to RDBMSScenariosStreamInsight ArchitectureConceptsTypical Patterns

What is Event Stream Processing

QueryDatabase loads data from diskYou query databaseData is staleWhat is Stream ProcessingContinuous processing of event streamsfrom multiple sourcesbased on a declarative query languagewith near-zero latency

StreamInsightQueryUnderstanding Streaming DataUsing an RDBMS:

Q. Given a time interval tell me the amount of Electricity units used per household per time interval from now until I say stop.Not EasyRequires requerying/pollingNeed to store the end of the last interval6Presumes at intervals you get units used 2008 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.4/8/2011Understanding Streaming DataThis is the streaming data paradigm in a nutshell ask questions about data in flight. Answering with StreamInsight

Create a window over the event streamPartition/Group By HouseholdSUM() readings within window

7Will show an example of this.Group by essentially splits the streams 2008 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.4/8/2011Complex Event Processing (CEP) is the continuous and incremental processing of event streams from multiple sources based on declarative query and pattern specifications with near-zero latency. What is CEP?SQL Server ConnectionsUpdates will be available at http://www.devconnections.com/updates/LasVegas_Fall09/SQL8Complex Event Processing (CEP) is the continuous and incremental processing of event streams from multiple sources based on declarative query and pattern specifications with near-zero latency. Database ApplicationsEvent-driven ApplicationsQuery ParadigmAd-hoc queries or requestsContinuous standing queriesWhat is CEP?SQL Server ConnectionsUpdates will be available at http://www.devconnections.com/updates/LasVegas_Fall09/SQL9Complex Event Processing (CEP) is the continuous and incremental processing of event streams from multiple sources based on declarative query and pattern specifications with near-zero latency. Database ApplicationsEvent-driven ApplicationsQuery ParadigmAd-hoc queries or requestsContinuous standing queriesLatencySeconds, hours, daysMilliseconds or lessWhat is CEP?SQL Server ConnectionsUpdates will be available at http://www.devconnections.com/updates/LasVegas_Fall09/SQL10Complex Event Processing (CEP) is the continuous and incremental processing of event streams from multiple sources based on declarative query and pattern specifications with near-zero latency. Database ApplicationsEvent-driven ApplicationsQuery ParadigmAd-hoc queries or requestsContinuous standing queriesLatencySeconds, hours, daysMilliseconds or lessData Query RateHundreds of events/sec> Tens of thousands of events/secWhat is CEP?SQL Server ConnectionsUpdates will be available at http://www.devconnections.com/updates/LasVegas_Fall09/SQL11Relational Database Applications Financial trading ApplicationsScenarios for Event-Driven Applications12Aggregate Data Rate (Events/sec.)Latency010100100010000100000~1millionMonthsDayshoursMinutesSeconds100 ms< 1msOperational Analytics Applications, e.g., Logistics, etc.Manufacturing ApplicationsMonitoring ApplicationsStreamInsight Target ScenariosData Warehousing ApplicationsWeb Analytics ApplicationsCEP Platform from Microsoft - Overview12Who might need CEPFraud DetectionReal-Time Trade RiskingAlgorithmic Trading/BettingMeter throughputsOil, Gas, Water, ElectricityUse to drive alarms, alerts etcTelemetry DataWindowingHoppingTumblingSnapshotCountTime WindowsTimeHopping Window15Time WindowsTimeTumbling Window16Snapshot Windows

Snapshot Windows

StreamInsight Key FeaturesRuntime environment for continuous and event-driven processingIn-memory processing, no persistenceDeclarative query language: LINQ.NET APIs for Input/Output AdaptersFlexible deployment modelsExtensibility: Aggregates, OperatorsRich diagnostic & debugging interfaceNo persistence is a biggie. This means if StreamInsight fails, the app dev is in charge of making sure that you restart from where you left off. This is different to some other engines that go through storage first. The problem there could be that the storing of the data and the retrieval of the data from storage could be expensive. You can of course pipe to dual destinations

We are covering Extensibility and the debugging interface

Does favour .Net developers. There is no UI per se so you have to be comfortable with coding. Streambase though has an SSIS style UI

19Standing QueriesQuery LogicEvent sourcesEvent targets

Devices, SensorsWeb serversEvent stores & Databases

Stock ticker, news feeds

Event stores & Databases

Pagers &Monitoring devicesKPI Dashboards, SharePoint UITrading stationsStreamInsightApplication DevelopmentInputAdaptersOutputAdaptersStreamInsight EngineQuery LogicQuery LogicStreamInsight Platform1234

StreamInsight Application at RuntimeSQL Server ConnectionsUpdates will be available at http://www.devconnections.com/updates/LasVegas_Fall09/SQL20Microsoft StreamInsight21CEP EngineOutput Adapters

Input AdaptersEventStanding Queries

Event sourcesEvent targets

EventEventEventEventEventEventC_IDC_NAMEC_ZIP

EventEvent

Static reference data

CEP Application DevelopmentDevelopment experience with .NET, C#, LINQ and Visual Studio 2008/10CEP platform from Microsoft to build event-driven applicationsEvent-driven applications are fundamentally different from traditional database applications: queries are continuous, consume and produce streams, and compute results incrementallyFlexible adapter SDK with high performance to connect to different event sources and sinksThe CEP platform does the heavy lifting for you to deal with temporal characteristics of event stream data

21TechEd North America 2009Central ConceptsEventsUser Payload + temporal propertiesType based on .NET structure or classPoint / Interval / EdgeStreamsSequence of events with specific typeFlow into standing queriesQueriesComposed of query operatorsApply desired semantics on eventsIn-memory, event-triggered, incremental computationsAdaptersProduce / Consume event streams (Connection Manager)Typical Query & Analytics PatternsIntroduce additional event fieldsFilter out eventsGrouping by one or more event propertiesAggregation for each event group over a set (a window)Rank events within windowsCorrelate and synchronize event streamsCheck for absence of eventsCreate result only when input changesRemove duplicatesReduce event rate, resample streamEnrich events with reference dataCollection of assets may change over timeLINQ Query Examples LINQ Example GROUP&APPLY, WINDOW:

from e3 in MyStream3group e3 by e3.i into SubStreamfrom win in SubStream.HoppingWindow( FiveMinutes,ThreeSeconds)select new { i = SubStream.Key, a = win.Avg(e => e.f) };LINQ Example JOIN, PROJECT, FILTER:

from e1 in MyStream1join e2 in MyStream2 on e1.ID equals e2.IDwhere e1.f2 == fooselect new { e1.f1, e2.f4 };JoinFilterProjectGroupingWindowProject &AggregateExtensibilityNeed for domain-specific extensionsIntegrate with functionality available in existing librariesUser-defined operators, functions, aggregatesCode written in .NET, deployed as .NET assemblyWindow-based semantics for UDOs, UDAsReceive a set of eventsProduce a value / set of eventsSQL Server ConnectionsUpdates will be available at http://www.devcon