Architectures lambda - Microsoft

Download Architectures lambda - Microsoft

Post on 13-Apr-2017

891 views

Category:

Technology

1 download

Embed Size (px)

TRANSCRIPT

<ul><li><p>AMBIENT INTELLIGENCE</p><p>#mstechdays techdays.microsoft.fr </p><p>tech days2015camps</p></li><li><p>Architectures Lambda</p><p>Michel Hubert / Georges Damien</p><p>@michelhubert @georges_damien</p></li><li><p>tech.days 2015#mstechdaysArchitecture Lambda</p><p>Directeur Technique chez Cellenza</p><p>MVP Azure</p><p>Microsoft P-Seller</p></li><li><p>tech.days 2015#mstechdaysArchitecture Lambda</p><p>Consultant chez Cellenza</p></li><li><p>tech.days 2015#mstechdaysArchitecture Lambda</p><p> Architectures Lambda</p></li><li><p>tech.days 2015#mstechdaysArchitecture Lambda</p><p> La meilleure faon de prdire le futur est de regarder le pass et le prsent ! </p><p>Pourquoi les architectures lambda ?</p></li><li><p>tech.days 2015#mstechdaysArchitecture Lambda</p><p>Historiquement, le BigData est une suite logique de la B.I.</p><p>Donc on a appliqu les techniques de la B.I. : Le Batch</p><p>Ce nest pas plus suffisant !</p><p>Des flux de donnes prendre en compte en temps-rel</p><p>Des historiques trs volumineux qui reclent de la valeur</p><p>Pourquoi les architectures lambda ?</p></li><li><p>tech.days 2015#mstechdaysArchitecture Lambda</p><p>Architecture lambda ?</p></li><li><p>tech.days 2015#mstechdaysArchitecture Lambda</p><p>La base de donnes classique :</p><p>Architecture base sur des donnes immuables</p><p>UPDATE</p></li><li><p>tech.days 2015#mstechdaysArchitecture Lambda</p><p>Stockage immuable :</p><p>Architecture base sur des donnes immuables</p></li><li><p>tech.days 2015#mstechdaysArchitecture Lambda</p><p>Quels cas dusage ?</p></li><li><p>tech.days 2015#mstechdaysArchitecture Lambda</p></li><li><p>tech.days 2015#mstechdaysArchitecture Lambda</p><p>Prenons un scnario Exemple :</p><p> Analyse temps rel des comportements,</p><p> Calcul du Taux dabandon de panier,</p><p> Prvision de stock</p><p> Dtection de Fraude</p><p> Analyse dune campagne marketing</p><p> Quels produits ne dclenchent pas dachat ? Problme de stock ? De prix ?</p><p>Architecture Lambda</p></li><li><p>tech.days 2015#mstechdaysArchitecture Lambda</p><p>La Master Data</p></li><li><p>tech.days 2015#mstechdaysArchitecture Lambda</p><p>Azure Service Bus</p><p>Relay</p><p>Queue</p><p>Topic</p><p>Notification Hub</p><p>Event Hub</p><p>NAT and Firewall Traversal Service</p><p>Request/Response Services</p><p>Unbuffered with TCP Throttling</p><p>Many publishers and many consumers to </p><p>communicate over a FIFO like channel. </p><p>(Competing consumers and Queue-based </p><p>Load leveling scenarios)</p><p>Pub / Sub communication channel. Each </p><p>Consumer subscribes to a copy of message </p><p>High-scale notification distribution</p><p>Most mobile push notification services</p><p>Millions of notification targets</p></li><li><p>tech.days 2015#mstechdaysArchitecture Lambda</p><p> Event Hub vs Topics / Queues</p><p>Bus de messages</p><p>Topics / Queues Event Hub</p></li><li><p>tech.days 2015#mstechdaysArchitecture Lambda</p><p>Principe gnral</p><p>Event Producers</p><p>Azure Event Hub</p><p>&gt; 1M Producers&gt; 1GB/sec </p><p>Aggregate Throughput</p><p>Up to 32 partitions via portal, more on </p><p>request</p><p>Part</p><p>itio</p><p>ns </p><p>Direct </p><p>PartitionKey</p><p>Hash</p><p>Consumer Group(s)</p><p>Receivers</p><p>AMQP 1.0</p><p>Credit-based flow control</p><p>Client-side cursors</p><p>Offset by Id or Timestamp</p></li><li><p>tech.days 2015#mstechdaysArchitecture Lambda</p><p>Envoi des donnes</p><p>Partition 1</p><p>Partition 2</p><p>Partition n</p><p>Event 1</p><p>PartitionKey=A</p><p>Event 1</p><p>PartitionKey=B</p></li><li><p>tech.days 2015#mstechdays</p><p>Publication de message .Net / AMQP</p><p>Architecture Lambda</p></li><li><p>tech.days 2015#mstechdays</p><p>Publication de message via HTTP</p><p>Architecture Lambda</p><p>://.servicebus.windows.net//publisher//messages</p></li><li><p>tech.days 2015#mstechdaysArchitecture Lambda</p><p>Consommation des donnes</p><p>Partition 1</p><p>Partition 2</p><p>Partition n</p><p>Event 1</p><p>Pkey = A</p><p>Event 2</p><p>Pkey = B Receiver 6</p><p>Receiver 2</p><p>Worker n</p><p>Receiver 1</p><p>Receiver n</p><p>Worker 1</p></li><li><p>tech.days 2015#mstechdaysArchitecture Lambda</p><p>Consommation multiple</p><p>Partition 1</p><p>Partition 2</p><p>Partition n</p><p>Consumer Group C</p><p>Callback for prtn. 6</p><p>Callback for prtn. 2</p><p>Worker n</p><p>Callback for prtn. 1</p><p>Callback n</p><p>Worker 1Consumer Group B</p><p>Callback for prtn. 6</p><p>Callback for prtn. 2</p><p>Worker n</p><p>Callback for prtn. 1</p><p>Callback n</p><p>Worker 1Consumer Group A</p><p>Worker n</p><p>Callback for prtn. 6</p><p>Callback for prtn. 2</p><p>Callback for prtn. 1</p><p>Callback n</p><p>Worker 1</p></li><li><p>tech.days 2015#mstechdays</p><p>IEventProcessor</p><p>Architecture Lambda</p></li><li><p>tech.days 2015#mstechdays</p><p>IEventProcessor</p><p>Architecture Lambda</p></li><li><p>Architecture Lambda</p></li><li><p>tech.days 2015#mstechdaysArchitecture Lambda</p><p>Donnes en mouvement Complex Event Processing </p></li><li><p>tech.days 2015#mstechdaysArchitecture Lambda</p><p>Stream Analytics</p><p>SELECT count(*) FROM ParkingLot</p><p>WHERE type = 'Auto'AND color = 'Red'</p><p>QuestionCombien de voitures rouges dans le parking?</p><p>Rpondre avec une base de donne relationnelleMarcher jusquau parking</p><p>Compter les vhicules qui sont: Rouge, Voiture</p></li><li><p>tech.days 2015#mstechdaysArchitecture Lambda</p><p>Stream Analytics</p><p>La question est diffrenteCombien de voitures rouges sont passes au marqueur 18A sur lA-10 dans </p><p>la dernire heure?</p><p>Rpondre avec une base de donne relationnelleSarrter, faire se garer toutes les voitures qui arrivent pendant lheure dans un parking, les compter</p><p>Pas la meilleure des solutions</p></li><li><p>tech.days 2015#mstechdaysArchitecture Lambda</p><p>SELECT count(*) FROM A-10WHERE Type = Voiture and Color = RougeGROUP BY TumblingWindow(hour, 1)</p><p>La question est diffrenteCombien de voitures rouges sont passes au marqueur 18A sur lA-10 dans </p><p>la dernire heure?</p></li><li><p>tech.days 2015#mstechdaysArchitecture Lambda</p><p>TumblingWindow(minute,5) HoppingWindow(minute, 10 , 5)</p><p>SlidingWindow(minute, 3)</p></li><li><p>Architecture Lambda</p></li><li><p>tech.days 2015#mstechdaysArchitecture Lambda</p><p>Une usine donnes</p></li><li><p>Last Name First Name Country Age </p><p>Flasko Mike Canada 32</p><p>Anand Subbaraj USA 30</p><p>Gaurav Malhotra USA 72</p><p> . . .</p><p>Last Name First Name At risk of churning .</p><p>Flasko Mike Yes</p><p>Anand Subbaraj No</p><p>Gaurav Malhotra Yes</p><p> .</p></li><li><p>Call Log Files</p><p>Customer Table</p><p>Call Log Files</p><p>Customer Table</p><p>Customer Churn Table</p><p>Data Factory Concepts</p><p>Data Sources Ingest Transform &amp; Analyze Publish</p><p>Customer Call Details</p><p>Customers Likely to Churn</p></li><li><p>tech.days 2015#mstechdays</p><p>A managed cloud service for building &amp; operating data pipelines (aka. data flows) </p><p>1. Orchestrate, monitor &amp; schedule </p><p> compose data processing, storage &amp; movement services (on premises &amp; cloud)</p><p>2. Automatic infrastructure mgmt</p><p> combine pipeline intent w/ resource allocation &amp; mgmt</p><p> data movement as a service (global footprint &amp; on premises)</p><p>3. Single pane of glass</p><p> one place to manage your network </p><p>of data flows</p><p>Architecture Lambda</p></li><li><p>tech.days 2015#mstechdays</p><p>Pipeline (Active Period: July 2015 to July 2016) </p><p>Architecture Lambda</p></li><li><p>tech.days 2015#mstechdaysArchitecture Lambda</p></li><li><p>Architecture Lambda</p></li><li><p>tech.days 2015#mstechdaysArchitecture Lambda</p><p>Pour aller plus loin</p></li><li><p>tech.days 2015#mstechdaysArchitecture Lambda</p></li><li><p>tech.days 2015#mstechdaysArchitecture Lambda</p></li><li><p>tech.days 2015#mstechdaysArchitecture Lambda</p></li><li><p>tech.days 2015#mstechdaysArchitecture Lambda</p><p>Machine Learning</p></li><li><p>tech.days 2015#mstechdaysArchitecture Lambda</p><p>Machine Learning</p></li><li><p>Architecture Lambda</p></li><li><p>tech.days 2015#mstechdaysArchitecture Lambda</p><p>michel.hubert@cellenza.com</p><p>@michelhubert</p><p>Georges.damien@Cellenza.com</p>mailto:Michel.hubert@Cellenza.commailto:Georges.damien@Cellenza.com</li><li><p> 2015 Microsoft Corporation. All rights reserved. </p><p>#mstechdays techdays.microsoft.fr/camp</p><p>tech days2015camps</p><p>Architecture Lambda</p></li></ul>