Highway to heaven - Microservices Meetup Dublin

Download Highway to heaven - Microservices Meetup Dublin

Post on 17-Jan-2017




0 download

Embed Size (px)


<p>Highway to heaven</p> <p>Highway to heavenBuilding microservices in the cloudMicroservices Meetup Dublin, December 2015</p> <p>AgendaMeet AutoScout24Shifting gears How we build our servicesHow we organize ourselves</p> <p>@cdeger</p> <p>Meetup AutoScout24Shifting gearsHow we build our services Event Sourcing DynamoDB as Atom Feed From documents to events Evolving architecture PriceEstimation Shared nothing? Infrastructure CSV Log processingHow we organize ourselves Autonomous teams Infrastructure guild Focus sliders</p> <p>AutoScout24 - 30.000ft</p> <p>Producing classifieds vs. consuming classifieds</p> <p>Describe business.Printed ads in paper analogy. Bold parts are in progress.</p> <p>Baseline AutoScout24 ITHighly optimized, but of last decadeIT platform supported growth for &gt;6 yearsMicrosoft oriented stackEnterprise IT setup - MTBF over MTTRProven agile and lean principles</p> <p>C</p> <p>Proven delivery engine</p> <p>Last decade: PC-Web, Data centerMicrosoft oriented stack: Team Windows, Team LinuxAvailability = MTBF / (MTBF + MTTR)</p> <p>New CEO</p> <p>Scout24 was sold end of 2013New CEO Greg Ellis beginning of 2014Are you ready for the future?21st century internet company?</p> <p>Do you attract talent?</p> <p>We could hire good .NET devs, but mostly from banks, insurances companiesNo internet background; we need to teachTalents that could help us where sparse</p> <p>We started think about our ecosystem and the flywheel that drives it. .NET small, slowLinux/JVM larger, faster</p> <p>Project Tatsu</p> <p>Fly at the speed of fear - DisruptiveJapanese dragon: Flying beastRollercoaster: Sixflags Magic Mountain</p> <p>Started Nov. 2014 with one team, now at 4 teams.</p> <p>Monolith to MicroservicesData center to AWS.NET / Windows to JVM / LinuxDevs + Ops to DevOps cultureInvolve product peopleFive challenges</p> <p>.NET/Windows -&gt; Scala/LinuxOwn Data centers -&gt; AWSMonolith + Swimlanes -&gt; MicroservicesDev + Ops -&gt; EngineersProduct: Include business, find core and add some polish -&gt; Shiny new cut</p> <p>Technical transformation</p> <p>Vertical product slicesStranglerSelf containedMicroservices include UI</p> <p>Change the wheels while driving</p> <p>Strategic GoalsGoals of the business sideArchitectural PrinciplesHigh-Level PrinciplesDesign and Delivery PrinciplesTactical measuresReduce Time to MarketSpeed, Fast FeedbackCost Efficiency Collect metrics to allow decisions cost vs. value.Support Data-Driven DecisionsListen to users and validate hypothesis.Provide as many relevant metrics &amp; data as possible.You build it, you run itThe team is responsible for shaping, building, running and maintaining its products. Fast feedback from live and customers helps us to continuously improve.Organized around Business CapabilitiesBuild teams around products not projects. Follow the domain and respect bounded contexts. Inverse Conway Maneuver.Shared NothingBy default avoid sharing and tight coupling, except for the big things in common. Dont create the next monolith.Macro and Micro ArchitectureClear separation. Autonomous micro services within the rules and constraints of the macro architecture.AWS FirstFavor AWS platform service over managed service, over self-hosted OSS, over self-rolled solutions.Data-Driven/ Metric-DrivenCollect metrics from processes and applications. Analyze, alert and act on them.Eliminate Accidental ComplexityStrive to keep it simple. Focus on essential complexity. You build one, you delete one.Autonomous TeamsMake fast local decisions. Be responsible. Know your boundaries. Share findings.Infrastructure As CodeAutomate everything: Reproducible, traceable and tested.Immutable servers over snowflake servers.Collaboration CultureEngineers from all backgrounds work together in collaborative teams as engineers and share responsibilities. No silos.Be BoldGo into production early. Value monitoring over tests. Recover and learn. Optimize for MTTR not MTBF.</p> <p>Security, Compliance and Data PrivacySecurity must be included from the beginning and everybodys concern. Keep data-privacy in mind.Another goalWork in progress... Containment and BoundariesAlign blast radius and vendor lock-in with the boundaries of the organization or business capabilities.</p> <p>Make your own. Keep evolving.This is for reference only.</p> <p>We use this to guide discussions.Poster hangs in every room.The items on the right support items on the left.</p> <p>New strategic business goals are coming. Discussions.</p> <p>PrinciplesOrganized around business capabilitiesYou build it, you run itBe boldMacro and micro architectureShared nothing</p> <p>Organized around Business CapabilitiesBuild teams around products not projects. Follow the domain and respect bounded contexts. Inverse Conway ManeuverYou build it, you run itResponsibility to invent, run and maintain a product stays with the building team. Fast feedback from live and customers helps us to continuously improve.Be BoldFaster and simpler. Go into production early. Value semantic monitoring over tests. Recover and learn. Optimize for MTTR not MTBF. Macro and Micro ArchitectureClear separation. Autonomous micro services within the rules and constraints of the macro architecture.</p> <p>Shared infrastructureShared nothingAvailability over shared nothingConvenience offeringsNo side effectsFast local decisions over committeeRespect family ties</p> <p>Sharing implies dependencies.Share only with good reason.</p> <p>Availability over Shared nothing -&gt; Monitoring, Logging, required, Macro, enables troubleshooting and correlation across service boundaries,We share infrastructure via AWS platformGoCD optional, convenience offering, but how to e.g. improve cycle time.No side effects. Dashing, global shared state. -&gt; Solution own dashing serverFast local decisions over committeesRespect family tiesOneScout over XScout24</p> <p>How (not) to shareUse over re-useRe-use only after hardeningCopy npaste, OSS, libraryPull instead of push</p> <p>Unexpected ways of sharing</p> <p>How many environments?Which versions on staging?Prod differs anyway: Load, data, patternsV2V3V6V5V4V7V5V8EngineerCIDevStagingV1V4Prod</p> <p>All CD patterns from the book are given!But what happens to CD with Microservices</p> <p>Dev and prod is enoughConsumer driven contractsShadow trafficSemantic monitoringSmoke tests or canary releasesV2V3V6V5V4V7V5V8EngineerCIDevProd</p> <p>All CD patterns from the book are given!Environments Fakes on dev, CDCs as glue, be bold</p> <p>UnifiedLogs</p> <p>Ease of adoptionPart of Base AMI, ConventionsEvent Publisher for KinesisApplication events over log files.CloudTrail also used for AlertsUnexpected production changeData LakeEverything is written S3.Additional consumers anticipatedHeartbeat of the platformNo ssh to machines</p> <p>Event Sourcing and data pumpsOne way data highwayEvent Sourcing - store history of all changes</p> <p>One way data highwayHistory of all changes</p> <p>Design decision: No queries against DC. Data needs to be pushed into AWSAbility to replay all events from beginning of time.Events = All changes to a classified</p> <p>SQS + S3Kinesis + S3Kinesis + DynamoDBSQS + DynamoDBProxy + DynamoDBDynamoDBEvolution</p> <p>Fast evolution</p> <p>Skip:SQS + S3 Inspired by ImmobilienScout24Kinesis + S3 Feedback from AWSKinesis + DynomaDB Queryable, Storage costs not prohibitive (7x)SQS + DynamoDB Queue better than LATEST or TRIM_HORIZONProxy + DynamoDB Changes are collected via queue table in OracleDynamoDB DynamoDB is global service</p> <p>Frontend integrationShared nothingAutonomous teamsOne domainHigh optimisation</p> <p>Shared nothing Jigsaw as lightweight as possible, not: frontend monolith, portal, behaviourNo shared asset pipeline. Bring your own asset (aka asset Brotzeit).Autonomous teamsMicroservices imply frontend integrationSelf contained MS over UI monolithOne domainOnly www, https onlySubdomainsLocal storage sandboxed by origin (protocol+host+port). Breaks watchlist, last search etc.Cookies are okHigh optimisationGoogle pagespeedCaching</p> <p>Frontend integration</p> <p>PageSpeed Module</p> <p>css (page+fragment)js (page+fragment)</p> <p>ngx_pagespeed</p> <p>css (page)js (page)</p> <p>css (fragment)js (fragment)</p> <p>No shared asset pipeline.</p> <p>Pagesare accessible via (localised) URLare owned by one teamcould be cacheableFragmentsare parts of a pagedont know the original request should send cache headersAssetsShould be combined and minifiedCachingCloudFront Caching: Caching on edge locations. Respects Cache Headers from Jigsaw.PageSpeed Caching: Caches combined assets.Backend Caching: Respects Cache Headers from microservices.</p> <p>How to build autonomous teamsDo not fall back into old behavioursBeware of Mandelbrot teamsPager duty so that you run itFix broken windowsPart-time ops not workingNot all T-shapes are the sameWolfW</p> <p>How to build autonomous teamsDo not fall back into old behavioursBeware of Mandelbrot teams - dev team and ops team within the devops teamPart-time ops not working Some devs do not like ops (and the other way round) Not all Ts are shaped the same.Pager duty enforces you build it you run it behaviour.How not to ignore the pager, no broken window syndromOn call rotationName and shame</p> <p>Infrastructure guildAgree on things to doShare learningsDelegate implementation to teamsEmpty backlog should be normalInfrastructure product teams needed?</p> <p>Avoid disconnect infrastructure and product teams.</p> <p>Why infrastructure guild?Agree on things to doShare learningsHow we work:Work is done in the teamsFocus work on infrastructure needed by team.(Macro decision: Shared infrastructure: Logging, Monitoring, Security)Teams are responsible - SubsidiarityHow to handle long running/blocked stories?Keep delegating and resist temptation to take overDont treat infrastructure story as neglected childEmpty backlog should be normalHurdle to create new tasks.We want as few shared infrastructure as possible.Shirky: Institutions will try to preserve the problem to which they are the solution.Creates it own tasks.Risk of stories without value or prioritisation.I have found an old sticky on the floor, let's implement that for two weeks!</p> <p>Two stacksCash stack meets shiny new stackOne companyLights on in cash stackFeature freezeWhere to build new features?Ease of integration helps business people</p> <p>There is a cash stack!</p> <p>Dont split into shiny new stack and legacy.Legacy is not used as a word anymore.Lights on in Cash stack - AWSFeature freeze: Where to build new features?Ease of integration between new and old stack helps business people to switch fluently between both worlds.Where to draw the line - Focus on technology change - Minimum viable integrationTime to market vs. fast re-platforming (N+1 systems)</p> <p>cdeger@autoscout24.com @cdeger</p> <p>AttributionsBlue sky, white-gray clouds by nature protector Natubico, www.vivism.info [CC BY-SA 3.0] http://commons.wikimedia.org/wiki/File%3ABlue_sky%2C_white-gray_clouds.JPGA Danish Perspective by NASA [Public domain] http://commons.wikimedia.org/wiki/File%3AA_Danish_Perspective.jpghttp://commons.wikimedia.org/wiki/File%3ANASAComputerRoom7090.NARA.jpgGREGEINRADAmazon16 by Neil Palmer/CIAT [CC BY-SA 2.0] https://www.flickr.com/photos/ciat/5641594952 BERGSTEIGERBarber in Cameroon by James Emery from Douglasville, United States (Daddy Joe_1355) [CC BY 2.0] http://commons.wikimedia.org/wiki/File%3ABarber_in_Cameroon.jpgWide objectives by Kivela (Own work) [Public domain] href="http://commons.wikimedia.org/wiki/File%3AWide_objectives.jpgTransformer Fire Barrier by GerryS1 (Own work) [CC BY-SA 3.0 or GFDL] http://commons.wikimedia.org/wiki/File%3ATransformer_Fire_Barrier.jpg</p> <p>Attributions (cont)Alonso Renault Pitstop Chinese GP 2008 by Bert van Dijk (Pitstop F1 ING Renault) [CC BY-SA 2.0] http://commons.wikimedia.org/wiki/File%3AAlonso_Renault_Pitstop_Chinese_GP_2008.jpgPrinciple of Panchasheel by Prakash Adhikary (Own work) [CC BY 3.0] http://commons.wikimedia.org/wiki/File%3APrinciple_of_Panchasheel.JPGTraffic Jam by Doo Ho Kim [CC BY-SA 2.0] https://www.flickr.com/photos/titicat/3049591547Pellets by The original uploader was Richard Mayer at German Wikipedia [GFDL or CC-BY-SA-3.0] http://commons.wikimedia.org/wiki/File%3APellets.jpgPipes and Valves by Uwe Hermann [CC BY-SA 2.0] https://www.flickr.com/photos/73628542@N00/6272975359Size variation in Coccinella undecimpunctata (2127991716) by Gilles San Martin from Namur, Belgium [CC BY-SA 2.0] http://commons.wikimedia.org/wiki/File%3ASize_variation_in_Coccinella_undecimpunctata_(2127991716).jpgMille crpe by Laitr Keiows (Own work) [CC BY-SA 3.0 or GFDL] http://commons.wikimedia.org/wiki/File%3AMille_cr%C3%AApe.jpgCountry Energy power line replacement 01 by Bidgee (Own work) [CC BY-SA 3.0] http://commons.wikimedia.org/wiki/File%3ACountry_Energy_power_line_replacement_01.jpgPuzzling by Bernd Gessler (Own work) [CC BY-SA 3.0] https://commons.wikimedia.org/wiki/File%3APuzzling.JPG</p> <p>Attributions (cont)Sharing Sucks (4536747557) by eyeliam from Portland, United States [CC BY 2.0] http://commons.wikimedia.org/wiki/File%3ASharing_Sucks_(4536747557).jpg7Line 9184 (8263568241) by Metropolitan Transportation Authority of the State of New York (7Line_9184 Uploaded by tm) [CC BY 2.0] http://commons.wikimedia.org/wiki/File%3A7Line_9184_(8263568241).jpgEngland rugby team 1905 by Russell &amp; Sons (The Graphic) [Public domain or Public domain] http://commons.wikimedia.org/wiki/File%3AEngland_rugby_team_1905.jpgWandergeselle by Sigismund von Dobschtz [CC BY-SA 3.0] http://commons.wikimedia.org/wiki/File%3AWandergeselle_02.JPGFaber-Rechenschieber 5304 by User:Karl Gruber (Own work) [CC BY-SA 4.0] http://commons.wikimedia.org/wiki/File%3AFaber-Rechenschieber_5304.JPGWheel clamps Texas by Richard Anderson from Denton, United States (Boots.) [CC BY-SA 2.0] http://commons.wikimedia.org/wiki/File%3AWheel_clamps_Texas.jpgGuadalupeNOLA15Oct07Thanks by Infrogmation of New Orleans (Photo by Infrogmation) [GFDL or CC BY-SA 3.0] http://commons.wikimedia.org/wiki/File%3AGuadalupeNOLA15Oct07Thanks.jpgAtariBasic by Calin99 (Own work) [GPL] http://commons.wikimedia.org/wiki/File%3AAtariBasic.pngSpare wheel by Brian Snelson [CC BY 2.0] https://commons.wikimedia.org/wiki/File:Spare_wheel_-_Flickr_-_exfordy.jpg</p>