Disposable Environments at Scale

Download Disposable Environments at Scale

Post on 05-Jul-2015




2 download

Embed Size (px)


Presentation from ZFS Day, 2 October 2012 (http://zfsday.com)


<ul><li> 1. Disposable Environments at Scale/ or: How I Learned to Stop Worrying and Love ZFS</li></ul> <p> 2. Eric Sproul Build Engineer SysadminConsultant Twitter: @eirescot 3. Recipe for Successenablement (n): the act of providing (someone)with adequate power, means, opportunity or authority(to do something)This is a story of how ZFS enabled business success 4. Background Etsy is the worlds handmade marketplace Experiencing rapid growthEvery department needed to understandhow the business was evolvingAsked OmniTI for help 5. The SituationNo data warehouseAnalytical queries against PostgreSQL OLTP Large initial size (~250 GB)Forecast to reach 1 TB+ within a year 6. Problems Long-running queries to OLTP destroy web performanceRe-running reports produces different results Inflexible reporting interface 7. Requirements Relieve pressure fromOLTP database Make production data continuously availableEnable correlation of other sources Flexible reporting UI 8. Solution Create separate BI analytics DB Build it on ZFS 9. Initial CapabilitiesETL to collate table-level datafrom multiple databasesRun deep analytic querieswithout impacting website New web UI enables ad-hoc reporting 10. Reaping the Benefits of ZFSSnapshotsFaster backupsSimple replica creation 11. Reaping the Benefits of ZFS Compression Extend usable life of storage:PgSQL logical: 1.3TOn-disk: 653G (2.0x)Intelligent resilver Shorter rebuilds == Reduced risk of data loss 12. We Want More! Monthly reports now on BI system Occasional problems require re-running reports Still get different results, same as before 13. We Want More!Need to test report changesFine to dev with small mock-upStaging requires something that looks like production 14. The Next LevelDisposable environmentsRun on slave replica R/W copy of BI data Discard when finished 15. Disposable EnvironmentUse a non-global zoneset zonepath=/zones/bistageset autoboot=trueset limitpriv=default,dtrace_proc,dtrace_userset ip-type=sharedadd netset address= physical=bnx0endadd datasetset name=bi01tank/stageend 16. Disposable EnvironmentStarting state: ZFS datasets bi01tank/pgsql/data bi01tank/pgsql/data/91 bi01tank/pgsql/wal_archive bi01tank/pgsql/wal_archive/91 17. Disposable EnvironmentTake snapshots zfs snapshot -r bi01tank/pgsql/data@stage bi01tank/pgsql/data@stage bi01tank/pgsql/data/91@stage bi01tank/pgsql/wal_archive@stage bi01tank/pgsql/wal_archive/91@stage 18. Disposable EnvironmentCreate clones zfs clone @stage bi01tank/pgsql/data@stage bi01tank/pgsql/data/91@stage bi01tank/pgsql/wal_archive@stage bi01tank/pgsql/wal_archive/91@stage bi01tank/stage/data bi01tank/stage/data/91 bi01tank/stage/wal_archive bi01tank/stage/wal_archive/91 19. Disposable EnvironmentZone now sees a full, writable copy of data Unchanged data is referenced to originChanges accumulate to clone 20. Disposable Environment pgsql/data/91 pgsql/data/91@stage stage/data/91 Unchanged data referenced from snapshot Change accounted to clone Live FS Snapshot Clone 21. Next-Level Results Any report can be re-runon the same data Massage existing data orbring more in for ad-hoc reportTest changes to reports and web UI 22. Next-Level ResultsWhen finished with the environment: Shut down zoneDelete clone &amp; origin snap 23. Return on Investment BI database runs on 2 machinesOLTP database lifetime extendedtwo years past expectation Faster, more granular, and ad-hoc reportingenables better decisions by management 24. Bonus! With the same technique,we can safely test:PostgreSQL upgrades Schema changes 25. Thank You ZFS, Zones and much more are available to the community viaillumos and its distributions Go forth and enable your business!</p>