GETTING DATA OUT OFGETTING DATA OUT OFYOUR CLOUD(S) WITHYOUR CLOUD(S) WITH
OPEN SOURCEOPEN SOURCEOBTAINING SUPERIOR INSIGHTOBTAINING SUPERIOR INSIGHT
Created'by'Cody'Herriges
INTRODUCTIONINTRODUCTIONPuppet'LabsOpera4ons'EngineerSystems'Team'lead
OUR PROBLEMSOUR PROBLEMSNo'consumable'historic'dataIncomplete'real'4me'dataData'collec4on'wasn't'eBec4vely'scalableTime'wasted'correla4ng'mul4ple'interfaces'by'handShort'lived'issues'almost'impossible'to'understandMetrics'gathering'can'require'secondary'purchases
OUR PRIVATE CLOUDSOUR PRIVATE CLOUDSAging'VMware'stack'(vCenter'graph'failures)Budding'OpenStack'expansion'(Incomplete'built'in'tooling)
GETTING MORE INFORMATIONGETTING MORE INFORMATIONA'4me'box'for'adequate'collec4on'is'not'reasonableVMware'KB:'2003885
CEILOMETER NOT TARGETED AT MECEILOMETER NOT TARGETED AT MENo'func4ons'to'help'analyze'codeNot'able'to'be'broken'down'by'hypervisorOriginally'developed'with'a'slant'towards'billing'and'chargebackLacking'4me'granularity
EMC'S MONITORING & REPORTINGEMC'S MONITORING & REPORTINGMostly'workedNever'had'a'4me'it'failed'to'provide'a'graphIts'downfall'was'the'inability'export'data'progroma4callyAutomated'reported'geared'towards'managers
CLOSEST I GOT TO AUTOMATING DATA EXPORTCLOSEST I GOT TO AUTOMATING DATA EXPORTA REPLAY OF A FIREBUG SESSION IN RUBYA REPLAY OF A FIREBUG SESSION IN RUBY
uri0 = URI.parse("http://vnx-reporting1.ops.puppetlabs.net:58080/VNX-MR/")
response0 = http.get(uri0.request_uri)
cookie = response0.get_fields('set-cookie')
uri1 = URI.parse("http://vnx-reporting1.ops.puppetlabs.net:58080/VNX-MR/j_security_check?j_password=changeme&j_username=admin"
response1 = http.get(uri1.request_uri, {'Cookie' => cookie.first})
cookie = response1.get_fields('set-cookie')
uri2 = URI.parse("http://vnx-reporting1.ops.puppetlabs.net:58080/VNX-MR/")
response2 = http.get(uri2.request_uri, { 'Cookie' => cookie.first })
all_cookies = response2.get_fields('set-cookie')
cookies_array = Array.new
all_cookies.each { | cookie |
cookies_array.push(cookie.split('; ')[0])
}
cookies = cookies_array.join('; ')
uri3 = URI.parse("http://vnx-reporting1.ops.puppetlabs.net:58080/VNX-MR/report.csv?report&select=0-0-cf550d56&display=0&mode=frg&statistics=none&lower=0.0&upper=$type=3&period=0&durationType=l&duration=15m&itz=Canada%2FPacific"
response3 = http.get(uri3.request_uri, { 'Cookie' => cookies })
MY SINGLE PANE OF GLASSMY SINGLE PANE OF GLASSNOT YOUR'S!NOT YOUR'S!
Vendor'single'panes'of'glass'are'seldom'singleMight'"work"'if'you're'the'tradi4onal'enterprisevirtualiza4on'or'storage'teamOnly'achievable'if'I'can'provide'my'own'inputs'andbusiness'logic
THE TOOLS THAT MADE IT POSSIBLETHE TOOLS THAT MADE IT POSSIBLEElas4cSearchLogstashKibanaGrafanaGraphiteDiamondStatsFeederCloudTrail
TACKLING MOST OF OUR VMWARE PROBLEMTACKLING MOST OF OUR VMWARE PROBLEMSTATSFEEDER+GRAPHITE+GRAFANASTATSFEEDER+GRAPHITE+GRAFANA
Wasn't'able'to'force'everyone'to'a'diBerent'plaZormStatsFeeder'grabs'real'4me'stats'for'all'hypervisors'andVMsDumps'to'CSV'and'we'batch'convert'to'graphitePulls'all'discrete'instances'of'vCenter'into'a'singledashboardpuppet[graphitepuppet[statsfeederBasic'con\g
LESS IMPORTANT VMWARE INFORMATIONLESS IMPORTANT VMWARE INFORMATIONELASTICSEARCH+LOGSTASH+KIBANA (ELK)ELASTICSEARCH+LOGSTASH+KIBANA (ELK)
Really'noisyHard'to'si^'throughSearchable'for'events'like'everything'involving'a'speci\cLUNvCenter'log'forwardingESXi'log'forwarding
THEN WE STARTED LEARNING THINGSTHEN WE STARTED LEARNING THINGS
WE ARE AND MUST PURCHASE MULTIPLEWE ARE AND MUST PURCHASE MULTIPLETIMES MORE HARDWARE THAN WE USE ONTIMES MORE HARDWARE THAN WE USE ON
AVERAGE!AVERAGE!
MAKING A PUBLIC CLOUD DECISIONMAKING A PUBLIC CLOUD DECISIONACTUALLY WE HAVE FOUR ALREADY...ACTUALLY WE HAVE FOUR ALREADY...
MAKING A BETTER PUBLIC CLOUD DECISIONMAKING A BETTER PUBLIC CLOUD DECISIONI'think'I'am'changing...Insight'and'manageability'over'performance
ELK+CLOUDTRAILELK+CLOUDTRAILUNDERSTANDING WHO IS DOING WHAT IN AWSUNDERSTANDING WHO IS DOING WHAT IN AWS
Track'every'API'callOnly'way'to'reliably'know'who'launched'a'speci\c'instanceidUsed'to'track'down'rogue'comprimised'instances
Tip:'
puppet[elas4csearchpuppet[logstash
S3'has'a'bat'namespaceResults'in'a'simple'logstash'con\g
DIAMOND+GRAPHITEDIAMOND+GRAPHITE"SOLVED" OUR STORGE METRICS ISSUES AND AUGMENTS CEILOMETER"SOLVED" OUR STORGE METRICS ISSUES AND AUGMENTS CEILOMETER
That'EMC'we'couldn't'automate...replaced'by'LinuxDiamond'kvm,'libvirtkvm,'and'iostat'collectors/sys/kernel/debug'and'/sys/fs/cgroup'must'be'mountedfor'kvm'statsgarethr[diamond'(we'forked'it)
NEW CHALLENGESNEW CHALLENGESHave'to'scale'Elas4cSearch'&'GraphiteData'normaliza4onMore'and'more'dashboards'to'generate
