getting data out of your cloud(s) with open source · getting data out of your cloud(s) with open...

GETTING DATA OUT OFGETTING DATA OUT OFYOUR CLOUD(S) WITHYOUR CLOUD(S) WITH

OPEN SOURCEOPEN SOURCEOBTAINING SUPERIOR INSIGHTOBTAINING SUPERIOR INSIGHT

Created'by'Cody'Herriges

INTRODUCTIONINTRODUCTIONPuppet'LabsOpera4ons'EngineerSystems'Team'lead

OUR PROBLEMSOUR PROBLEMSNo'consumable'historic'dataIncomplete'real'4me'dataData'collec4on'wasn't'eBec4vely'scalableTime'wasted'correla4ng'mul4ple'interfaces'by'handShort'lived'issues'almost'impossible'to'understandMetrics'gathering'can'require'secondary'purchases

OUR PRIVATE CLOUDSOUR PRIVATE CLOUDSAging'VMware'stack'(vCenter'graph'failures)Budding'OpenStack'expansion'(Incomplete'built'in'tooling)

GETTING MORE INFORMATIONGETTING MORE INFORMATIONA'4me'box'for'adequate'collec4on'is'not'reasonableVMware'KB:'2003885

CEILOMETER NOT TARGETED AT MECEILOMETER NOT TARGETED AT MENo'func4ons'to'help'analyze'codeNot'able'to'be'broken'down'by'hypervisorOriginally'developed'with'a'slant'towards'billing'and'chargebackLacking'4me'granularity

EMC'S MONITORING & REPORTINGEMC'S MONITORING & REPORTINGMostly'workedNever'had'a'4me'it'failed'to'provide'a'graphIts'downfall'was'the'inability'export'data'progroma4callyAutomated'reported'geared'towards'managers

CLOSEST I GOT TO AUTOMATING DATA EXPORTCLOSEST I GOT TO AUTOMATING DATA EXPORTA REPLAY OF A FIREBUG SESSION IN RUBYA REPLAY OF A FIREBUG SESSION IN RUBY

uri0 = URI.parse("http://vnx-reporting1.ops.puppetlabs.net:58080/VNX-MR/")

response0 = http.get(uri0.request_uri)

cookie = response0.get_fields('set-cookie')

uri1 = URI.parse("http://vnx-reporting1.ops.puppetlabs.net:58080/VNX-MR/j_security_check?j_password=changeme&j_username=admin"

response1 = http.get(uri1.request_uri, {'Cookie' => cookie.first})

cookie = response1.get_fields('set-cookie')

uri2 = URI.parse("http://vnx-reporting1.ops.puppetlabs.net:58080/VNX-MR/")

response2 = http.get(uri2.request_uri, { 'Cookie' => cookie.first })

all_cookies = response2.get_fields('set-cookie')

cookies_array = Array.new

all_cookies.each { | cookie |

cookies_array.push(cookie.split('; ')[0])

}

cookies = cookies_array.join('; ')

uri3 = URI.parse("http://vnx-reporting1.ops.puppetlabs.net:58080/VNX-MR/report.csv?report&select=0-0-cf550d56&display=0&mode=frg&statistics=none&lower=0.0&upper=$type=3&period=0&durationType=l&duration=15m&itz=Canada%2FPacific"

response3 = http.get(uri3.request_uri, { 'Cookie' => cookies })

MY SINGLE PANE OF GLASSMY SINGLE PANE OF GLASSNOT YOUR'S!NOT YOUR'S!

Vendor'single'panes'of'glass'are'seldom'singleMight'"work"'if'you're'the'tradi4onal'enterprisevirtualiza4on'or'storage'teamOnly'achievable'if'I'can'provide'my'own'inputs'andbusiness'logic

THE TOOLS THAT MADE IT POSSIBLETHE TOOLS THAT MADE IT POSSIBLEElas4cSearchLogstashKibanaGrafanaGraphiteDiamondStatsFeederCloudTrail

TACKLING MOST OF OUR VMWARE PROBLEMTACKLING MOST OF OUR VMWARE PROBLEMSTATSFEEDER+GRAPHITE+GRAFANASTATSFEEDER+GRAPHITE+GRAFANA

Wasn't'able'to'force'everyone'to'a'diBerent'plaZormStatsFeeder'grabs'real'4me'stats'for'all'hypervisors'andVMsDumps'to'CSV'and'we'batch'convert'to'graphitePulls'all'discrete'instances'of'vCenter'into'a'singledashboardpuppet[graphitepuppet[statsfeederBasic'con\g

LESS IMPORTANT VMWARE INFORMATIONLESS IMPORTANT VMWARE INFORMATIONELASTICSEARCH+LOGSTASH+KIBANA (ELK)ELASTICSEARCH+LOGSTASH+KIBANA (ELK)

Really'noisyHard'to'si^'throughSearchable'for'events'like'everything'involving'a'speci\cLUNvCenter'log'forwardingESXi'log'forwarding

THEN WE STARTED LEARNING THINGSTHEN WE STARTED LEARNING THINGS

WE ARE AND MUST PURCHASE MULTIPLEWE ARE AND MUST PURCHASE MULTIPLETIMES MORE HARDWARE THAN WE USE ONTIMES MORE HARDWARE THAN WE USE ON

AVERAGE!AVERAGE!

MAKING A PUBLIC CLOUD DECISIONMAKING A PUBLIC CLOUD DECISIONACTUALLY WE HAVE FOUR ALREADY...ACTUALLY WE HAVE FOUR ALREADY...

MAKING A BETTER PUBLIC CLOUD DECISIONMAKING A BETTER PUBLIC CLOUD DECISIONI'think'I'am'changing...Insight'and'manageability'over'performance

ELK+CLOUDTRAILELK+CLOUDTRAILUNDERSTANDING WHO IS DOING WHAT IN AWSUNDERSTANDING WHO IS DOING WHAT IN AWS

Track'every'API'callOnly'way'to'reliably'know'who'launched'a'speci\c'instanceidUsed'to'track'down'rogue'comprimised'instances

Tip:'

puppet[elas4csearchpuppet[logstash

S3'has'a'bat'namespaceResults'in'a'simple'logstash'con\g

DIAMOND+GRAPHITEDIAMOND+GRAPHITE"SOLVED" OUR STORGE METRICS ISSUES AND AUGMENTS CEILOMETER"SOLVED" OUR STORGE METRICS ISSUES AND AUGMENTS CEILOMETER

That'EMC'we'couldn't'automate...replaced'by'LinuxDiamond'kvm,'libvirtkvm,'and'iostat'collectors/sys/kernel/debug'and'/sys/fs/cgroup'must'be'mountedfor'kvm'statsgarethr[diamond'(we'forked'it)

NEW CHALLENGESNEW CHALLENGESHave'to'scale'Elas4cSearch'&'GraphiteData'normaliza4onMore'and'more'dashboards'to'generate

ENDENDCreated'by'Cody'Herriges'

PuppetConf''14