splunk:uningops and*dev*(before* devops*was*cool)* · cloud hosted database servers enernoc hosted...
TRANSCRIPT
![Page 1: Splunk:UningOps and*Dev*(before* DevOps*was*cool)* · Cloud Hosted Database Servers EnerNOC Hosted Web Servers EnerNOC Hosted Application Servers Cloud Hosted Web Servers Cloud Hosted](https://reader035.vdocuments.mx/reader035/viewer/2022062922/5f08a49d7e708231d42304da/html5/thumbnails/1.jpg)
Copyright © 2015 Splunk Inc.
Grace Sumner Sr. Produc>on Opera>ons Engineer, EnerNOC
Splunk: Uni>ng Ops and Dev (before DevOps was cool)
![Page 2: Splunk:UningOps and*Dev*(before* DevOps*was*cool)* · Cloud Hosted Database Servers EnerNOC Hosted Web Servers EnerNOC Hosted Application Servers Cloud Hosted Web Servers Cloud Hosted](https://reader035.vdocuments.mx/reader035/viewer/2022062922/5f08a49d7e708231d42304da/html5/thumbnails/2.jpg)
Disclaimer
2
During the course of this presenta>on, we may make forward looking statements regarding future events or the expected performance of the company. We cau>on you that such statements reflect our current expecta>ons and es>mates based on factors currently known to us and that actual events or results could differ materially. For important factors that may cause actual results to differ from those contained in our forward-‐looking statements, please review our filings with the SEC. The forward-‐looking statements made in the this presenta>on are being made as of the >me and date of its live presenta>on. If reviewed aUer its live presenta>on, this presenta>on may not contain current or
accurate informa>on. We do not assume any obliga>on to update any forward looking statements we may make.
In addi>on, any informa>on about our roadmap outlines our general product direc>on and is subject to change at any >me without no>ce. It is for informa>onal purposes only and shall not, be incorporated into any contract or other commitment. Splunk undertakes no obliga>on either to develop the features
or func>onality described or to include any such feature or func>onality in a future release.
![Page 3: Splunk:UningOps and*Dev*(before* DevOps*was*cool)* · Cloud Hosted Database Servers EnerNOC Hosted Web Servers EnerNOC Hosted Application Servers Cloud Hosted Web Servers Cloud Hosted](https://reader035.vdocuments.mx/reader035/viewer/2022062922/5f08a49d7e708231d42304da/html5/thumbnails/3.jpg)
Agenda • EnerNOC
• Splunk and EnerNOC – Dev and Ops and Dev …
• Splunk raises all ships (my story)
• Our pain is your gain (lessons learned)
• Q&A
3
![Page 4: Splunk:UningOps and*Dev*(before* DevOps*was*cool)* · Cloud Hosted Database Servers EnerNOC Hosted Web Servers EnerNOC Hosted Application Servers Cloud Hosted Web Servers Cloud Hosted](https://reader035.vdocuments.mx/reader035/viewer/2022062922/5f08a49d7e708231d42304da/html5/thumbnails/4.jpg)
The Power of
4
Small Data. Big Data. Huge Data.
COLLECT DATA FROM ANYWHERE
SEARCH AND ANALYZE EVERYTHING
GAIN REAL-‐TIME OPERATIONAL INTELLIGENCE
![Page 5: Splunk:UningOps and*Dev*(before* DevOps*was*cool)* · Cloud Hosted Database Servers EnerNOC Hosted Web Servers EnerNOC Hosted Application Servers Cloud Hosted Web Servers Cloud Hosted](https://reader035.vdocuments.mx/reader035/viewer/2022062922/5f08a49d7e708231d42304da/html5/thumbnails/5.jpg)
About Grace
…Actually, lets do that later.
5
![Page 6: Splunk:UningOps and*Dev*(before* DevOps*was*cool)* · Cloud Hosted Database Servers EnerNOC Hosted Web Servers EnerNOC Hosted Application Servers Cloud Hosted Web Servers Cloud Hosted](https://reader035.vdocuments.mx/reader035/viewer/2022062922/5f08a49d7e708231d42304da/html5/thumbnails/6.jpg)
EnerNOC at a Glance
6
A leading provider of SaaS-‐based Energy Intelligence SoUware (EIS) and related solu>ons
Market leader in demand response (DR)
Global company (over 1,300 employees in countries across North America, APAC, Europe) with HQ in Boston, MA
![Page 7: Splunk:UningOps and*Dev*(before* DevOps*was*cool)* · Cloud Hosted Database Servers EnerNOC Hosted Web Servers EnerNOC Hosted Application Servers Cloud Hosted Web Servers Cloud Hosted](https://reader035.vdocuments.mx/reader035/viewer/2022062922/5f08a49d7e708231d42304da/html5/thumbnails/7.jpg)
EnerNOC’s Energy Intelligence SoUware For enterprises: pladorm and solu>ons focus on the 3 drivers of energy expense
How you buy it
Budgets and Procurement
Utility Bill Management (UBM)
How much you use
Visibility and Reporting
Facility Optimization Project Tracking
When you use it
Demand Response
Demand Management
![Page 8: Splunk:UningOps and*Dev*(before* DevOps*was*cool)* · Cloud Hosted Database Servers EnerNOC Hosted Web Servers EnerNOC Hosted Application Servers Cloud Hosted Web Servers Cloud Hosted](https://reader035.vdocuments.mx/reader035/viewer/2022062922/5f08a49d7e708231d42304da/html5/thumbnails/8.jpg)
8
More than 70,000 sites and devices stream data into EnerNOC’s energy intelligence software platform
Over 6,000 companies globally rely on EnerNOC to drive energy savings
Demonstrated expertise trusted by the largest companies in the world:
EnerNOC Is Transforming Energy Management Across Industries
Increase Use of Enterprise Energy Intelligence SoKware
![Page 9: Splunk:UningOps and*Dev*(before* DevOps*was*cool)* · Cloud Hosted Database Servers EnerNOC Hosted Web Servers EnerNOC Hosted Application Servers Cloud Hosted Web Servers Cloud Hosted](https://reader035.vdocuments.mx/reader035/viewer/2022062922/5f08a49d7e708231d42304da/html5/thumbnails/9.jpg)
9
ULlity and Grid Operator Partnerships EnerNOC has Extensive Exper>se Working With U>li>es and Grid Operators Globally
Our utility partners include:
We also operate in wholesale markets:
![Page 10: Splunk:UningOps and*Dev*(before* DevOps*was*cool)* · Cloud Hosted Database Servers EnerNOC Hosted Web Servers EnerNOC Hosted Application Servers Cloud Hosted Web Servers Cloud Hosted](https://reader035.vdocuments.mx/reader035/viewer/2022062922/5f08a49d7e708231d42304da/html5/thumbnails/10.jpg)
The Case for Demand Response Balancing supply and demand on the electricity grid is difficult and expensive. End users that provide a balancing resource are compensated for the service.
Annual Electricity Demand As a Percent of Available Capacity
50%
100%
Winter Spring Summer Fall
75%
25%
90%
![Page 11: Splunk:UningOps and*Dev*(before* DevOps*was*cool)* · Cloud Hosted Database Servers EnerNOC Hosted Web Servers EnerNOC Hosted Application Servers Cloud Hosted Web Servers Cloud Hosted](https://reader035.vdocuments.mx/reader035/viewer/2022062922/5f08a49d7e708231d42304da/html5/thumbnails/11.jpg)
Ops vs Ops
11
![Page 12: Splunk:UningOps and*Dev*(before* DevOps*was*cool)* · Cloud Hosted Database Servers EnerNOC Hosted Web Servers EnerNOC Hosted Application Servers Cloud Hosted Web Servers Cloud Hosted](https://reader035.vdocuments.mx/reader035/viewer/2022062922/5f08a49d7e708231d42304da/html5/thumbnails/12.jpg)
Ops vs Ops
12
![Page 13: Splunk:UningOps and*Dev*(before* DevOps*was*cool)* · Cloud Hosted Database Servers EnerNOC Hosted Web Servers EnerNOC Hosted Application Servers Cloud Hosted Web Servers Cloud Hosted](https://reader035.vdocuments.mx/reader035/viewer/2022062922/5f08a49d7e708231d42304da/html5/thumbnails/13.jpg)
13
Usually when we talk about Dev Ops:
![Page 14: Splunk:UningOps and*Dev*(before* DevOps*was*cool)* · Cloud Hosted Database Servers EnerNOC Hosted Web Servers EnerNOC Hosted Application Servers Cloud Hosted Web Servers Cloud Hosted](https://reader035.vdocuments.mx/reader035/viewer/2022062922/5f08a49d7e708231d42304da/html5/thumbnails/14.jpg)
• As an energy company, we like to measure things: – Staff of HVAC experts with
decades of experience to guide what to measure (air flow, temperature inside and out)
– Add metrics based on customer needs
• As a technology company, we really like to measure things: – Performance engineers,
architects and developers with years of experience to guide what to measure
– Add metrics based on what Ops needs
EnerNOC Data
![Page 15: Splunk:UningOps and*Dev*(before* DevOps*was*cool)* · Cloud Hosted Database Servers EnerNOC Hosted Web Servers EnerNOC Hosted Application Servers Cloud Hosted Web Servers Cloud Hosted](https://reader035.vdocuments.mx/reader035/viewer/2022062922/5f08a49d7e708231d42304da/html5/thumbnails/15.jpg)
EnerNOC EIS PlaQorm
EnerNOC Database Servers
Cloud Hosted Database Servers
EnerNOC Hosted Web
Servers
EnerNOC Hosted
Application Servers
Cloud Hosted Web
Servers
Cloud Hosted
Application Servers
32TB Persisted
4000/1000 Users Daily/Hourly 100 Countries
Data Streaming and Processing at Scale
15
![Page 16: Splunk:UningOps and*Dev*(before* DevOps*was*cool)* · Cloud Hosted Database Servers EnerNOC Hosted Web Servers EnerNOC Hosted Application Servers Cloud Hosted Web Servers Cloud Hosted](https://reader035.vdocuments.mx/reader035/viewer/2022062922/5f08a49d7e708231d42304da/html5/thumbnails/16.jpg)
Before Splunk in Eng
• Cron jobs • Perl scripts to process log files, insert
data • “Huge” mysql databases • Manually send emails with slick Excel
charts and PowerPoint • New metrics could take weeks • Files being copied all over
• Ran out of disk one day…
![Page 17: Splunk:UningOps and*Dev*(before* DevOps*was*cool)* · Cloud Hosted Database Servers EnerNOC Hosted Web Servers EnerNOC Hosted Application Servers Cloud Hosted Web Servers Cloud Hosted](https://reader035.vdocuments.mx/reader035/viewer/2022062922/5f08a49d7e708231d42304da/html5/thumbnails/17.jpg)
Before Splunk in Ops
Tight regula>ons • 2 second data • 2ms response requirements
• 24x7x365 par>cipa>on
Ops tools provide • High level aggregates
• Hourly updates
![Page 18: Splunk:UningOps and*Dev*(before* DevOps*was*cool)* · Cloud Hosted Database Servers EnerNOC Hosted Web Servers EnerNOC Hosted Application Servers Cloud Hosted Web Servers Cloud Hosted](https://reader035.vdocuments.mx/reader035/viewer/2022062922/5f08a49d7e708231d42304da/html5/thumbnails/18.jpg)
Ops aUer Splunk
![Page 19: Splunk:UningOps and*Dev*(before* DevOps*was*cool)* · Cloud Hosted Database Servers EnerNOC Hosted Web Servers EnerNOC Hosted Application Servers Cloud Hosted Web Servers Cloud Hosted](https://reader035.vdocuments.mx/reader035/viewer/2022062922/5f08a49d7e708231d42304da/html5/thumbnails/19.jpg)
Aler>ng 101
What makes a good alert?
Ac>onable Instruc>ve Timely Manageable
![Page 20: Splunk:UningOps and*Dev*(before* DevOps*was*cool)* · Cloud Hosted Database Servers EnerNOC Hosted Web Servers EnerNOC Hosted Application Servers Cloud Hosted Web Servers Cloud Hosted](https://reader035.vdocuments.mx/reader035/viewer/2022062922/5f08a49d7e708231d42304da/html5/thumbnails/20.jpg)
Exact manner of problems is not always known in advance…
Create beker Outlook rules?
![Page 21: Splunk:UningOps and*Dev*(before* DevOps*was*cool)* · Cloud Hosted Database Servers EnerNOC Hosted Web Servers EnerNOC Hosted Application Servers Cloud Hosted Web Servers Cloud Hosted](https://reader035.vdocuments.mx/reader035/viewer/2022062922/5f08a49d7e708231d42304da/html5/thumbnails/21.jpg)
![Page 22: Splunk:UningOps and*Dev*(before* DevOps*was*cool)* · Cloud Hosted Database Servers EnerNOC Hosted Web Servers EnerNOC Hosted Application Servers Cloud Hosted Web Servers Cloud Hosted](https://reader035.vdocuments.mx/reader035/viewer/2022062922/5f08a49d7e708231d42304da/html5/thumbnails/22.jpg)
Dashboards & Alerts
22
![Page 23: Splunk:UningOps and*Dev*(before* DevOps*was*cool)* · Cloud Hosted Database Servers EnerNOC Hosted Web Servers EnerNOC Hosted Application Servers Cloud Hosted Web Servers Cloud Hosted](https://reader035.vdocuments.mx/reader035/viewer/2022062922/5f08a49d7e708231d42304da/html5/thumbnails/23.jpg)
Splunk moves back from Ops to Dev • Development takes off in many direc>ons
– Many microservices – Cloud pladorms being developed in AWS
ê Servers are cakle, not pets
• Need to start selng standardsà Splunk is the standard
23
![Page 24: Splunk:UningOps and*Dev*(before* DevOps*was*cool)* · Cloud Hosted Database Servers EnerNOC Hosted Web Servers EnerNOC Hosted Application Servers Cloud Hosted Web Servers Cloud Hosted](https://reader035.vdocuments.mx/reader035/viewer/2022062922/5f08a49d7e708231d42304da/html5/thumbnails/24.jpg)
Splunk in Dev • What does it mean Splunk is the standard? • Logging format – >mestamps, key value pairs, unique IDs • Learn to log – not alert
– Give info about what an error means, let the stakeholder decide if its cri>cal
24
![Page 25: Splunk:UningOps and*Dev*(before* DevOps*was*cool)* · Cloud Hosted Database Servers EnerNOC Hosted Web Servers EnerNOC Hosted Application Servers Cloud Hosted Web Servers Cloud Hosted](https://reader035.vdocuments.mx/reader035/viewer/2022062922/5f08a49d7e708231d42304da/html5/thumbnails/25.jpg)
Mul>ple Environment Aler>ng • Saw an error in dev, couldn’t reproduce but could be cri>cal – use aler>ng to catch it in produc>on
25
![Page 26: Splunk:UningOps and*Dev*(before* DevOps*was*cool)* · Cloud Hosted Database Servers EnerNOC Hosted Web Servers EnerNOC Hosted Application Servers Cloud Hosted Web Servers Cloud Hosted](https://reader035.vdocuments.mx/reader035/viewer/2022062922/5f08a49d7e708231d42304da/html5/thumbnails/26.jpg)
Mul>ple Environment Monitoring
26
![Page 27: Splunk:UningOps and*Dev*(before* DevOps*was*cool)* · Cloud Hosted Database Servers EnerNOC Hosted Web Servers EnerNOC Hosted Application Servers Cloud Hosted Web Servers Cloud Hosted](https://reader035.vdocuments.mx/reader035/viewer/2022062922/5f08a49d7e708231d42304da/html5/thumbnails/27.jpg)
27
![Page 28: Splunk:UningOps and*Dev*(before* DevOps*was*cool)* · Cloud Hosted Database Servers EnerNOC Hosted Web Servers EnerNOC Hosted Application Servers Cloud Hosted Web Servers Cloud Hosted](https://reader035.vdocuments.mx/reader035/viewer/2022062922/5f08a49d7e708231d42304da/html5/thumbnails/28.jpg)
28
![Page 29: Splunk:UningOps and*Dev*(before* DevOps*was*cool)* · Cloud Hosted Database Servers EnerNOC Hosted Web Servers EnerNOC Hosted Application Servers Cloud Hosted Web Servers Cloud Hosted](https://reader035.vdocuments.mx/reader035/viewer/2022062922/5f08a49d7e708231d42304da/html5/thumbnails/29.jpg)
Performance Metrics from Day 1
29
![Page 30: Splunk:UningOps and*Dev*(before* DevOps*was*cool)* · Cloud Hosted Database Servers EnerNOC Hosted Web Servers EnerNOC Hosted Application Servers Cloud Hosted Web Servers Cloud Hosted](https://reader035.vdocuments.mx/reader035/viewer/2022062922/5f08a49d7e708231d42304da/html5/thumbnails/30.jpg)
30
http://www.bostonglobe.com/business/technology/2015/03/08/enernoc-faces-important-transition/TtJ7ejXmauZUKaHvtsZoxL/story.html
![Page 31: Splunk:UningOps and*Dev*(before* DevOps*was*cool)* · Cloud Hosted Database Servers EnerNOC Hosted Web Servers EnerNOC Hosted Application Servers Cloud Hosted Web Servers Cloud Hosted](https://reader035.vdocuments.mx/reader035/viewer/2022062922/5f08a49d7e708231d42304da/html5/thumbnails/31.jpg)
Eng sees: Ops sees:
![Page 32: Splunk:UningOps and*Dev*(before* DevOps*was*cool)* · Cloud Hosted Database Servers EnerNOC Hosted Web Servers EnerNOC Hosted Application Servers Cloud Hosted Web Servers Cloud Hosted](https://reader035.vdocuments.mx/reader035/viewer/2022062922/5f08a49d7e708231d42304da/html5/thumbnails/32.jpg)
A likle more about me…
![Page 33: Splunk:UningOps and*Dev*(before* DevOps*was*cool)* · Cloud Hosted Database Servers EnerNOC Hosted Web Servers EnerNOC Hosted Application Servers Cloud Hosted Web Servers Cloud Hosted](https://reader035.vdocuments.mx/reader035/viewer/2022062922/5f08a49d7e708231d42304da/html5/thumbnails/33.jpg)
My story • Started in opera>ons, using Splunk as a way to see what was really happening
• Building alerts, giving access we never had before • Became interested in the administra>on, helped design and build out new highly available cluster
• Got Splunk 6 admin cer>fica>on last year
33
![Page 34: Splunk:UningOps and*Dev*(before* DevOps*was*cool)* · Cloud Hosted Database Servers EnerNOC Hosted Web Servers EnerNOC Hosted Application Servers Cloud Hosted Web Servers Cloud Hosted](https://reader035.vdocuments.mx/reader035/viewer/2022062922/5f08a49d7e708231d42304da/html5/thumbnails/34.jpg)
Splunk raises all ships Why is Splunk par>cularly well suited to raising all ships in technical literacy? 1) Accessible – easy query language 2) Real – this is the data, it’s not translated 3) Powerful – once you dig in, it has almost endless possibili>es
34
![Page 35: Splunk:UningOps and*Dev*(before* DevOps*was*cool)* · Cloud Hosted Database Servers EnerNOC Hosted Web Servers EnerNOC Hosted Application Servers Cloud Hosted Web Servers Cloud Hosted](https://reader035.vdocuments.mx/reader035/viewer/2022062922/5f08a49d7e708231d42304da/html5/thumbnails/35.jpg)
Tips and Tricks • Importance of a Splunk knowledge manager – someone who can translate business needs
• Find the pieces of data your team wants to see • Logging is cheap. Outages are not. • Pay aken>on to alert management • Real>me searches – be careful of usage
35
![Page 36: Splunk:UningOps and*Dev*(before* DevOps*was*cool)* · Cloud Hosted Database Servers EnerNOC Hosted Web Servers EnerNOC Hosted Application Servers Cloud Hosted Web Servers Cloud Hosted](https://reader035.vdocuments.mx/reader035/viewer/2022062922/5f08a49d7e708231d42304da/html5/thumbnails/36.jpg)
Tips and Tricks • How many indexes? • RTFM -‐ Read the documenta>on
– Ask ques>ons, come to .conf
• Search head cluster stories – Moving from a standalone search head has some quirks
• Think about administra>on as you go
36
![Page 37: Splunk:UningOps and*Dev*(before* DevOps*was*cool)* · Cloud Hosted Database Servers EnerNOC Hosted Web Servers EnerNOC Hosted Application Servers Cloud Hosted Web Servers Cloud Hosted](https://reader035.vdocuments.mx/reader035/viewer/2022062922/5f08a49d7e708231d42304da/html5/thumbnails/37.jpg)
Summary
37
• Know the data, love the data, share the data • Involve Ops by finding out what they need
-‐ Dashboards give insights to opaque technologies -‐ Good alerts will make data accessible and ac>onable
• Set some standards -‐ Learn to log, let Ops alert -‐ Splunk across environments
• Use Splunk to raise all ships
![Page 38: Splunk:UningOps and*Dev*(before* DevOps*was*cool)* · Cloud Hosted Database Servers EnerNOC Hosted Web Servers EnerNOC Hosted Application Servers Cloud Hosted Web Servers Cloud Hosted](https://reader035.vdocuments.mx/reader035/viewer/2022062922/5f08a49d7e708231d42304da/html5/thumbnails/38.jpg)
Next Steps
• Get in touch – [email protected], [email protected], or LinkedIn
• Boston Splunk Users Group
38
![Page 39: Splunk:UningOps and*Dev*(before* DevOps*was*cool)* · Cloud Hosted Database Servers EnerNOC Hosted Web Servers EnerNOC Hosted Application Servers Cloud Hosted Web Servers Cloud Hosted](https://reader035.vdocuments.mx/reader035/viewer/2022062922/5f08a49d7e708231d42304da/html5/thumbnails/39.jpg)
THANK YOU