Copyright  ©  2015  Splunk  Inc.  

Linus Myrefelt @ Statnett SF

The  Future  is  Electric  

2  

The  future  is  "Electric"  

3  

Important  

What  I  am  going  to  talk  about  does  not  neccessarily  represent  :      •  The  truth  •  Splunk’s  opinions  or  thoughts  •  StatneG’s  opinions  or  thoughts    •  My  own  thoughts  

4  

Agenda  " StatneG?  "   Linux  "   Where  it  all  started  -­‐  "SPll"  troubleshooPng  (/  Root  cause  analyPcs)  "   The  new  driver  -­‐  devOps  /  agile  development  and  rapid  deployment  "   What  we  want  to  do  -­‐  ApplicaPon  management  /  IT  Service  Management  "   What  we  all  do  -­‐  Security  (doh!)  "   The  future  is  electric!  –  Next  step  on  our  journey  "   Take  aways  and  Pps  for  your  journey  and  success!  

5  

StatneG,  what?  "   Make  sure  that  the  lights  are  on  in  Norway  "   State  owned  company  "   Quite  small  (~1500)  "   We  own,  build  and  maintain  the  Norwegian  power-­‐grid  "   Regulates  the  market  "   "The  spider  in  the  web"  

6  

SPll  a  young  company  " Light-­‐weight  company  "   Small  environment  in  server/endpoint  numbers  "   Large  and  complex  network  "   MS  dominated  "   Large  group  of  developers  " Heavily  project  focused  organisaPon  " Heavily  depending  on  IT  " Heavily  regulated  

7  

(linu(s|x))  Background  and  Role  "   Born  and  raised  in  the  smålandian  woods  "   Geek  /  "Hacker"  since  childhood  "   Living  Oslo  /  Norway  (same  same  but  different)  "   Trying  to  speak  Swedish  in  Norway  and  Norwegian  in  Sweden  " Splunker  since  ~version  4  "   Before:  Consultant  doing  APM,  NPM,  Splunk  and  "security"  "   Now:  Building  "Next-­‐Gen"  log  and  monitoring  plagorm  at  StatneG  "   Not  a  "PowerPoint  warrior"    

8  

My  3  (4  including  Splunk  t-­‐shirts)  favorite  things  J  

9  

This  is  what  I  belive  in!  

10  

This  is  what  I  belive  in!  

11  

12  

Let’s  get  down  to  business  –  use  cases  

TroubleshooPng  

Development  

IT  Service  Management  

Security  

13  

Where  it  all  started  -­‐  troubleshooPng  (/  Root  cause  analyPcs)  

14  

Our  iniPal  pain  

How  do  you  troubleshoot  amongst  1000s  of  servers?  What  about  many  1000s  of  network  devices?    What  if  you  have  100s  of  thousands  of  communicaPon  points?    How  do  you  go  about  and  do  just  that?    

15  

Our  SoluPon  to  the  problem  …  started  with  networking  

16  

Enabled  us  to…  " Maintain  our  infrastructure  posture  " Track  faulty  devices  " Earlier  and  controlled  replacement  

" Correlate  events    "   Spot  trends  on  network  " Bigger  picture  with  drilldown  

17  

And  the  SituaPon  now?  "   The  good  guys  use  Splunk  for  root  cause  analy5cs  (tuff  word)  "   The  bad  ones  use  me  or  my  colleague  for  root  cause  analy5cs  (s5ll  a  tuff  word)  

 

18  

The  new  driver  -­‐  devOps  /  Agile  development  and  rapid  deployment  

19  

Our  developers  were  struggling  with:  "   MulPple  Stages  " Across  "zones  /  network  segments"  " Amongst  mulPple  servers  " Use  of  crypPc  tool  with  a  hard  to  get  syntax  –  tail,  grep,awk,sed  mm  " Customized  event  viewer  "   Not  scalable  " Genng  access  to  the  right  data    "   In  a  Pmely  fashion  

20  

SoluPon  -­‐  They  threw  it  into  splunk  J    

And  they  created  a  big  fat  mess  -­‐  …  sPll  like  using  grep  and  awk  for  your  life      

21  

What  we  want  to  do  -­‐  ApplicaPon  management  /  IT  Service  management  

22  

Our  ops  guys  were  struggling  with…  "   SPll  kind  of  of  young  company  " Started  to  mature    "   Old  but  good  siloed  tools  "   Not  very  user-­‐friendly  or  accessible  

" Need  something  more  unifying    " HolisPc  overview  of  services  and  KPIs  " Give  stak´holders  the  right  informaPon  "   Technical  overview  with  drill  downs  into  alerts  and  events  

23  

Our  soluPon  for  Ops  "   Re-­‐designed,  re-­‐architectured  and  scaled  up  soluPon  "   Splunk  agent  deployed    "   Part  of  standard  image  and  rouPnes  "   Different  departments  pushing  for  expansion  

" Need  to  seGle  on  informaPon  model    

24  

APM  /  IT  Service  Management  

25  

Security  –  All  "your"  data  belongs  to  me  

26  

In  Security  we  struggle  with  the  following  things  

"   Too  few  people  …  already  heavily  occupied    "   Not  enough  (good)  people  to  hire  "   No  single  pane  of  overview  "   Hard  to  keep  up  with  todays  threat  "   No  real  "Malware  popup"  

27  

We  want  to  do  more  " Improve  our  security  posture  " Enable  the  right  peope  with  data  "   Do  more  with  less  "   AND  " Being  able  to  keeping  track  of  aGackers  

" Threat  intel  ,  i.e  blacklists  …  =  Noise  " Researching  IP  /  AGackers  is  part  of  the  game  

28  

How  we  are  trying  to  do  it  " UPlizing  Splunk  and  data  as  enabler  " Automate  boring  and  Pme-­‐consuming  tasks  

" We  combine  freely  tools  with  homebrewed  " Scraping  public  api  and  web  services  " Everything  "hosPle"  that  goes  in  and  out  

29  

Security  

Manager  approv

ed    

30  

The  future  is  electric!  –  Next  step  on  our  Journey  

31  

The  future  is  electric!  " ConPnue  to  roll  out  agent  " Collect  applicaPon  logs  " Expanding  use-­‐cases  " Work  hard  on  normalisaPon  "   InformaPon  model  "   Service  modelling  "   More  integraPons  into  splunk  " Keep  adding  reports  and  alerts  

32  

Top  Takeaways  /  My  Tips  "   Invest  in  educaPon  for  (different)  users  "   Use  PS  or  a  trusted  local  partner  "   Before  reaching  maturity  …  maybe  start  small  

33  

Quote  Box  

Our  mission  is  to  make  machine  data  accessible,  useable  and  valuable  to  everyone.  

Thank  You