building business service intelligence with itsi

Copyright©2016SplunkInc.

BuildingBusinessServiceIntelligencewith

SplunkITServiceIntelligence

DavidMillisStaffArchitect,ITOperaDonsAnalyDcs

Tuesday,October4,2016Calgary

WiFi:MarrioH_CONFERENCE/splunk

SetupBeforeYouCanPlay1.Downloadthispresenta9onslidedeck:h;ps://splunk.box.com/v/ITSI-HandsOn-Calgary2.Ifyouhavenotdonesoalready,SignupfortheFREESplunkITSIOnlineSandbox:•  hHp://splunk.com/itsi•  Select"FreeOnlineSandbox"

3.Pleasetestaccesstoyoursandbox;•  Chrome,Firefox,Safari

arerecommended;•  IEisNOTrecommended

4.AYerloggingin,selectITServiceIntelligencefromthelistofappsattheleY

2

3

Agenda•  IntroducDonsandSetUp•  WhatisServiceIntelligence?•  Splundamentals--CoreSplunkinITOps•  WhatisITServiceIntelligence(ITSI)?•  Let'sPlay!•  What'sNext?•  HappyHour!

SafeHarborStatementDuring the course of this presentaDon, wemaymake forward looking statements regarding futureeventsortheexpectedperformanceofthecompany.WecauDonyouthatsuchstatementsreflectourcurrentexpectaDonsandesDmatesbasedonfactorscurrentlyknowntousandthatactualeventsorresultscoulddiffermaterially.Forimportantfactorsthatmaycauseactualresultstodifferfromthosecontainedinourforward-lookingstatements,pleasereviewourfilingswiththeSEC.Theforward-lookingstatementsmadeinthispresentaDonarebeingmadeasoftheDmeanddateofitslivepresentaDon.IfreviewedaYeritslivepresentaDon,thispresentaDonmaynotcontaincurrentoraccurateinformaDon.WedonotassumeanyobligaDontoupdateanyforwardlookingstatementswemaymake.InaddiDon,anyinformaDonaboutourroadmapoutlinesourgeneralproductdirecDonandissubjecttochangeatanyDmewithoutnoDce. It is for informaDonalpurposesonlyandshallnotbe incorporated intoanycontract or other commitment. Splunk undertakes no obligaDon either to develop the features orfuncDonalitydescribedortoincludeanysuchfeatureorfuncDonalityinafuturerelease.

4

WhatisServiceIntelligence?

Enablingabusiness-awareITMeasuringandrepor<ngonindicatorsthatma?er

Unlockingopera9onalefficienciesCollabora<ngacrosssilostoimproveserviceopera<ons

Usingdata-drivendecisionmakingSolvingproblemsandan<cipa<ngpiBallswithsophis<catedanaly<csandpowerfulinsightsfrommachinedata

ServiceModelingThere’saBeHerWay…

Key Takeaways

1 TradiDonalMethodsareoutdated

ServiceMappingisCriDcal

3 Data-DrivenServiceInsights

Challenging Traditional Methods

Network

InfrastructureLayer

HPNNMi,HPNA,SolarWinds,CASpectrum,

StorageHPStorageOperaDons,NetApp,EMC

ServerHPOV/Sitescope,SCOM,Nagios,Tivoli,BMCPatrol,CAUIM

74%

-36%

ApplicaD

onLayer

SyntheDcAPMAppD,NewRelic,Dynatrace,HPAPM,CA,IBM,Appica

ByteCodeInstrumentaDon

AppD,NewRelic,Dynatrace,HPDiag,CAWily

AdapDveThresholding

HPSHA,BMCProacDveNet,NetuiDve,Preelert

HPRun-TimeServiceModelCAServiceOperaDonsInsight

IBMNetCool/OmnibusServiceModeldefiniDon&CorrelaDonEngine

Business Layer

Aggregation/Correlation/Visualization

Service Layer

Challenges•  Toomanydisparatecomponents•  DifficulttodefineServiceModel•  Laborintensive•  MostimplementaDonsfail•  Veryimportantsourceis

missing!(machinedata)

Data Approach With Splunk>

Network

InfrastructureLayer

Packet,Payload,Traffic,UDlizaDon,Perf

StorageUDlizaDon,Capacity,Performance

ServerPerformance,Usage,Dependency

74%

-36%

ApplicaD

onLayer

SyntheDcAPMAvailability,Capacity,UserExperience

ByteCodeInstrumentaDonUsage,Experience,Performance,Quality

AdapDveThresholdingApps,Services,Systems

Splunk>isthemissinglink•  DataFidelity•  SingleRepositoryforALLdata•  EasiertoManageServices•  ReducedIntegraDons•  ReducedPointSoluDons•  CollaboraDveApproach•  QuickDmetovalue

MACHINEDATA

Data Fabric Platform

Service Intelligence

BringSubjectExpertsTogether

DesignBeforeConfiguring

BestPracDcesforServiceModeling

StartWithaProblemWorth

Solving



Solving

•  Whatarethetopbusinessservicesinyourenterprise?

•  Howdoyoumeasurethecustomerexperiencewiththeseservices?

•  Whatisthecustomerexperiencewiththeseservices?

CriDcalServices



Solving

•  HowoYendocustomersexperienceissueswiththeservice?

•  Whenissuesarise,whogetsinvolvedinresolvingthem?

•  Howdoteamsworktogethertoresolveissues?

IssueFrequency



Solving

•  What’stheaverageDmetoissueresoluDon?

•  What’stheimpactwhencustomershaveabadexperiencewithyourservices?

Impact


BringSubjectExpertsTogether

•  Business&ServiceOwners,meetIT•  IT,meetBusiness&ServiceOwners


DesignBeforeConfiguring

• DefineYourServices–TechnicalANDBusiness• DefineYourKPI’sforeachservice• Understandservicedependencies&interacDons

COLLECTDATAFROMANYWHERE

SEARCHANDANALYZE

EVERYTHING

DELIVERREAL-TIMEOPERATIONALINTELLIGENCETOITANDTHEBUSINESS

THEPOWEROF

Splundamentals--CoreSplunkinITOps

Data-drivenserviceinsightsforroot-causeisolaDonandimprovedserviceoperaDons

withamarkeDngcatchphrasethatistoolong

INTRODUCING


SPLUNKITSERVICEINTELLIGENCE

Time-SeriesIndex

PlarormforMachineData

DynamicServiceModels

Schema-on-Read DataModel CommonInforma9onModel

At-a-GlanceProblemAnalysis

EarlyWarningonDeviaDons

SimplifiedIncidentWorkflows

ITSICoreConcepts

21

WhatisaService?

ServiceRequestsResponses

InITSI,aServiceisalogicalgroupoftechnologycomponentsthatauserdeemsneedtobemonitoredtogether.

ItcanoYenbegeneralizedasa“blackbox”whichwesendrequests,andexpectresponses

22

WhatisaService?

DNS RequestsResponses

TechnicalServices

AuthRequestsResponses

WebRequestsResponses

Servicescanbelowerlevel(technical)…

23

WhatisaService?


TechnicalServices

CustomerTransac9ons

RequestsResponses

BusinessServices

AuthRequestsResponses

WebRequestsResponses

SupportDesk RequestsResponses

Servicescanalsobehigherlevel(business)…

24

WhatisaService?

PacketNetwork

HypervisorandHosts

RBMDBs

StorageTier

APIServices

WebServices

CustomerTransac9ons

MobileAPI/

Middlew

are

PartnerPortal

DNS

ServicescanencompassmulDpleDersoftheITdomain.Servicesmayalsodependuponotherservices

25

WhatisaKPI?


KPI:NumberofrequestsKPI:ErrorrateKPI:AverageresponseDmeKPI:ServerCPUloadKPI:ServernetworkI/Ferrors

CustomerTransac9ons

RequestsResponses

KPI:NumberoftransacDonsKPI:ErrorrateKPI:AverageresponseDmeKPI:CountofIncidentTicketsKPI:SyntheDcTransxHealth

KPIsandHealthscoresconsDtutethemeansbywhichServicesaremonitored.

26

KeyPerformanceIndicators(KPIs)

27

AKeyPerformanceIndicator(KPI)isaSplunksavedsearchcreatedwithintheITSIUIthathelpsmonitoraspecificfieldlikeCPU,Memory,NumberofErrors

andsoon.KPIsarecontainedwithinServices.

ServiceHealthScores

28

AHealthscoreisascoreform0-100(0beingcriDcaland100beingnormal)thathelpsdeterminethehealthofaService.ItiscalculatedbasedonallKPIs

importanceanditsstatus(e.g.green,orange,red),onceeveryminute.

ITSITour

29

ServiceDecomposiDoninITSI

30

1-Whatisahigh-valuebusinessservice?(“OnlineStore”inBuHercupGames)


31

1-Whatisahigh-valuebusinessservice?(OnlineStore)

2-Processflow,andunderlyingsub-services?(Web->Middleware->DB->Middleware->Web)


32


2-Processflow,andunderlyingsub-services?(Web->Middleware…)

3-Foreach(sub)service:KPIstoshowhealth&status?(Database:errors,SQLhits,responseDme,…)


33


2-Processflow&underlyingsub-services?(Web->Middleware…)

3-Foreach(sub)service:KPIs?(Database:errors,SQLhits,…)

4-ForeachKPI:NeedaSplunksearch(index=DB(warn*ORerror*)|statscount)


34


2-Processflow&underlyingsub-services?(Web->Middleware…)

3-Foreach(sub)service:KPIs?(Database:errors,SQLhits,…)

4-ForeachKPI:NeedaSplunksearch(index=DB(warn*ORerror*)|statscount)

Let’sPlay!


36

CLICK“GlassTables”


37

CLICK(openinnewtab)“Bu?ercupGamesBusinessProcess(INPROGRESS)”


38

CLICK(openinnewtab)“Bu?ercupGamesOnlineStore”

ServiceDecomp:TheBusinessProcesses

39

ServiceDecomp:End-To-EndProcessFlow

40

NewRequirements!

41

●  CreateanewKPIfortheDBService:●  NetworkUDlizaDon

●  ModifytheExecuDveGlassTableinordertoshowofftheservicesyouslaveover

“WEonlyhaveabout15minTODOWHAT???!!???”Thinkabouthowlongthiswouldtakeyoutoday?

42

ConfiguraDonofDBService

ClickConfigure>ClickServices

Let’sTalkEnDDes

43

●  SelectDBService

●  EnDDesaretherelevantthingswhichsupportthisservice(usuallyhosts)

●  Selecttherightentrieswithfilters,ANDs,ORs●  OriginalEnDtylistcancomefromCMDB,

spreadsheet,Splunksearch,others

AKPIin5minutes?Absolutely!

44

ClickNew–GenericKPI

SelectDataModel●  HostOpera<ngSystem●  Network●  #bytes●  Next

Callit“NetworkU>liza>on”,withyourusernameupfront

KPIsConDnued….

45

SplunkBuildsSearchesforyou–OhYeah,that’shappeningJ

●  SelectYesforSplitby&Filterop<ons●  SelecthostforEn>tyLookup&Aliasop<ons●  ClickNext

AlmostThere…

46

Select●  KPISearchSchedule:EveryMinute●  En>tyCalcula>on:Average●  Service/AggCalcula>on:Average●  Calcula>onWindow:LastMinute●  ClickNext

●  Unit:Bps●  ClickNext

FinalSteps…

47

Setyourthresholds:●  Aggregate(All)●  PerEn>ty

●  Click“AddThreshold”TWICE●  MaketheNeapolitanicecreamcolors

Yellow,Green,Yellow●  Dragtheslidersaroundinordertoget

thecurrentdatagraphenDrelyinsidetheGreen(normal)band

●  ClickFinish●  OtheropDonsarealsoavailable,

includingadapDvethresholdsandanomalydetecDon

AdapDveThresholds

48

WhatifyourKPIdatalookslikethis?

49

AdapDveThresholdsStaDcthresholdswillnotwork…

50

AdapDveThresholdsAdapDveThresholdingworksbeauDfullywithcyclical(andotherdynamic)data

AnomalyDetecDon

51

●  MachineLearning

●  WorkswellfordatawithpaHerns

●  Requiressome“training”(trial&error)tozeroinonbestsensiDvity

●  MoresophisDcatedcapabiliDescoming!(mulDvariate,morealgorithms,etc)

Let’sFixthatGlassTable

52

ClonetheGlassTable

53

ReturntoSavedGlassTablespage(clickonGlassTablesintheuppermenubar)CLICKEditfor“BuHercupGamesBusinessProcess(INPROGRESS)”•  SelectClone•  Title:Addyourusername

tothefront•  Permissions:SharedinApp•  ClickClonePage•  ClickonyournewGlassTable

fromthelist,toviewit

Edit&HaveFun!

54

ClickonEditintheupperrightcornerofyourGlassTableUsethe“Services”panelontheleYtoselectIndividualKPIs,orAggregateServiceHealthScores•  Choose2KPIsfromOnlineStorethatwouldbeusefulin

the“OrderProcess”secDon•  Dragtheselectedwidgetsontothecanvas,posiDoningin

thegrayoval•  What’sthedifferencebetweenthe

andtoolsatthetopleY?

MoreFunwiththeGlassTableEditor…

55

UsetheConfigura>onspanelontherighttoeditaselectedwidget•  CanchangethevisualizaDontype,drilldown

behavior,andotherse}ngs•  YoushouldhitSavefrequently•  RevertAllChangescanbehelpful,occasionally

Finishingup…

56

•  AddaServiceHealthScorewidgetforOnlineStoreunderBuHercup

•  ChooseaVizTypewithasparklinegraph,thenresizetomakeitlookpreHy

•  ModifytheCustomDrilldownacDontogotothesavedglasstable,BuVercupGamesOnlineStore

•  BonusPoints:Makethelabelbigger,morereadable

•  ClickSave•  Viewwhendone

ATroubleshooDngExercise

57

Let’suseITSItotroubleshootanoutage●  StartatyourGlassTable,“<UserName>BuHercupBusinessProcess”●  CustomerCarereportsthatunhappycustomersarecomplainingoffailures

andlongdelayswhentryingtopurchase●  Thecallsbegancominginataroundthetopofthelasthour.●  IntheupperrightcorneroftheGlassTable,changetheDmepickerfromNow

toXX:00:00.0,whereXXistheprevioushour.Forexample,ifitiscurrently14:05,settheDmepickerto13:00:00.0,thenApply

●  Thisishowwecan“Dmetravel”backtoseecondiDonsataparDcularoutage–ohyeah!

ATroubleshooDngExercise,cont’d

58

●  TheOnlineStoreseemstobedegraded,justasCustomerCarereported.ClickonthewidgetunderBuHercuptodrilldownfurther

ATroubleshooDngExercise,cont’d.

59

●  TheOnlineStoreGlassTableshowsamuchmoredetailedview,includingtheimpactedcustomer-facingKPIsatthefarleY(Revenue,etc)

●  Basedonthisviewofalltherelevantservices,wheredoyouthinktherootcauselies?

●  Whichserviceshouldwetroubleshootfirst?●  ClickonHealthwidgetforthatservice,to

drilldowntoaDeepDive

DeepDive

60

●  DeepDiveshowsmulDpleKPIsandHealthScoresinparallel“swimlanes”.

●  TheHealthScoreforthisServiceisthetopswimlane.Canyouseewhenitbeginstodegradefrom100%?

●  MousingoverthispointinDme,canyouspottheKPIwiththeleadingfaultindicaDon,i.e.,whatfailedfirst?

●  Toimprovereadability,makesurethePrimaryTimeRange(lowerleYcorner)issettoPresets>Last60minutes

MulD-KPIAlertsandNotableEvents

61

●  ClickonNotableEventsReview●  MulDpleKPIsandHealthscorescan

becombinedinsophisDcatedwaystocreateMulD-KPIalerts

●  WhenaMulD-KPIalertfires,oneoftheoutcomesisthecreaDonofaNotableEvent

●  NotableEventsallowNOCpersonnelandotherstotriageandcoordinateeventmanagementefforts

ServiceAnalyzer

62

●  ClickonServiceAnalyzer>DefaultServiceAnalyzer

●  Backwherewestarted!●  Thisviewshowsa“no-frills”listof

services(top)andhoHestKPIs(boHom)

●  ProvidesaccessintoServiceDetails●  ItisusefulforNOCsandothers

whoneedahigh-levelsituaDonalview

AdvancedExercises

63

Summary

64

●  High-valueservicescanbedecomposedandmodeledinITSI,usingmachinedatafromtherelevantsystems

●  ServicesandKPIscanbecreatedinminutes,withsophisDcatedthresholdingtechniquestodisDnguish“normal”from“notnormal”

●  GlassTablesallowservicehealthandKPImetricstobedisplayedinawaythatmakessensetospecificgroups,suchasExecuDveLeadership,BusinessServiceOwners,theNOC,DevOps&Others

●  DeepDivesallowKPIstobecomparedside-by-sideacrossanyDmerange,acceleraDngrootcauseanalysisandsignificantlyreducingMTTR

●  Mul9-KPIAlertsandNotableEventsreducealertnoise,producingacDonableeventsandameanstomanagethem

●  …andit’sfuntobuild!

Possibili9es

Order Analysis – Core Splunk Record Systems

Payment Processing

Patient Queue Analysis

Patient Processing Service Health Transactions

Co-Pay

Insurance

MHR

Check-In Procedures

Pharmacy

Assessment

Call Center Service

Service Health Transactions

ACD Analysis – Core Splunk Call Wait History

Inbound Analysis

Social Media

Online Msg

Social Media

Mail Support VOIP Service

Inbound Calls

Banking

68

69

HealthcareOperations

70

End User Experience for Streaming Video

Whatever This Is

71

Hospital Emergency Rooms

72

Sign Up Now – We’re here to help!

Harness the creativity and domain knowledge of your organization to unlock the value of data and solve an important Business Service problem through a joint service intelligence

workshop with key stakeholders

Define methods for: ›  Proactive service monitoring ›  Reduced risk and failures ›  Faster issue resolution ›  Increased business performance

What is it? ›  1 Day Onsite Workshop ›  Tightly linked with value ›  Collaborative approach ›  Build your own Glass Table

ReferenceStuff

74

●  ITSIGuidebook:InyourITSIinstance:Search->Dashboards->ITSISandboxGuide

●  ITSIDocumentaDon:hHp://docs.splunk.com/DocumentaDon/ITSI

ThankYou

[email protected]

building business service intelligence with itsi

Technology