building*a*crystal*ball:* - splunkconf · agenda theproblem* exisngtools finding*a*bezer*way*...
TRANSCRIPT
![Page 1: Building*A*Crystal*Ball:* - SplunkConf · Agenda TheProblem* ExisngTools Finding*A*BeZer*Way* Implementaon* Results* Caveats* Quesons * 4](https://reader033.vdocuments.mx/reader033/viewer/2022042005/5e6f388043f4e536bc148d56/html5/thumbnails/1.jpg)
Copyright © 2016 Splunk Inc.
Building A Crystal Ball: Forecas@ng Future Values For Mul@-‐cyclic Time Series Metrics In Splunk
Mike Fisher
![Page 2: Building*A*Crystal*Ball:* - SplunkConf · Agenda TheProblem* ExisngTools Finding*A*BeZer*Way* Implementaon* Results* Caveats* Quesons * 4](https://reader033.vdocuments.mx/reader033/viewer/2022042005/5e6f388043f4e536bc148d56/html5/thumbnails/2.jpg)
Disclaimer
2
During the course of this presenta@on, we may make forward looking statements regarding future events or the expected performance of the company. We cau@on you that such statements reflect our current expecta@ons and es@mates based on factors currently known to us and that actual events or results could differ materially. For important factors that may cause actual results to differ from those contained in our forward-‐looking statements, please review our filings with the SEC. The forward-‐looking statements made in the this presenta@on are being made as of the @me and date of its live presenta@on. If reviewed aRer its live presenta@on, this presenta@on may not contain current or
accurate informa@on. We do not assume any obliga@on to update any forward looking statements we may make. In addi@on, any informa@on about our roadmap outlines our general product direc@on and is
subject to change at any @me without no@ce. It is for informa@onal purposes only and shall not, be incorporated into any contract or other commitment. Splunk undertakes no obliga@on either to develop the features or func@onality described or to include any such feature or func@onality in a future release.
![Page 3: Building*A*Crystal*Ball:* - SplunkConf · Agenda TheProblem* ExisngTools Finding*A*BeZer*Way* Implementaon* Results* Caveats* Quesons * 4](https://reader033.vdocuments.mx/reader033/viewer/2022042005/5e6f388043f4e536bc148d56/html5/thumbnails/3.jpg)
About Me
3
• Splunk user/administrator for 7 years
• Work for a Fortune 100 financial firm
• Currently leading a Monitoring and Opera@onal Intelligence team
• I was not a Sta@s@cs major in college!
![Page 4: Building*A*Crystal*Ball:* - SplunkConf · Agenda TheProblem* ExisngTools Finding*A*BeZer*Way* Implementaon* Results* Caveats* Quesons * 4](https://reader033.vdocuments.mx/reader033/viewer/2022042005/5e6f388043f4e536bc148d56/html5/thumbnails/4.jpg)
Agenda
The Problem Exis@ng Tools Finding A BeZer Way Implementa@on Results Caveats Ques@ons
4
![Page 5: Building*A*Crystal*Ball:* - SplunkConf · Agenda TheProblem* ExisngTools Finding*A*BeZer*Way* Implementaon* Results* Caveats* Quesons * 4](https://reader033.vdocuments.mx/reader033/viewer/2022042005/5e6f388043f4e536bc148d56/html5/thumbnails/5.jpg)
The Problem
![Page 6: Building*A*Crystal*Ball:* - SplunkConf · Agenda TheProblem* ExisngTools Finding*A*BeZer*Way* Implementaon* Results* Caveats* Quesons * 4](https://reader033.vdocuments.mx/reader033/viewer/2022042005/5e6f388043f4e536bc148d56/html5/thumbnails/6.jpg)
Many Time Series Contain Cyclic PaZerns
6
Sales per hour
Web page hits
Network traffic
![Page 7: Building*A*Crystal*Ball:* - SplunkConf · Agenda TheProblem* ExisngTools Finding*A*BeZer*Way* Implementaon* Results* Caveats* Quesons * 4](https://reader033.vdocuments.mx/reader033/viewer/2022042005/5e6f388043f4e536bc148d56/html5/thumbnails/7.jpg)
Some Have Mul@ple Concurrent Cycles
7
Daily cycle
Weekly cycle
![Page 8: Building*A*Crystal*Ball:* - SplunkConf · Agenda TheProblem* ExisngTools Finding*A*BeZer*Way* Implementaon* Results* Caveats* Quesons * 4](https://reader033.vdocuments.mx/reader033/viewer/2022042005/5e6f388043f4e536bc148d56/html5/thumbnails/8.jpg)
How Do We Know What’s Normal?
Sales per minute -‐ Are sales abnormally low right now?
Web page hits -‐ Is my web site experiencing high traffic?
Network traffic -‐ Is that spike in network traffic expected?
8
![Page 9: Building*A*Crystal*Ball:* - SplunkConf · Agenda TheProblem* ExisngTools Finding*A*BeZer*Way* Implementaon* Results* Caveats* Quesons * 4](https://reader033.vdocuments.mx/reader033/viewer/2022042005/5e6f388043f4e536bc148d56/html5/thumbnails/9.jpg)
How Do We Set Alert Thresholds?
9
Here vs Here
![Page 10: Building*A*Crystal*Ball:* - SplunkConf · Agenda TheProblem* ExisngTools Finding*A*BeZer*Way* Implementaon* Results* Caveats* Quesons * 4](https://reader033.vdocuments.mx/reader033/viewer/2022042005/5e6f388043f4e536bc148d56/html5/thumbnails/10.jpg)
How Do We Alert?
10
… if we don’t know what’s normal at any given @me?
???
![Page 11: Building*A*Crystal*Ball:* - SplunkConf · Agenda TheProblem* ExisngTools Finding*A*BeZer*Way* Implementaon* Results* Caveats* Quesons * 4](https://reader033.vdocuments.mx/reader033/viewer/2022042005/5e6f388043f4e536bc148d56/html5/thumbnails/11.jpg)
Exis@ng Tools
![Page 12: Building*A*Crystal*Ball:* - SplunkConf · Agenda TheProblem* ExisngTools Finding*A*BeZer*Way* Implementaon* Results* Caveats* Quesons * 4](https://reader033.vdocuments.mx/reader033/viewer/2022042005/5e6f388043f4e536bc148d56/html5/thumbnails/12.jpg)
Splunk’s predict() Command
The predict command forecasts values for one or more sets of @me-‐series data.
Two algorithms that deal with seasonal data:
LLP – Seasonal local level
LLP5 -‐ Combines local level trend and seasonal local level
12
![Page 13: Building*A*Crystal*Ball:* - SplunkConf · Agenda TheProblem* ExisngTools Finding*A*BeZer*Way* Implementaon* Results* Caveats* Quesons * 4](https://reader033.vdocuments.mx/reader033/viewer/2022042005/5e6f388043f4e536bc148d56/html5/thumbnails/13.jpg)
Al’s Online Toy Barn Sales
13
index=summary search_name= "Sales -‐ Summary -‐ 10 min count" | @mechart span=10m sum(count) as actual | predict actual as pred algorithm=LLP upper90=high lower90=low future_@mespan=432
![Page 14: Building*A*Crystal*Ball:* - SplunkConf · Agenda TheProblem* ExisngTools Finding*A*BeZer*Way* Implementaon* Results* Caveats* Quesons * 4](https://reader033.vdocuments.mx/reader033/viewer/2022042005/5e6f388043f4e536bc148d56/html5/thumbnails/14.jpg)
Forecast Using LPP 5 weeks of data, 3 days of forecast, 90% confidence intervals
14
![Page 15: Building*A*Crystal*Ball:* - SplunkConf · Agenda TheProblem* ExisngTools Finding*A*BeZer*Way* Implementaon* Results* Caveats* Quesons * 4](https://reader033.vdocuments.mx/reader033/viewer/2022042005/5e6f388043f4e536bc148d56/html5/thumbnails/15.jpg)
LLP Forecast vs Reality
15
![Page 16: Building*A*Crystal*Ball:* - SplunkConf · Agenda TheProblem* ExisngTools Finding*A*BeZer*Way* Implementaon* Results* Caveats* Quesons * 4](https://reader033.vdocuments.mx/reader033/viewer/2022042005/5e6f388043f4e536bc148d56/html5/thumbnails/16.jpg)
Forecast Using LPP5 5 weeks of data, 3 days of forecast, 90% confidence intervals
16
![Page 17: Building*A*Crystal*Ball:* - SplunkConf · Agenda TheProblem* ExisngTools Finding*A*BeZer*Way* Implementaon* Results* Caveats* Quesons * 4](https://reader033.vdocuments.mx/reader033/viewer/2022042005/5e6f388043f4e536bc148d56/html5/thumbnails/17.jpg)
LLP Forecast vs Reality
17
![Page 18: Building*A*Crystal*Ball:* - SplunkConf · Agenda TheProblem* ExisngTools Finding*A*BeZer*Way* Implementaon* Results* Caveats* Quesons * 4](https://reader033.vdocuments.mx/reader033/viewer/2022042005/5e6f388043f4e536bc148d56/html5/thumbnails/18.jpg)
The Future Is Fuzzy…
18
![Page 19: Building*A*Crystal*Ball:* - SplunkConf · Agenda TheProblem* ExisngTools Finding*A*BeZer*Way* Implementaon* Results* Caveats* Quesons * 4](https://reader033.vdocuments.mx/reader033/viewer/2022042005/5e6f388043f4e536bc148d56/html5/thumbnails/19.jpg)
Finding A BeZer Way
![Page 20: Building*A*Crystal*Ball:* - SplunkConf · Agenda TheProblem* ExisngTools Finding*A*BeZer*Way* Implementaon* Results* Caveats* Quesons * 4](https://reader033.vdocuments.mx/reader033/viewer/2022042005/5e6f388043f4e536bc148d56/html5/thumbnails/20.jpg)
Requirements
Handle mul@-‐cyclic @me series Fast
Efficient Accurate Reusable
20
![Page 21: Building*A*Crystal*Ball:* - SplunkConf · Agenda TheProblem* ExisngTools Finding*A*BeZer*Way* Implementaon* Results* Caveats* Quesons * 4](https://reader033.vdocuments.mx/reader033/viewer/2022042005/5e6f388043f4e536bc148d56/html5/thumbnails/21.jpg)
Predict The Future
21
…without hiring this guy
![Page 22: Building*A*Crystal*Ball:* - SplunkConf · Agenda TheProblem* ExisngTools Finding*A*BeZer*Way* Implementaon* Results* Caveats* Quesons * 4](https://reader033.vdocuments.mx/reader033/viewer/2022042005/5e6f388043f4e536bc148d56/html5/thumbnails/22.jpg)
The Data index=summary search_name="Sales -‐ Summary -‐ 10 min count" | @mechart span=1h sum(count) as actual
![Page 23: Building*A*Crystal*Ball:* - SplunkConf · Agenda TheProblem* ExisngTools Finding*A*BeZer*Way* Implementaon* Results* Caveats* Quesons * 4](https://reader033.vdocuments.mx/reader033/viewer/2022042005/5e6f388043f4e536bc148d56/html5/thumbnails/23.jpg)
Al’s Online Toy Barn Sales
![Page 24: Building*A*Crystal*Ball:* - SplunkConf · Agenda TheProblem* ExisngTools Finding*A*BeZer*Way* Implementaon* Results* Caveats* Quesons * 4](https://reader033.vdocuments.mx/reader033/viewer/2022042005/5e6f388043f4e536bc148d56/html5/thumbnails/24.jpg)
Week-‐over-‐week View
![Page 25: Building*A*Crystal*Ball:* - SplunkConf · Agenda TheProblem* ExisngTools Finding*A*BeZer*Way* Implementaon* Results* Caveats* Quesons * 4](https://reader033.vdocuments.mx/reader033/viewer/2022042005/5e6f388043f4e536bc148d56/html5/thumbnails/25.jpg)
Week Over Week 10 Minute Resolu@on
25
![Page 26: Building*A*Crystal*Ball:* - SplunkConf · Agenda TheProblem* ExisngTools Finding*A*BeZer*Way* Implementaon* Results* Caveats* Quesons * 4](https://reader033.vdocuments.mx/reader033/viewer/2022042005/5e6f388043f4e536bc148d56/html5/thumbnails/26.jpg)
Mul@-‐Series View
![Page 27: Building*A*Crystal*Ball:* - SplunkConf · Agenda TheProblem* ExisngTools Finding*A*BeZer*Way* Implementaon* Results* Caveats* Quesons * 4](https://reader033.vdocuments.mx/reader033/viewer/2022042005/5e6f388043f4e536bc148d56/html5/thumbnails/27.jpg)
Take A Slice of Time 11 AM – Noon
![Page 28: Building*A*Crystal*Ball:* - SplunkConf · Agenda TheProblem* ExisngTools Finding*A*BeZer*Way* Implementaon* Results* Caveats* Quesons * 4](https://reader033.vdocuments.mx/reader033/viewer/2022042005/5e6f388043f4e536bc148d56/html5/thumbnails/28.jpg)
The Slice in Numbers 11:00 11:10 11:20 11:30 11:40 11:50 12:00
04/20/16 374 327 313 337 330 331 437
04/13/16 304 295 291 300 318 317 358
04/06/16 311 300 301 323 325 331 376
03/30/16 319 323 328 339 353 357 395
03/23/16 312 318 319 329 335 355 394
![Page 29: Building*A*Crystal*Ball:* - SplunkConf · Agenda TheProblem* ExisngTools Finding*A*BeZer*Way* Implementaon* Results* Caveats* Quesons * 4](https://reader033.vdocuments.mx/reader033/viewer/2022042005/5e6f388043f4e536bc148d56/html5/thumbnails/29.jpg)
The Target 11:00 11:10 11:20 11:30 11:40 11:50 12:00
04/27/16 ???
04/20/16 374 327 313 337 330 331 437
04/13/16 304 295 291 300 318 317 358
04/06/16 311 300 301 323 325 331 376
03/30/16 319 323 328 339 353 357 395
03/23/16 312 318 319 329 335 355 394
![Page 30: Building*A*Crystal*Ball:* - SplunkConf · Agenda TheProblem* ExisngTools Finding*A*BeZer*Way* Implementaon* Results* Caveats* Quesons * 4](https://reader033.vdocuments.mx/reader033/viewer/2022042005/5e6f388043f4e536bc148d56/html5/thumbnails/30.jpg)
Average 11:00 11:10 11:20 11:30 11:40 11:50 12:00
04/27/16 ???
04/20/16 374 327 313 337 330 331 437
04/13/16 304 295 291 300 318 317 358
04/06/16 311 300 301 323 325 331 376
03/30/16 319 323 328 339 353 357 395
03/23/16 312 318 319 329 335 355 394
Average = 333.57 Standard Devia@on = 31.66
![Page 31: Building*A*Crystal*Ball:* - SplunkConf · Agenda TheProblem* ExisngTools Finding*A*BeZer*Way* Implementaon* Results* Caveats* Quesons * 4](https://reader033.vdocuments.mx/reader033/viewer/2022042005/5e6f388043f4e536bc148d56/html5/thumbnails/31.jpg)
High and Low Bounds
𝑝𝑟𝑒𝑑𝑖𝑐𝑡𝑖𝑜𝑛=𝑎𝑣𝑒𝑟𝑎𝑔𝑒
𝑏𝑜𝑢𝑛𝑑𝑠=𝑝𝑟𝑒𝑑𝑖𝑐𝑡𝑖𝑜𝑛±𝑠𝑡𝑑𝑒𝑣 ∗√1/1− 𝑐𝑜𝑛𝑓𝑖𝑑𝑒𝑛𝑐𝑒/100
![Page 32: Building*A*Crystal*Ball:* - SplunkConf · Agenda TheProblem* ExisngTools Finding*A*BeZer*Way* Implementaon* Results* Caveats* Quesons * 4](https://reader033.vdocuments.mx/reader033/viewer/2022042005/5e6f388043f4e536bc148d56/html5/thumbnails/32.jpg)
High and Low Bounds
Average = 333.57 Standard devia@on = 31.65
predicted = average
bounds = predicted +/-‐ stdev * (sqrt(1/(1-‐confidence/100)))
low = 333.57 – 31.65 * (sqrt(1/(1-‐90/100))) = 233.46 high = 333.57 + 31.65 * (sqrt(1/(1-‐90/100))) = 433.68
![Page 33: Building*A*Crystal*Ball:* - SplunkConf · Agenda TheProblem* ExisngTools Finding*A*BeZer*Way* Implementaon* Results* Caveats* Quesons * 4](https://reader033.vdocuments.mx/reader033/viewer/2022042005/5e6f388043f4e536bc148d56/html5/thumbnails/33.jpg)
How’d We Do?
33
Predicted = 333.57 Low bound = 233.46 High bound = 433.68
Apr 27 11:30 actual = 318
![Page 34: Building*A*Crystal*Ball:* - SplunkConf · Agenda TheProblem* ExisngTools Finding*A*BeZer*Way* Implementaon* Results* Caveats* Quesons * 4](https://reader033.vdocuments.mx/reader033/viewer/2022042005/5e6f388043f4e536bc148d56/html5/thumbnails/34.jpg)
Implementa@on
![Page 35: Building*A*Crystal*Ball:* - SplunkConf · Agenda TheProblem* ExisngTools Finding*A*BeZer*Way* Implementaon* Results* Caveats* Quesons * 4](https://reader033.vdocuments.mx/reader033/viewer/2022042005/5e6f388043f4e536bc148d56/html5/thumbnails/35.jpg)
So….. How Do We Do That In Splunk?
35
Simple.
Just build a macro.
![Page 36: Building*A*Crystal*Ball:* - SplunkConf · Agenda TheProblem* ExisngTools Finding*A*BeZer*Way* Implementaon* Results* Caveats* Quesons * 4](https://reader033.vdocuments.mx/reader033/viewer/2022042005/5e6f388043f4e536bc148d56/html5/thumbnails/36.jpg)
forecast5w(val,confidence,rel@me,days) eval w=case( (_@me>rela@ve_@me(now(), "$rel@me$@d-‐5w-‐30m") AND _@me<=rela@ve_@me(now(), "$rel@me$@d-‐5w+$days$d+30m")), 5, (_@me>rela@ve_@me(now(), "$rel@me$@d-‐4w-‐30m") AND _@me<=rela@ve_@me(now(), "$rel@me$@d-‐4w+$days$d+30m")), 4, (_@me>rela@ve_@me(now(), "$rel@me$@d-‐3w-‐30m") AND _@me<=rela@ve_@me(now(), "$rel@me$@d-‐3w+$days$d+30m")), 3, (_@me>rela@ve_@me(now(), "$rel@me$@d-‐2w-‐30m") AND _@me<=rela@ve_@me(now(), "$rel@me$@d-‐2w+$days$d+30m")), 2, (_@me>rela@ve_@me(now(), "$rel@me$@d-‐1w-‐30m") AND _@me<=rela@ve_@me(now(), "$rel@me$@d-‐1w+$days$d+30m")), 1 ) | eval shiR=case(isnotnull(w), "+"+w+"w-‐30m +"+w+"w-‐20m +"+w+"w-‐10m +"+w+"w-‐0m +"+w+"w+10m +"+w+"w+20m +"+w+"w+30m") | where isnotnull(shiR) | makemv shiR | mvexpand shiR | eval @me=rela@ve_@me(_@me, shiR) | eventstats avg($val$) as pred by @me | eval upper=if($val$>pred,$val$,pred) | eval lower=if($val$<pred,$val$,pred) | stats avg($val$) as pred, stdev(upper) as ustdev, stdev(lower) as lstdev by @me | eval low=pred-‐lstdev*(sqrt(1/(1-‐$confidence$/100))) | eval low=if(low<0, 0, low) | eval high=pred+ustdev*(sqrt(1/(1-‐$confidence$/100))) | eval _@me=@me | @mechart span=10m min(pred) as pred, min(low) as low, min(high) as high | where _@me>rela@ve_@me(now(), "$rel@me$@d") AND _@me<=rela@ve_@me(now(), "$rel@me$+$days$d@d")
36
![Page 37: Building*A*Crystal*Ball:* - SplunkConf · Agenda TheProblem* ExisngTools Finding*A*BeZer*Way* Implementaon* Results* Caveats* Quesons * 4](https://reader033.vdocuments.mx/reader033/viewer/2022042005/5e6f388043f4e536bc148d56/html5/thumbnails/37.jpg)
Any Ques@ons?
37
![Page 38: Building*A*Crystal*Ball:* - SplunkConf · Agenda TheProblem* ExisngTools Finding*A*BeZer*Way* Implementaon* Results* Caveats* Quesons * 4](https://reader033.vdocuments.mx/reader033/viewer/2022042005/5e6f388043f4e536bc148d56/html5/thumbnails/38.jpg)
How Do We Do That In Splunk, Really?
38
Short answer:
Time travel and cloning
![Page 39: Building*A*Crystal*Ball:* - SplunkConf · Agenda TheProblem* ExisngTools Finding*A*BeZer*Way* Implementaon* Results* Caveats* Quesons * 4](https://reader033.vdocuments.mx/reader033/viewer/2022042005/5e6f388043f4e536bc148d56/html5/thumbnails/39.jpg)
Time Travel
ShiR data points from prior weeks forward in @me to where they are needed.
Target T-‐1w T-‐2w T-‐3w T-‐4w T-‐5w
39
![Page 40: Building*A*Crystal*Ball:* - SplunkConf · Agenda TheProblem* ExisngTools Finding*A*BeZer*Way* Implementaon* Results* Caveats* Quesons * 4](https://reader033.vdocuments.mx/reader033/viewer/2022042005/5e6f388043f4e536bc148d56/html5/thumbnails/40.jpg)
10:40 10:50 11:00 11:10 11:20 11:30 11:40
Apr27 337
Cloning
10:30 10:40 10:50 11:00 11:10 11:20 11:30
Apr27 337
Each data point will be used seven @mes as the forecast window slides by.
10:50 11:00 11:10 11:20 11:30 11:40 11:50
Apr27 337
11:00 11:10 11:20 11:30 11:40 11:50 12:00
Apr27 337
11:10 11:20 11:30 11:40 11:50 12:00 12:10
Apr27 337
11:20 11:30 11:40 11:50 12:00 12:10 12:20
Apr27 337
10:30 10:40 10:50 11:00 11:10 11:20 11:30 11:40 11:50 12:00 12:10 12:20 12:30
Apr20 337
Forecast window
11:30 11:40 11:50 12:00 12:10 12:20 12:30
Apr27 337
40
![Page 41: Building*A*Crystal*Ball:* - SplunkConf · Agenda TheProblem* ExisngTools Finding*A*BeZer*Way* Implementaon* Results* Caveats* Quesons * 4](https://reader033.vdocuments.mx/reader033/viewer/2022042005/5e6f388043f4e536bc148d56/html5/thumbnails/41.jpg)
Cloning
10:30 10:40 10:50 11:00 11:10 11:20 11:30
Apr27 337
Duplicate each data point so it can be used to calculate seven different forecast points.
Apr 27, 11:30 3131
10:40 10:50 11:00 11:10 11:20 11:30 11:40
Apr27 337
10:50 11:00 11:10 11:20 11:30 11:40 11:50
Apr27 337
11:00 11:10 11:20 11:30 11:40 11:50 12:00
Apr27 337
11:10 11:20 11:30 11:40 11:50 12:00 12:10
Apr27 337
11:20 11:30 11:40 11:50 12:00 12:10 12:20
Apr27 337
11:30 11:40 11:50 12:00 12:10 12:20 12:30
Apr27 337
Apr 20
11:30 337
41
![Page 42: Building*A*Crystal*Ball:* - SplunkConf · Agenda TheProblem* ExisngTools Finding*A*BeZer*Way* Implementaon* Results* Caveats* Quesons * 4](https://reader033.vdocuments.mx/reader033/viewer/2022042005/5e6f388043f4e536bc148d56/html5/thumbnails/42.jpg)
But How Do We Do That In Splunk?
Use rela@ve_@me() to calculate the @me shiRs.
Use makemv and mvexpand to duplicate data.
42
![Page 43: Building*A*Crystal*Ball:* - SplunkConf · Agenda TheProblem* ExisngTools Finding*A*BeZer*Way* Implementaon* Results* Caveats* Quesons * 4](https://reader033.vdocuments.mx/reader033/viewer/2022042005/5e6f388043f4e536bc148d56/html5/thumbnails/43.jpg)
Take Timechart Output As Our Input
index=summary search_name="Sales -‐ Summary -‐ 10 min count" | @mechart span=10m sum(count) as actual | `forecast5w(actual,90,+1d,1)`
43
![Page 44: Building*A*Crystal*Ball:* - SplunkConf · Agenda TheProblem* ExisngTools Finding*A*BeZer*Way* Implementaon* Results* Caveats* Quesons * 4](https://reader033.vdocuments.mx/reader033/viewer/2022042005/5e6f388043f4e536bc148d56/html5/thumbnails/44.jpg)
Arguments To The Macro $val$ -‐ The name of the field to forecast $confidence$ -‐ A number, 0 < N < 100, that
determines the width of the bounds $rel@me$ -‐ Start @me of the forecast rela@ve to
current @me $days$ -‐ How many days to forecast
44
![Page 45: Building*A*Crystal*Ball:* - SplunkConf · Agenda TheProblem* ExisngTools Finding*A*BeZer*Way* Implementaon* Results* Caveats* Quesons * 4](https://reader033.vdocuments.mx/reader033/viewer/2022042005/5e6f388043f4e536bc148d56/html5/thumbnails/45.jpg)
Only ShiR The Data We Need
Example, for five weeks ago:
_@me >rela@ve_@me(now(), "$rel@me$@d-‐5w-‐30m") AND
_@me <= rela@ve_@me(now(), "$rel@me$@d-‐5w+$days$d+30m")
45
![Page 46: Building*A*Crystal*Ball:* - SplunkConf · Agenda TheProblem* ExisngTools Finding*A*BeZer*Way* Implementaon* Results* Caveats* Quesons * 4](https://reader033.vdocuments.mx/reader033/viewer/2022042005/5e6f388043f4e536bc148d56/html5/thumbnails/46.jpg)
For each week of data: Compute shiRs needed to move the data to seven loca@ons needed for the forecast.
Compu@ng the Time Jump For example: to shiR from five weeks ago to the target week
$rel@me$+5w-‐30m, $rel@me$+5w-‐20m, $rel@me$+5w-‐10m, $rel@me$+5w-‐0m, $rel@me$+5w+10m, $rel@me$+5w+20m, $rel@me$+5w+30m
46
![Page 47: Building*A*Crystal*Ball:* - SplunkConf · Agenda TheProblem* ExisngTools Finding*A*BeZer*Way* Implementaon* Results* Caveats* Quesons * 4](https://reader033.vdocuments.mx/reader033/viewer/2022042005/5e6f388043f4e536bc148d56/html5/thumbnails/47.jpg)
The Full ShiR
eval w=case( (_@me>rela@ve_@me(now(), "$rel@me$@d-‐5w-‐30m") AND _@me<=rela@ve_@me(now(), "$rel@me$@d-‐5w+$days$d+30m")), 5, (_@me>rela@ve_@me(now(), "$rel@me$@d-‐4w-‐30m") AND _@me<=rela@ve_@me(now(), "$rel@me$@d-‐4w+$days$d+30m")), 4, (_@me>rela@ve_@me(now(), "$rel@me$@d-‐3w-‐30m") AND _@me<=rela@ve_@me(now(), "$rel@me$@d-‐3w+$days$d+30m")), 3, (_@me>rela@ve_@me(now(), "$rel@me$@d-‐2w-‐30m") AND _@me<=rela@ve_@me(now(), "$rel@me$@d-‐2w+$days$d+30m")), 2, (_@me>rela@ve_@me(now(), "$rel@me$@d-‐1w-‐30m") AND _@me<=rela@ve_@me(now(), "$rel@me$@d-‐1w+$days$d+30m")), 1 )
| eval shiR=case(isnotnull(w), "+"+w+"w-‐30m +"+w+"w-‐20m +"+w+"w-‐10m +"+w+"w-‐0m +"+w+"w+10m +"+w+"w+20m +"+w+"w+30m")
47
![Page 48: Building*A*Crystal*Ball:* - SplunkConf · Agenda TheProblem* ExisngTools Finding*A*BeZer*Way* Implementaon* Results* Caveats* Quesons * 4](https://reader033.vdocuments.mx/reader033/viewer/2022042005/5e6f388043f4e536bc148d56/html5/thumbnails/48.jpg)
Drop What We Don’t Need
| where isnotnull(shiR)
48
![Page 49: Building*A*Crystal*Ball:* - SplunkConf · Agenda TheProblem* ExisngTools Finding*A*BeZer*Way* Implementaon* Results* Caveats* Quesons * 4](https://reader033.vdocuments.mx/reader033/viewer/2022042005/5e6f388043f4e536bc148d56/html5/thumbnails/49.jpg)
Clone The Data And Compute New Time For Each Event
| makemv shiR | mvexpand shiR | eval @me=rela@ve_@me(_@me, shiR)
49
![Page 50: Building*A*Crystal*Ball:* - SplunkConf · Agenda TheProblem* ExisngTools Finding*A*BeZer*Way* Implementaon* Results* Caveats* Quesons * 4](https://reader033.vdocuments.mx/reader033/viewer/2022042005/5e6f388043f4e536bc148d56/html5/thumbnails/50.jpg)
Do The Math | eventstats avg($val$) as pred by @me | eval upper=if($val$>pred,$val$,pred) | eval lower=if($val$<pred,$val$,pred) | stats avg($val$) as pred, stdev(upper) as ustdev, stdev(lower) as lstdev by @me | eval low=pred-‐lstdev*(sqrt(1/(1-‐$confidence$/100))) | eval low=if(low<0, 0, low) | eval high=pred+ustdev*(sqrt(1/(1-‐$confidence$/100)))
50
![Page 51: Building*A*Crystal*Ball:* - SplunkConf · Agenda TheProblem* ExisngTools Finding*A*BeZer*Way* Implementaon* Results* Caveats* Quesons * 4](https://reader033.vdocuments.mx/reader033/viewer/2022042005/5e6f388043f4e536bc148d56/html5/thumbnails/51.jpg)
_@me Travel! | eval _@me=@me
* This doesn’t work reliably in Splunk versions prior to 5.4.3.
51
![Page 52: Building*A*Crystal*Ball:* - SplunkConf · Agenda TheProblem* ExisngTools Finding*A*BeZer*Way* Implementaon* Results* Caveats* Quesons * 4](https://reader033.vdocuments.mx/reader033/viewer/2022042005/5e6f388043f4e536bc148d56/html5/thumbnails/52.jpg)
Post Jump Cleanup | @mechart span=10m min(pred) as pred, min(low) as low, min(high) as high | where _@me>rela@ve_@me(now(), "$rel@me$@d")
AND _@me<=rela@ve_@me(now(), "$rel@me$+$days$d@d")
52
![Page 53: Building*A*Crystal*Ball:* - SplunkConf · Agenda TheProblem* ExisngTools Finding*A*BeZer*Way* Implementaon* Results* Caveats* Quesons * 4](https://reader033.vdocuments.mx/reader033/viewer/2022042005/5e6f388043f4e536bc148d56/html5/thumbnails/53.jpg)
forecast5w(val,confidence,rel@me,days) eval w=case( (_@me>rela@ve_@me(now(), "$rel@me$@d-‐5w-‐30m") AND _@me<=rela@ve_@me(now(), "$rel@me$@d-‐5w+$days$d+30m")), 5, (_@me>rela@ve_@me(now(), "$rel@me$@d-‐4w-‐30m") AND _@me<=rela@ve_@me(now(), "$rel@me$@d-‐4w+$days$d+30m")), 4, (_@me>rela@ve_@me(now(), "$rel@me$@d-‐3w-‐30m") AND _@me<=rela@ve_@me(now(), "$rel@me$@d-‐3w+$days$d+30m")), 3, (_@me>rela@ve_@me(now(), "$rel@me$@d-‐2w-‐30m") AND _@me<=rela@ve_@me(now(), "$rel@me$@d-‐2w+$days$d+30m")), 2, (_@me>rela@ve_@me(now(), "$rel@me$@d-‐1w-‐30m") AND _@me<=rela@ve_@me(now(), "$rel@me$@d-‐1w+$days$d+30m")), 1 ) | eval shiR=case(isnotnull(w), "+"+w+"w-‐30m +"+w+"w-‐20m +"+w+"w-‐10m +"+w+"w-‐0m +"+w+"w+10m +"+w+"w+20m +"+w+"w+30m") | where isnotnull(shiR) | makemv shiR | mvexpand shiR | eval @me=rela@ve_@me(_@me, shiR) | eventstats avg($val$) as pred by @me | eval upper=if($val$>pred,$val$,pred) | eval lower=if($val$<pred,$val$,pred) | stats avg($val$) as pred, stdev(upper) as ustdev, stdev(lower) as lstdev by @me | eval low=pred-‐lstdev*(sqrt(1/(1-‐$confidence$/100))) | eval low=if(low<0, 0, low) | eval high=pred+ustdev*(sqrt(1/(1-‐$confidence$/100))) | eval _@me=@me | @mechart span=10m min(pred) as pred, min(low) as low, min(high) as high | where _@me>rela@ve_@me(now(), "$rel@me$@d") AND _@me<=rela@ve_@me(now(), "$rel@me$+$days$d@d")
53
![Page 54: Building*A*Crystal*Ball:* - SplunkConf · Agenda TheProblem* ExisngTools Finding*A*BeZer*Way* Implementaon* Results* Caveats* Quesons * 4](https://reader033.vdocuments.mx/reader033/viewer/2022042005/5e6f388043f4e536bc148d56/html5/thumbnails/54.jpg)
Results
![Page 55: Building*A*Crystal*Ball:* - SplunkConf · Agenda TheProblem* ExisngTools Finding*A*BeZer*Way* Implementaon* Results* Caveats* Quesons * 4](https://reader033.vdocuments.mx/reader033/viewer/2022042005/5e6f388043f4e536bc148d56/html5/thumbnails/55.jpg)
Generate A Forecast
index=summary search_name="Sales -‐ Summary -‐ 10 min count" | @mechart span=10m sum(count) as actual | `forecast5w(actual, 90.0, +1d, 3)` Run over the last 5 weeks.
55
![Page 56: Building*A*Crystal*Ball:* - SplunkConf · Agenda TheProblem* ExisngTools Finding*A*BeZer*Way* Implementaon* Results* Caveats* Quesons * 4](https://reader033.vdocuments.mx/reader033/viewer/2022042005/5e6f388043f4e536bc148d56/html5/thumbnails/56.jpg)
Forecast Using forecast5w()
5 weeks of data, 3 days of forecast, 90% confidence intervals
56
![Page 57: Building*A*Crystal*Ball:* - SplunkConf · Agenda TheProblem* ExisngTools Finding*A*BeZer*Way* Implementaon* Results* Caveats* Quesons * 4](https://reader033.vdocuments.mx/reader033/viewer/2022042005/5e6f388043f4e536bc148d56/html5/thumbnails/57.jpg)
forecast5w() vs Reality
57
![Page 58: Building*A*Crystal*Ball:* - SplunkConf · Agenda TheProblem* ExisngTools Finding*A*BeZer*Way* Implementaon* Results* Caveats* Quesons * 4](https://reader033.vdocuments.mx/reader033/viewer/2022042005/5e6f388043f4e536bc148d56/html5/thumbnails/58.jpg)
Automa@c Forecas@ng Save search as “Sales Volume Forecast” and schedule to run every day over the previous 5 weeks.
index=summary search_name="Sales -‐ Summary -‐ 10 min count" | @mechart span=10m sum(count) as actual | `forecast(actual, 90.0, +1d, 1)`
58
![Page 59: Building*A*Crystal*Ball:* - SplunkConf · Agenda TheProblem* ExisngTools Finding*A*BeZer*Way* Implementaon* Results* Caveats* Quesons * 4](https://reader033.vdocuments.mx/reader033/viewer/2022042005/5e6f388043f4e536bc148d56/html5/thumbnails/59.jpg)
Alert index=summary
search_name="Sales -‐ Summary -‐ 10 min count" OR search_name="Sales Volume Forecast"
| where count<low
59
![Page 60: Building*A*Crystal*Ball:* - SplunkConf · Agenda TheProblem* ExisngTools Finding*A*BeZer*Way* Implementaon* Results* Caveats* Quesons * 4](https://reader033.vdocuments.mx/reader033/viewer/2022042005/5e6f388043f4e536bc148d56/html5/thumbnails/60.jpg)
Test The Alert Based On History • Backfill the forecast for the last month or so: splunk cmd python fill_summary_index.py -‐app search \
-‐name "Sales Volume Forecast" -‐et -‐1mon -‐lt now -‐j 8
• Use @mechart to find out when your alert would have fired: index=summary
search_name="Sales -‐ Summary -‐ 10 min count" OR search_name="Sales Volume Forecast"
| @mechart sum(count) as count, sum(low) as low | where count<low
60
![Page 61: Building*A*Crystal*Ball:* - SplunkConf · Agenda TheProblem* ExisngTools Finding*A*BeZer*Way* Implementaon* Results* Caveats* Quesons * 4](https://reader033.vdocuments.mx/reader033/viewer/2022042005/5e6f388043f4e536bc148d56/html5/thumbnails/61.jpg)
Caveats
![Page 62: Building*A*Crystal*Ball:* - SplunkConf · Agenda TheProblem* ExisngTools Finding*A*BeZer*Way* Implementaon* Results* Caveats* Quesons * 4](https://reader033.vdocuments.mx/reader033/viewer/2022042005/5e6f388043f4e536bc148d56/html5/thumbnails/62.jpg)
Caveats
62
• Doesn’t perform well on low volume @me series data • Must adjust the default MAX_DAYS_HENCE in props to create forecast data more than two days in advance
• Needs a feedback loop so that abnormal data can be excluded from future forecast calcula@ons
• Your mileage may vary
![Page 63: Building*A*Crystal*Ball:* - SplunkConf · Agenda TheProblem* ExisngTools Finding*A*BeZer*Way* Implementaon* Results* Caveats* Quesons * 4](https://reader033.vdocuments.mx/reader033/viewer/2022042005/5e6f388043f4e536bc148d56/html5/thumbnails/63.jpg)
Wrap Up
![Page 64: Building*A*Crystal*Ball:* - SplunkConf · Agenda TheProblem* ExisngTools Finding*A*BeZer*Way* Implementaon* Results* Caveats* Quesons * 4](https://reader033.vdocuments.mx/reader033/viewer/2022042005/5e6f388043f4e536bc148d56/html5/thumbnails/64.jpg)
Go Forth And Predict The Future!
64
Now that you’ve seen how to build a crystal ball, the only ques@on is…
What will you forecast?
![Page 65: Building*A*Crystal*Ball:* - SplunkConf · Agenda TheProblem* ExisngTools Finding*A*BeZer*Way* Implementaon* Results* Caveats* Quesons * 4](https://reader033.vdocuments.mx/reader033/viewer/2022042005/5e6f388043f4e536bc148d56/html5/thumbnails/65.jpg)
Ques@ons?
![Page 66: Building*A*Crystal*Ball:* - SplunkConf · Agenda TheProblem* ExisngTools Finding*A*BeZer*Way* Implementaon* Results* Caveats* Quesons * 4](https://reader033.vdocuments.mx/reader033/viewer/2022042005/5e6f388043f4e536bc148d56/html5/thumbnails/66.jpg)
THANK YOU
![Page 67: Building*A*Crystal*Ball:* - SplunkConf · Agenda TheProblem* ExisngTools Finding*A*BeZer*Way* Implementaon* Results* Caveats* Quesons * 4](https://reader033.vdocuments.mx/reader033/viewer/2022042005/5e6f388043f4e536bc148d56/html5/thumbnails/67.jpg)
Photo Credits
67
rjrgmc28 hZps://www.flickr.com/photos/rjrgmc28/ (cropped from original) License: hZps://[email protected]/licenses/by-‐sa/2.0/
Judy van der Velden hZps://www.flickr.com/photos/judy-‐van-‐der-‐velden/ License: hZps://[email protected]/licenses/by-‐nc-‐nd/2.0/
Silverisdead hZps://www.flickr.com/photos/56624456@N00/ License: hZps://[email protected]/licenses/by/2.0/
Nicolas Connault hZps://www.flickr.com/photos/nicolasconnault/ License: hZps://[email protected]/licenses/by-‐nd/2.0/