five lines i could not draw
TRANSCRIPT
![Page 1: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/1.jpg)
5 lines I couldn’t draw
Hi everybody
![Page 2: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/2.jpg)
I’m [email protected]@davejosephsen
github: djosephsen
I’m Dave and I work on the Ops team at Librato. In fact you’ve caught me in a bit of a transitional period because I’ve recently decided to move back to ops after spending two years as Librato’s developer evangelist, which has been a fascinating role that’s given me the opportunity to really branch out learn many new things.
![Page 3: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/3.jpg)
Thought Leader Power Moves
For example Here’s a shot of me doing an all male panel, which is just one of the many thought-leader power moves I’ve perfected over the last couple years in my former role as developer evangelist.
![Page 4: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/4.jpg)
Thought Leader Power Moves
I can also do that faux-earnest, touching my fingers together while wearing a blazer and pontificating at you thing. That’s totally in my repertoire so if you’re ever in doubt of the validity of my argument I can touch my fingers together and become super reasonable looking
![Page 5: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/5.jpg)
• “Resource”: •Human-being who works here
!
• “Lead”: •Human-being who doesn’t work here
!
• “Content Marketing: •Annoying people on twitter
!
• “Engagement” •Tricking people into talking to you
I’ve penetrated the marketing team’s vernacular, so yeah.. that took two years, but I can definitely sit in those meetings now and.. I mean I pretty much know mostly what’s going on I think.
![Page 6: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/6.jpg)
Thought Leader Power Moves
git push -f origin master!
I can force-push master from a vendor booth. That was a huge achievement get in this role. By the way I’ve heard if you force push master from an all male panel they have to make you CTO. Pro tip. But yeah the vender booth is like a second home to me now. I’ve done quite a bit of venderboothing over the last few years
![Page 7: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/7.jpg)
Graphing Nagios
Venderboothing
Here we are at twilio signal a few months ago. And something happened at signal I want to tell you about, because it happens a lot in the course of my venderboothing endeavors where
![Page 8: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/8.jpg)
Venderboothing
I’ll be there in the booth, you know, my fingernails lightly resting against one another. And I’ll be engaging with leads, and filling our funnel top with branding, and this neckbeardy dude
![Page 9: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/9.jpg)
will kind of slide up, and lurk. He’ll just stand there glaring at me as I do my booth dance, impossible to ignore, like He’s like, heavy. I mean not literally heavy I’m not body-shaming him, I just mean he’s like… laden with discontent you know?
![Page 10: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/10.jpg)
Graphing Nagios
BLARG ANOMOLY DETECTION!!!
and after listening for a while he’ll just spontaneously interrupt by blurting out something like “WHAT ABOUT ANOMOLY DETECTION!”. And by the way in every case this has been what my wife calls a “catestrophic digression”,
![Page 11: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/11.jpg)
Graphing Nagios
like this interruption will be so awkward and so rude that it will send people scurrying away from the booth out of either embarrassment or genuine fear for their own safety. And I’ll want to run away too but not running away from incredulous neck beards is another skill I’ve developed so instead
![Page 12: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/12.jpg)
Graphing Nagios
I’ll don my thought leader cap and be like “what an informed and thought provoking question, Let me ask you, what kind of monitoring tools are you using today?”
![Page 13: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/13.jpg)
and inevitably, he’ll launch into this apocalyptic tale of woe, wherein he will describe the faustian hell in which he is currently trapped. Like his companies product
![Page 14: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/14.jpg)
Perl!
is, I’m not making this up, a bunch of perl scripts that
![Page 15: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/15.jpg)
OTHER PEOPLES Perl!
some consultant wrote back in 2006, and for monitoring they have these
![Page 16: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/16.jpg)
Graphing NagiosMORE Perl!
other perl scripts that a different consultant wrote in 2008 and those perl scripts are watching those first perl scripts
![Page 17: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/17.jpg)
Graphing NagiosWINDOWS!!
And by the way they all run on Windows so it’s not just perl
![Page 18: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/18.jpg)
Graphing NagiosACTIVESTATE PERL!
but active state perl, and these active state perl scripts
![Page 19: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/19.jpg)
EMAILZ!
send emails when unexpected things happen or error states are detected
![Page 20: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/20.jpg)
Graphing NagiosMAPI EMAILZ!!!
and they’re using… MAPI on windows XP to do this,
![Page 21: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/21.jpg)
Graphing NagiosTHOUSANDS OF MAPI EMAILZ!!!
And of course they’re sending 5000 emails per day because unexpected state is basically always happening and of course nobody is paying any attention to the emails.
![Page 22: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/22.jpg)
Graphing NagiosTHOUSANDS OF MAPI EMAILZ!!!
and usually by this point he feels awful and I feel awful so I’ll say something “like wow, that sounds awful it must be a really frustrating and stressful environment for you, like I’m sorry to hear it. And he’ll be all”
![Page 23: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/23.jpg)
Graphing Nagios
it’s fine…. and I’ll be like. Man. You sure? because that sounds really bad, like I sincerely want to just hug you right now. Would you like a hug?
![Page 24: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/24.jpg)
Graphing Nagios
And he’ll be like no really it’s fine, I don’t need a AAAHHH oh my god hug me. And we’ll hug it out and he’ll cry a little bit, and then I’ll be like. You know what you should do? You should vi that .qmail-root file mister neckbeard. You know?
![Page 25: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/25.jpg)
Graphing Nagios| cat > /dev/null && echo ‘emailz,1,g’ | /usr/bin/statsd
Throw something like this in there. Just dev null those messages and count them instead. I’m not saying that’s your permanent solution or anything
![Page 26: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/26.jpg)
Graphing Nagios
but i mean you don’t even know what that signal LOOKS like. You might be able to alert on simple message volume, or the derivative, or maybe just seeing that line will TEACH you something about the behavior or rhythm of your system.
![Page 27: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/27.jpg)
Graphing NagiosYa think?
And he’ll be like? “fa foa foa You think?”
![Page 28: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/28.jpg)
Graphing NagiosDO EEET!
And I’ll be like heck yeah mister neckbeard. Go giterdone, and he’ll leave happy and with a lighter heart. It’s a funny story. But it’s also a true story. I’ve had this conversation at least 20 times. And today surrounded by all of you and my librato teamies it’s easy to forget, but if I’m being honest?
![Page 29: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/29.jpg)
This was me one job ago
I was this neckbeard. This was me. Not a long time ago either but like one job ago, I was him. And I more or less knew everything about monitoring that I know now, and yet I couldn’t .
![Page 30: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/30.jpg)
Graphing Nagios
#1
draw this line, any more than he could. And I can hear you like Really dave emails per second? I mean yes, of course I could draw a line of emails per second, but then so can he. What neither of us could do is make that cognitive leap of applying monitoring tools as a means of understanding system behavior independent of alerting. So this is line number one I couldn’t draw.
![Page 31: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/31.jpg)
Graphing NagiosSpoiler alert:
There will be 4 more
And I come across these pretty commonly these days where I’ll see some data that we’re using internally and be like man.. I never could have drawn one job ago and I just kind of thought it’d be interesting to explore the reasons why. Live on stage in front of a room full of visionaries.
![Page 32: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/32.jpg)
I was carrying a misapprehension about what monitoring was and
whom it was for
Because in every case the problem wasn’t a lack of technical aptitude, it was wrapped up in my beliefs and expectations. In this particular case, the problem . The thing that’s causing our cognitive dissonance is that these perl scripts are sending alerts already. So to us— to mr neck beard and I, it seemed like monitoring was already in place. Box checked. Monitoring done.
![Page 33: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/33.jpg)
Because to us monitoring the thing that made alerts happen. We had no means of describing an undertaking called monitoring that was meaningfully discreet from alerting. That’s why monitoring is an ops thing. It’s about uptime. So obviously ops owns it. So as an ops person the pattern was that we would install or inherit this thing
![Page 34: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/34.jpg)
this, guard-dog-like entity. I would tell it where to sit, and it would bark whenever it felt like something wasn’t right.
![Page 35: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/35.jpg)
And it was great at barking. It’d bark all day and night bark bark bark bark bark. And that sucked. so I thought, maybe I should train it. So I’d put in a ticket to train it
![Page 36: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/36.jpg)
Squirrel! ZOMG SQUIRREL!!! It’s RIGHT THERE!
YOUGUISE? SQUIRRELL!
but I’d never work the ticket because it was literally nobody’s priority but mine. I’d feel guilty if I worked on it because it seemd self indulgent or I’d feel guilty if I didn’t because dogs sitting over there barking at squirrels all night and it’s just embarrassing
![Page 37: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/37.jpg)
But what about squirrels?
and maybe eventually I’d get some time to take it aside and be like listen. Thread contention? Bark. Ice cream trucks? NO BARK. But no matter what I told it, was never something that could help me out with stuff like these perl emails because
![Page 38: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/38.jpg)
Oh sure. Blame the puppies Dave
ultimately our relationship was Prescriptive in the wrong direction. The dog was always telling me what I was interested in, so the best we could ever hope to achieve was this on going negotiation about what to bark at how much barking was enough barking, it always became about the barking
![Page 39: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/39.jpg)
Monitoring is not FOR alerting
Here are two important things I no longer believe. These are the things that I think makes me different from mister neckbeard. First, I don’t believe monitoring is for alerting. It’s not about uptime.
![Page 40: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/40.jpg)
Nobody OWNS monitoring
Next, monitoring is neither my responsibility nor is it my burden. It’s not mine. not my pet. It’s a tape measure. It’s a tape measure that I get to share with every engineer I work with.
![Page 41: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/41.jpg)
Ops owns Monitoring
Everyone owns
Monitoring
I think if you’ve been listening to the real speakers, basically all the ones who aren’t me, you’ll find that this is the probably most important underlying belief that differentiates them from mister neckbeard. People who run effective monitoring infrastructure
![Page 42: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/42.jpg)
believe we all get to ask questions. We all get to measure things. Not just ops, not just dev, not just DBA, everybody who cares, gets to measure, and we all get to use the same tape measure, and it’s perfectly reasonable to expect to get accurate, timely answers to our questions, and that’s what MONITORING is for.
![Page 43: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/43.jpg)
Monitoring is FOR asking questions
It’s the infrastructure that makes it possible for everyone to understand system behavior. That’s what monitoring is for me now, today. That’s how it works. Not because I installed or bought some particular collection of tools or learned about percentiles, but exactly because my expectations have changed. Does that make sense?
![Page 44: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/44.jpg)
You might be fascinated with anomaly detection
because your input signal sucks?
What If I Told You:
And mark was right when he said tools matter, but the tools are there today and yet many still suffer. Like you can build the metrics infrastructure you need right now, that’s hard, and expensive but possible. Or you can buy it, and that’s easier and expensive but possible. But to make either of those actually work, you still need to change the people. That’s a lot harder. No combination of bleeding edge tools, no amount of fancy anomaly detection is going to save mister neck beard until he understands that monitoring is not for alerting, and that measuring things is everyone’s job.
![Page 45: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/45.jpg)
Complexity Isolates
And Mr Neckbeard has another problem too. He’s embracing complexity. See? To him, those emails are his burden to bear, his lot in life. They are the hand grenade upon which he will jump to save us all. And that belief isolates him. It mires him in complexity, and he believes that’s fine. Let me show you what I mean..
![Page 46: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/46.jpg)
#2
Here’s line number two that I couldn’t draw. And I can hear you saying Dave, that isn’t even a line. Like, you literally had one job dude and the first line was disappointing and this isn’t even a line. But the line you aren’t seeing here is actually a REALLY important … lack of a line.
![Page 47: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/47.jpg)
because what you aren’t seeing here is the number of people currently using the Librato API who are being throttled. So slight digression for context this is a pretty common problem when new users are wiring us up for the first time and what things do they send?
![Page 48: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/48.jpg)
All the things! Cheslock knows all about that you can ask him.. So rather than surprise people with a million dollar bill, We catch unlikely new ingest throttling and we’ll shoot an email or whatever that’s like hey um, maybe dial that back unless you actually want to pay us the GDP of uraguy every month.
![Page 49: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/49.jpg)
and a whole bunch of metrics like this are physically mounted to the wall next to our support team, because they’re the ones who are going to be there to help the user understand why they’re suddenly getting http500’s but the interesting part is that these weren’t created for support. These metrics in fact, were originally put in place by the engineer who implemented throttling to understand what that signal looked like
![Page 50: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/50.jpg)
And this made me wonder like, how did this happen. How did first level support begin repurposing API metrics?
![Page 51: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/51.jpg)
With whom shall I share my bounty of hard-won
metric data?
Was there some cuddly API engineer who, in a spontaneous bout of altruism went to go
![Page 52: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/52.jpg)
devops unicorn cuddle with the support team and make rainbow metrics babies of team spirit? That’s amazing, I want to meet these api engineers, so I went to go talk to them and
![Page 53: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/53.jpg)
they seemed like typical software engineers who exhibit the typical demeanor and mannerisms that one expects software engineers to manifest. So yeah long story short I think what’s happening here
![Page 54: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/54.jpg)
Cynefin
is something like emergent cynefin.
![Page 55: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/55.jpg)
and if you’re not familiar, cynefin is a framework that helps us make sense of complexity
![Page 56: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/56.jpg)
The idea is that you categorize the complexity you’re dealing with, and then you attempt to move from whatever category you’re in, to the next less complex category until you hit
![Page 57: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/57.jpg)
obvious right here, which you can see is sort of an uphill climb
![Page 58: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/58.jpg)
Things you need to move: • Control •Understanding •Standardization
but my time at Librato has convinced me that cynefin can be an emergent property of effective monitoring systems. By which I mean effective monitoring just sort of organically provides you a lot of the stuff you need to move toward obvious
![Page 59: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/59.jpg)
And I can hear you like really dave? emergent cynefin properties? You should have stuck with unicorn babies of team spirit. Like do you even know what you sound like when you say shit like that? which, yes, I hear myself
![Page 60: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/60.jpg)
Cynefin(no, for reals tho)
I mean the processes line up pretty well. What you need to climb the cynefin ladder is pretty much what you give a decent telemetry system to people who understand that their job is to measure things.
![Page 61: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/61.jpg)
And this is a perfect example. Our support team was able to move from a very opaque and chaotic form of complexity straight to obvious by repurposing monitoring data from another team, and today my friend Nik on the support team can look up at that very real, very physical wall and say
![Page 62: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/62.jpg)
“Behold. Throttled users!”, and go talk to them about it. That’s a first-level support team that organically understands the concept of http return codes, and services oriented architecture, and API backpressure. Nobody wrote them a manual for that. That’s Cynefin at work.
![Page 63: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/63.jpg)
I could never draw lines like this… not-line. And the reason I couldn’t draw this not-line was because like mister neck beard I thought that embracing complex things 5000 janky perl emails was my job. I thought complexity was my lot in life,
![Page 64: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/64.jpg)
Me dissecting somebody’s javascript circa 2003
I was totally the guy who would sit down with that incomprehensible bowl of spagetti code that some mean-spirited consultant wrote in 1997 and I’d be like I hate not understanding this. “I’m not leaving until this is understood. I’m not going to a meeting, I’m not going to lunch, I’m not going home”. And managers would come looking for me like did dave show up today. and my teamies would have to be like
![Page 65: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/65.jpg)
don’t bug him he’s dissecting some janky code. This is a real picture, this is my boss steve at IBM global services bringing me a sandwich. So For the millenials in the audience, this is actually what team spirit looked like in corporate america in the late 90’s. But then once I had it figured it out I’d become the owner of that janky perl forever. The only person who ever understood it and then people would be like
![Page 66: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/66.jpg)
yo dave, that janky perl thing is broke again. Right? They’d dump it on me when it broke and that’s perfectly rational, because why should they crawl down there with me? Why should I want them to?
![Page 67: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/67.jpg)
Graphing NagiosEXPERIENCE
5000 JANKY EMAILS
pain hurts y'all. It’s painful, so embracing it just isolates you, even from other engineers, because there’s only so much pain each of us can endure, we just can’t really go around willy-nilly embracing each others pain. It’s just not a tenable scaling model.
![Page 68: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/68.jpg)
But simplicity feels fantastic. Simplicity wants to be shared and celebrated. I should have always been working to reduce complexity instead of just accepting it, but I never realized that my monitoring tools could help simplify things.
![Page 69: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/69.jpg)
I used to think this was about as simple as simple got. I used to make things like this when I understood something. Well I still do I’ll draw a really complicated picture of the really complicated thing. I was so close, but I just never took that next step. the one that was like lets
![Page 70: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/70.jpg)
#3
Simplify that into something that isn’t painful to understand. This is line 3, and now you’re like dave that’s also not a line, so not only are you 1 for three on following through with the click bait listicle title you sold us, line one was super boring, and you sir, are a lying deceitful faud.
![Page 71: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/71.jpg)
#3
so OK captain pedantic, here’s the line. I couldn’t have drawn it because I never had an amazing dashboard like this beneath the lines I would draw
![Page 72: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/72.jpg)
#3
this dashboard is actually a simplification of
![Page 73: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/73.jpg)
This diagram. the curator of those metrics, took this diagram and made it obvious making one row
![Page 74: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/74.jpg)
Row Per SLB
for every SLB in that architecture diagram, which, if you think about it is an interesting way to simplify your understanding of service ingress because what does every service have in common? a load balancer.
![Page 75: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/75.jpg)
Latency
Availability Traffic
so for each load balancer lets break down a few golden signals and these are like, if any blocking outage happens inside any of these services, you’re going to see it in one of these signals, it’s guaranteed. If you don’t see it in one of these signals, then it’s by definition not a blocking problem.
![Page 76: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/76.jpg)
#3
And I want to stress that again, the person who curated this view was not the person who wrote the instrumentation to get this data. Different people, different concerns, and the monitoring system is enabling them to work together to reduce complexity, and aid comprehension.
![Page 77: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/77.jpg)
I never could have drawn that line over this amazing dashboard, I never realized I could use monitoring tools to build bridges to help other people understand the pain I was experiencing.
![Page 78: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/78.jpg)
Everybody gets to measure things
Nobody OWNS monitoring
And then one day I hire into this shop where everybody can measure things, and nobody owns monitoring
![Page 79: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/79.jpg)
and all these people are building all this stuff, and they’re taking measurements as they go
![Page 80: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/80.jpg)
And then other people get a hold of those signals and refine them, and cynefin happens
![Page 81: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/81.jpg)
And bam, suddenly first-level support understands API backpressure. And I’m trippin out like three weeks ago I was trapped in a perpetual
![Page 82: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/82.jpg)
Srsly tho; squirrels. Bark or don’t bark?
tire-fire with this clueless watch-dog and NOBODY cared. Like nobody even KNEW. I was alone with my bij despite being surrounded by other engineers
![Page 83: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/83.jpg)
#4
I’m sorry it’s just a stark contrast. Like, check out line number four here. This measures the storage latency introduced by our API matching metric names to UIDs. Point being, this is subtle latency metric. Like I can’t describe it to you in less than 15 words. But check this out,
![Page 84: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/84.jpg)
<redacted>
An Ops guy named Benjo, is working with it. That’s pretty crazy right? I mean where I come from, in the tire-fire it’s atypical for ops people to work with latency data that describes job execution inside the database. I mean in the tire fire this is what we referred to as somebody else’s problem. But OK Maybe Ben’s just a really savvy guy.
![Page 85: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/85.jpg)
<redacted>
But, wait benjo the ops person is not only wise to this intricate db latency issue but he’s correlating it back to system metrics. Ok huh, that seems extraordinarily astute to me, I mean even if you have the domain knowledge
![Page 86: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/86.jpg)
<redacted>
Wait hold on a sec, he is actually talking to Data Engineering about this? Ok, savvy, astute and brave. Or maybe he just doesn’t know that data engineers are mean
![Page 87: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/87.jpg)
<redacted>
<redacted>
woah, what? they’re actually responding and working together with him? And evidently so is the Front-End Team? Like What in the actual HELL is happening here? How does Ben the ops guy have all this Data Engineering domain knowledge? And why isn’t anyone being mean to him? Where are the passive-aggressive insults? The hostility and mistrust I’ve come to expect from engineers working in other teams? I mean this is dev and ops, this is
![Page 88: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/88.jpg)
This is DOGS AND CATS. LIVING TOGETHER. I’m SHOCKED. It’s Shocking!
![Page 89: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/89.jpg)
#4
I’ve certainly never been able to draw lines like that. I never knew enough about what was going on around me to even work with data like this. I mean this is a Line that literally bridge disciplines.
![Page 90: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/90.jpg)
#4
Look, if you squint, you can almost see the bridge that this line creates, between
![Page 91: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/91.jpg)
#4
Data Engineering and Operations. And again I can’t help but wonder how this happened. Is it possible that effective monitoring can bring about cultural change?
![Page 92: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/92.jpg)
#4
Because it looks to me like that’s what’s happened here. It looks like the combination of an effective telemetry infrastructure, combined with people who understand that measuring things is their job, has ultimately changed how people interact with each other in this shop. Good monitoring changes people. That’s kind of mindblowing.
![Page 93: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/93.jpg)
#5
So speaking of culture how much time do I have, ok good, because this is the good part, Line number 5. So this is a funny story about Bryan one of our integrations engineers, and this happened a few months ago now, and for the record I publicly apologize in advance to Brian I’m sorry dude, if you’re watching for shaming you like this on the internet, but in my defense.. it was pretty funny though
![Page 94: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/94.jpg)
so what happened was, Bryan was working on making our UI faster. And up top he rolls out a change, and that change? Makes the UI faster. So mission accomplished, good jorb Bryan you done it. And to be clear he’s graphed the performance data. I mean job done, homework done, he HAS a graph showing the stuff becoming faster.. but what he’s pasted in here
![Page 95: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/95.jpg)
is not that graph. It’s the mouse over of the tooltip. like he drew the graph, moused over it, took a screenshot of the tooltip of the individual datapoints, for a single polling interval, and then pasted THAT in channel. And for context, not only does Brian work for a startup whose singular purpose is the drawing linegraphs depicting time series data, but his Boss at the time is literally
![Page 96: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/96.jpg)
Line graphs FTW
the inventor of graphite AND this conference. Basically he works at line-graph-co for the godfather of linegraphs. And he’s basically just walked up to the godfather of line graphs like “Behold my assortment of individual datapoints!” but wait.. it gets better..
![Page 97: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/97.jpg)
And then he says… ignore the zeros! omg so amazing.
![Page 98: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/98.jpg)
I mean look.. if we ignore the zero’s literally we have two values. Like Bryan, sit down, I think it’s time we had the data-to-ink ratio talk bro. Once upon a time there was a man named Tufte…
![Page 99: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/99.jpg)
And I also want to point out the time stamps here, because it’s only a matter of seconds before his team begins expressing their confusion, like wait.. what?
![Page 100: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/100.jpg)
Is me messing with me RN?
Like I can almost see Dixon at home, head tilted to the side, unsure if this is some kind of elaborate troll, he’s like maybe everybody got together and agreed to not paste any line graphs in channel for the whole week. Which actually is kind of a brilliant troll and also something we’d totally do, but no, this was all bryan
![Page 101: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/101.jpg)
Anyway, then Bryan facepalms… and throws line number 5 in the channel
![Page 102: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/102.jpg)
And I never could have drawn this line, because I’ve never had a team around me that actually cared this much about what I was working on. If I came to someone with some amazing data that I was super proud of they’d look at it like
![Page 103: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/103.jpg)
thatscoolIguessorwhatever
yeah um wow. Thats cool i guess or whatever. doctor mc-showoff. Anywayz pretty busy so please stop being in my cubicle now.
![Page 104: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/104.jpg)
but look at the love here, these people want to geek out on the data with you. They want to celebrate your win. Not just in the fortune 500 goals and gift cards way, but by actually quantifying the y-axis of your success. They want to comprehend your win so bad they are confused when they lack sufficient data to comprehend your win, which I find astounding.
![Page 105: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/105.jpg)
and at the risk of sounding campy I guess I just wanted to say I love mah teamies at Librato and I love all of you as well, and I wanted to thank you for working to make effective monitoring happen in your shops, and building tools to make it happen for other people. So, Sincerely, thank you, you make me want to come to work every day.
![Page 106: Five Lines I Could Not Draw](https://reader031.vdocuments.mx/reader031/viewer/2022021918/588899d91a28ab264b8b4a5d/html5/thumbnails/106.jpg)
Questions?
@davejosephsen
And at this point the conference organizers have insisted that I allow you to ask questions but I have read the code of conduct which has literally nothing to say on the subject of speakers shouting smoke bomb and running off stage if they are confronted with a hostile question. SO that is a right I reserve.