my talk at linux piter 2015
TRANSCRIPT
![Page 1: My talk at Linux Piter 2015](https://reader034.vdocuments.mx/reader034/viewer/2022042723/5872cdb21a28ab74188b46d7/html5/thumbnails/1.jpg)
Performance optimization in LinuxTales from the trenches
Alex Chistyakov, Principal Engineer, Git in SkyLinux Piter 2015
![Page 2: My talk at Linux Piter 2015](https://reader034.vdocuments.mx/reader034/viewer/2022042723/5872cdb21a28ab74188b46d7/html5/thumbnails/2.jpg)
Who are we?
● A small consulting company based in SPb.
● Web operations
● Automation
● Performance tuning
● Sponsors of local DevOps meetup
![Page 3: My talk at Linux Piter 2015](https://reader034.vdocuments.mx/reader034/viewer/2022042723/5872cdb21a28ab74188b46d7/html5/thumbnails/3.jpg)
Who are you?
● Linux fans?
● Developers?
● Web developers, maybe?
● System architects?
● Performance engineers?
![Page 4: My talk at Linux Piter 2015](https://reader034.vdocuments.mx/reader034/viewer/2022042723/5872cdb21a28ab74188b46d7/html5/thumbnails/4.jpg)
Okay, why Linux?
● Is there anything else?
● According to W3Cook stats, Linux
serves 95.8% of public web sites
● And it's on the desktop too!
● (At least on my desktop)
![Page 5: My talk at Linux Piter 2015](https://reader034.vdocuments.mx/reader034/viewer/2022042723/5872cdb21a28ab74188b46d7/html5/thumbnails/5.jpg)
Linux in the perfect world
● Linux fans?
● Developers?
● Web developers, maybe?
● System architects?
● Performance engineers?
![Page 6: My talk at Linux Piter 2015](https://reader034.vdocuments.mx/reader034/viewer/2022042723/5872cdb21a28ab74188b46d7/html5/thumbnails/6.jpg)
Linux in the real world
● Linux fans?
● Developers?
● Web developers, maybe?
● System architects?
● Performance engineers?
![Page 7: My talk at Linux Piter 2015](https://reader034.vdocuments.mx/reader034/viewer/2022042723/5872cdb21a28ab74188b46d7/html5/thumbnails/7.jpg)
504 on the main page!
● A customer is stressed extremely
● Reaction should be quick and effective
● The obvious plan does not work
● We should be prepared!
![Page 8: My talk at Linux Piter 2015](https://reader034.vdocuments.mx/reader034/viewer/2022042723/5872cdb21a28ab74188b46d7/html5/thumbnails/8.jpg)
The obvious plan
● 1) Change something
● 2) Expect the situation to become better
● 3) Wait anxiously
● 4) ????
● 5) PROFIT!!!
● This plan is quite popular in fact
for some reason (simplicity?)
![Page 9: My talk at Linux Piter 2015](https://reader034.vdocuments.mx/reader034/viewer/2022042723/5872cdb21a28ab74188b46d7/html5/thumbnails/9.jpg)
The proper plan (top secret!)
● 1) Gather metrics (you have them already, don't you?)
● 2) Analyze metrics
● 3) Elaborate a hypothesis
● 4) Plan and make a single change
● 5) Repeat until success
● If you were proficient in physics at high
school, this plan should sound extremely familiar
![Page 10: My talk at Linux Piter 2015](https://reader034.vdocuments.mx/reader034/viewer/2022042723/5872cdb21a28ab74188b46d7/html5/thumbnails/10.jpg)
How to collect/analyze metrics?
● Brendan Gregg's observability tools diagram
![Page 11: My talk at Linux Piter 2015](https://reader034.vdocuments.mx/reader034/viewer/2022042723/5872cdb21a28ab74188b46d7/html5/thumbnails/11.jpg)
How do we collect/analyze?
● atop (30 sec intervals)
![Page 12: My talk at Linux Piter 2015](https://reader034.vdocuments.mx/reader034/viewer/2022042723/5872cdb21a28ab74188b46d7/html5/thumbnails/12.jpg)
How do we collect/analyze?
● atop (30 sec intervals)
● Ex-graphite stack (Grafana, InfluxDB or OpenTSDB
or Cyanite/Cassandra, but NOT Whisper, collectd)
![Page 13: My talk at Linux Piter 2015](https://reader034.vdocuments.mx/reader034/viewer/2022042723/5872cdb21a28ab74188b46d7/html5/thumbnails/13.jpg)
How do we collect/analyze?
● atop (30 sec intervals)
● Ex-graphite stack (Grafana, InfluxDB or OpenTSDB
or Cyanite/Cassandra, but NOT Whisper, collectd)
● NewRelic
![Page 14: My talk at Linux Piter 2015](https://reader034.vdocuments.mx/reader034/viewer/2022042723/5872cdb21a28ab74188b46d7/html5/thumbnails/14.jpg)
How do we collect/analyze?
● atop (30 sec intervals)
● Ex-graphite stack (Grafana, InfluxDB or OpenTSDB
or Cyanite/Cassandra, but NOT Whisper, collectd)
● NewRelic
● pidstat (not iotop)
![Page 15: My talk at Linux Piter 2015](https://reader034.vdocuments.mx/reader034/viewer/2022042723/5872cdb21a28ab74188b46d7/html5/thumbnails/15.jpg)
How do we collect/analyze?
● atop (30 sec intervals)
● Ex-graphite stack (Grafana, InfluxDB or OpenTSDB
or Cyanite/Cassandra, but NOT Whisper, collectd)
● NewRelic
● pidstat (not iotop)
● perf top and perf record
![Page 16: My talk at Linux Piter 2015](https://reader034.vdocuments.mx/reader034/viewer/2022042723/5872cdb21a28ab74188b46d7/html5/thumbnails/16.jpg)
How do we collect/analyze?
● atop (30 sec intervals)
● Ex-graphite stack (Grafana, InfluxDB or OpenTSDB
or Cyanite/Cassandra, but NOT Whisper, collectd)
● NewRelic
● pidstat (not iotop)
● perf top and perf record
● sysdig
![Page 17: My talk at Linux Piter 2015](https://reader034.vdocuments.mx/reader034/viewer/2022042723/5872cdb21a28ab74188b46d7/html5/thumbnails/17.jpg)
How do we collect/analyze?
● atop (30 sec intervals)
● Ex-graphite stack (Grafana, InfluxDB or OpenTSDB
or Cyanite/Cassandra, but NOT Whisper, collectd)
● NewRelic
● pidstat (not iotop)
● perf top and perf record
● sysdig
● iostat -x 1
![Page 18: My talk at Linux Piter 2015](https://reader034.vdocuments.mx/reader034/viewer/2022042723/5872cdb21a28ab74188b46d7/html5/thumbnails/18.jpg)
Most common case (a no-brainer)
● CPU time is too high (a lucky customer got an SSD)
● Disk saturation is above 60% (a not-so-lucky customer)
![Page 19: My talk at Linux Piter 2015](https://reader034.vdocuments.mx/reader034/viewer/2022042723/5872cdb21a28ab74188b46d7/html5/thumbnails/19.jpg)
Most common case (a no-brainer)
● CPU time is too high (a lucky customer got an SSD)
● Disk saturation is above 60% (a not-so-lucky customer)
● PHP and, of course, MySQL
![Page 20: My talk at Linux Piter 2015](https://reader034.vdocuments.mx/reader034/viewer/2022042723/5872cdb21a28ab74188b46d7/html5/thumbnails/20.jpg)
Most common case (a no-brainer)
● CPU time is too high (a lucky customer got an SSD)
● Disk saturation is above 60% (a not-so-lucky customer)
● PHP and, of course, MySQL
● InnoDB buffers are too low
● Synchronous commit is 'on'
![Page 21: My talk at Linux Piter 2015](https://reader034.vdocuments.mx/reader034/viewer/2022042723/5872cdb21a28ab74188b46d7/html5/thumbnails/21.jpg)
Most common case (a no-brainer)
● CPU time is too high (a lucky customer got an SSD)
● Disk saturation is above 60% (a not-so-lucky customer)
● PHP and, of course, MySQL
● InnoDB buffers are too low
● Synchronous commit is 'on'
● Too many slow queries
● Queries with 'filesort' in execution plan
![Page 22: My talk at Linux Piter 2015](https://reader034.vdocuments.mx/reader034/viewer/2022042723/5872cdb21a28ab74188b46d7/html5/thumbnails/22.jpg)
How to solve
● Install Anemometer, turn on slow queries log
● Range queries based on their cumulative exec time
● Read and understand execution plans
● Blame developers
● Cry in vain
![Page 23: My talk at Linux Piter 2015](https://reader034.vdocuments.mx/reader034/viewer/2022042723/5872cdb21a28ab74188b46d7/html5/thumbnails/23.jpg)
Story time!
● Case #1: measuring the measurer
● It seems that everybody still uses Graphite/Whisper
![Page 24: My talk at Linux Piter 2015](https://reader034.vdocuments.mx/reader034/viewer/2022042723/5872cdb21a28ab74188b46d7/html5/thumbnails/24.jpg)
Story time!
● Case #1: measuring the measurer
● It seems that everybody still uses Graphite/Whisper
● Even the big guys like Mail.Ru
![Page 25: My talk at Linux Piter 2015](https://reader034.vdocuments.mx/reader034/viewer/2022042723/5872cdb21a28ab74188b46d7/html5/thumbnails/25.jpg)
Story time!
● Case #1: measuring the measurer
● It seems that everybody still uses Graphite/Whisper
● Even the big guys like Mail.Ru
● Well, because SAS HDDs are cheap...
![Page 26: My talk at Linux Piter 2015](https://reader034.vdocuments.mx/reader034/viewer/2022042723/5872cdb21a28ab74188b46d7/html5/thumbnails/26.jpg)
Story time!
● Case #1: measuring the measurer
● It seems that everybody still uses Graphite/Whisper
● Even the big guys like Mail.Ru
● Well, because SAS HDDs are cheap…
● But...
![Page 27: My talk at Linux Piter 2015](https://reader034.vdocuments.mx/reader034/viewer/2022042723/5872cdb21a28ab74188b46d7/html5/thumbnails/27.jpg)
A challenger appears!
● InfluxDB vs. Whisper, July 2015
● The same set of metrics (carbon-relay-ng in the middle)
● And the winner is...
![Page 28: My talk at Linux Piter 2015](https://reader034.vdocuments.mx/reader034/viewer/2022042723/5872cdb21a28ab74188b46d7/html5/thumbnails/28.jpg)
Okay, we hate magic
● Whisper is just a set of RRD-like files on a plain old FS
● 20000 metrics lead you to 20000 files
● Accessing 20000 files every 10 secs is a major pain
● InfluxDB is a time series database based on an LSM-tree
● InfluxDB is much more write-optimized than 20000 separate
files on your ext4/XFS/you-name-it
● But, of course, SAS drives are quite cheap...
![Page 29: My talk at Linux Piter 2015](https://reader034.vdocuments.mx/reader034/viewer/2022042723/5872cdb21a28ab74188b46d7/html5/thumbnails/29.jpg)
In case you are not scared
● Case #2: the site got a SUDS (sudden unexpected death syndrome)
![Page 30: My talk at Linux Piter 2015](https://reader034.vdocuments.mx/reader034/viewer/2022042723/5872cdb21a28ab74188b46d7/html5/thumbnails/30.jpg)
In case you are not scared
● Case #2: the site got a SUDS (sudden unexpected death syndrome)
● Symptoms: everything slows down to a crawl (sounds familiar)
![Page 31: My talk at Linux Piter 2015](https://reader034.vdocuments.mx/reader034/viewer/2022042723/5872cdb21a28ab74188b46d7/html5/thumbnails/31.jpg)
In case you are not scared
● Case #2: the site got a SUDS (sudden unexpected death syndrome)
● Symptoms: everything slows down to a crawl (sounds familiar)
● NewRelic shows nothing unusual
![Page 32: My talk at Linux Piter 2015](https://reader034.vdocuments.mx/reader034/viewer/2022042723/5872cdb21a28ab74188b46d7/html5/thumbnails/32.jpg)
In case you are not scared
● Case #2: the site got a SUDS (sudden unexpected death syndrome)
● Symptoms: everything slows down to a crawl (sounds familiar)
● NewRelic shows nothing unusual
● Well, if parameters more suitable for a busy site that for a very
low traffic one can be called “nothing unusual”
● And this site is not busy at all
![Page 33: My talk at Linux Piter 2015](https://reader034.vdocuments.mx/reader034/viewer/2022042723/5872cdb21a28ab74188b46d7/html5/thumbnails/33.jpg)
Diagnostic card
● PHP is OK
● MySQL does not sort anything
● Top queries in MySQL sorted by total exec time are all indexed
● Every MySQL query runs very slow when there is even
moderate load
![Page 34: My talk at Linux Piter 2015](https://reader034.vdocuments.mx/reader034/viewer/2022042723/5872cdb21a28ab74188b46d7/html5/thumbnails/34.jpg)
But how did we solve it?
● Even a modern rig w/decent Xeons and SATA HDDs can be turned into a slug
● As simple as disabling AHCI in BIOS and staying on plain IDE
● Well this one was not that hard but was quite unusual
● Rented servers do not suffer from problems like this because
they are configured uniformly
● I can't easily explain how I came upon this solution, pure
intuition seemed to be involved
![Page 35: My talk at Linux Piter 2015](https://reader034.vdocuments.mx/reader034/viewer/2022042723/5872cdb21a28ab74188b46d7/html5/thumbnails/35.jpg)
Not scared enough yet?
● Then, case #3: another site got SUDS
![Page 36: My talk at Linux Piter 2015](https://reader034.vdocuments.mx/reader034/viewer/2022042723/5872cdb21a28ab74188b46d7/html5/thumbnails/36.jpg)
Not scared enough yet?
● Then, case #3: another site got SUDS
![Page 37: My talk at Linux Piter 2015](https://reader034.vdocuments.mx/reader034/viewer/2022042723/5872cdb21a28ab74188b46d7/html5/thumbnails/37.jpg)
Diagnostic card
● NewRelic blames PHP code
● Even the SSH console is slow
● Nothing unusual or unexpected in daily CPU load graphs
● CPU flamegraph shows nothing
![Page 38: My talk at Linux Piter 2015](https://reader034.vdocuments.mx/reader034/viewer/2022042723/5872cdb21a28ab74188b46d7/html5/thumbnails/38.jpg)
What is a «CPU flamegraph»?
![Page 39: My talk at Linux Piter 2015](https://reader034.vdocuments.mx/reader034/viewer/2022042723/5872cdb21a28ab74188b46d7/html5/thumbnails/39.jpg)
How did we solve it
● Analyzed atop recorded stats for outage periods
● atop is quite smart in fact and color suspected values in red or
blue
● IRQ % is over 50%
![Page 40: My talk at Linux Piter 2015](https://reader034.vdocuments.mx/reader034/viewer/2022042723/5872cdb21a28ab74188b46d7/html5/thumbnails/40.jpg)
How did we solve it
● Analyzed atop recorded stats for outage periods
● atop is quite smart in fact and color suspected values in red or
blue
● IRQ % is over 50%
● But what is “IRQ %” anyway?
● Oh, who cares, let's install Munin and get per-interrupt graphs
![Page 41: My talk at Linux Piter 2015](https://reader034.vdocuments.mx/reader034/viewer/2022042723/5872cdb21a28ab74188b46d7/html5/thumbnails/41.jpg)
A blast from the past
● Analyzed atop recorded stats for outage periods
● atop is quite smart in fact and color suspected values in red or
blue
● IRQ % is over 50%
● But what is “IRQ %” anyway?
● Oh, who cares, let's install Munin and get per-interrupt graphs
![Page 42: My talk at Linux Piter 2015](https://reader034.vdocuments.mx/reader034/viewer/2022042723/5872cdb21a28ab74188b46d7/html5/thumbnails/42.jpg)
How did we solve it
● Well we have not solved it yet
● The graph from previous slide is for past two days
● But at least we have a plan!
● https://help.ubuntu.com/community/ReschedulingInterrupts
![Page 43: My talk at Linux Piter 2015](https://reader034.vdocuments.mx/reader034/viewer/2022042723/5872cdb21a28ab74188b46d7/html5/thumbnails/43.jpg)
Summary
● Linux is cool
● Performance engineering is hard
● Don't panic!
![Page 44: My talk at Linux Piter 2015](https://reader034.vdocuments.mx/reader034/viewer/2022042723/5872cdb21a28ab74188b46d7/html5/thumbnails/44.jpg)
Thank you!
● Questions?
● Oh, BTW you can hire us!
● http://gitinsky.com
● Please do not forget to attend our meetups:
● http://meetup.com/Docker-Spb, http://meetup.com/Ansible-Spb,
http://meetup.com/DevOps-40