capacity planning for lamp mysql conf 2007

8/15/2019 Capacity Planning for LAMP MySQL Conf 2007

1/54

capacityplanning for

LAMPwhat happens after youre scalable

MySQL Conf and Expo

April 2007


2/54

John Allspaw

Engineering Manager(Operations) at flickr(Yahoo!)


3/54

Yay!

Youre scalable! (or not)

Now you can simply addhardware as you needcapacity.

(right ?)


4/54

But:

How many servers ?


5/54

BUT, um,

wait....How many databases ?

How many webservers ?How much shared storage ?

How many networkswitches ?What about caching ?

How many CPUs in all ofthese ?

How much RAM ?How many drives in each ?WHEN should we order all


6/54

some stats- ~35M photos in squid

cache (total)

- ~2M photos in squids RAM

- ~470M photos, 4 or 5

sizes of each

- 38k req/sec to memcached(12M objects)

- 2 PB raw storage(consumed about ~1.5TB on

Sunday)


7/54

capacity


8/54

capacity

doesntmean

speed


9/54

capacity is for

business


10/54

too soon too late

Buying enoughfor now

n

ot

en

oug

h

too

muc

h


11/54

3 main parts- Planning (what ?/why ?/when ?)

- Deployment(install/config/manage)

- Measurement (graph theworld)


12/54

boring queueing

theoryForced Flow Law:

Xi = Vi x X0

Littles Law:

N = X x R

Service Demand Law:Di = Vi x Si = Ui / X0


13/54

my theory

capacity planning mathshould be based on real

things, not abstract ones.


14/54

predicting the


15/54

consumable


16/54

concurrent

i ti


17/54

consi erations:

social

applications- Have the networkeffect

- Exponential growth

i ti


18/54

consi erations:

social

applicationsEvent-related growth(press, news event, social

trends, etc.)

Examples:

London bombing, holidays, tsunamis, etc.


19/54

What do you

haveNOW ?

When will your currentcapacity be depleted or

outgrown ?


20/54

finding

ceilings

MySQL (disk IO ?)

SQUID (disk IO ? or CPU ?)

memcached (CPU ? ornetwork ?)


21/54

forget

benchmarksboring

to use in capacityplanning...not usuallyworth the time

not representative of realload


22/54

test in

production


23/54

what do you

expect ?define what is acceptable

examples:

squid hits should takeless than X milliseconds

SQL queries less than Ymilliseconds, and also

keep up with replication


24/54measurement


25/54

accept the

observer effect

measurement is anecessity.

its not optional.


26/54

http://ganglia.s


27/54


28/54


29/54


30/54

super simple

graphing#!/bin/sh

/usr/bin/iostat -x 4 2 sda | grep

-v ^$ | tail -4 > /tmp/disk-io.tmp

UTIL=`grep sda /tmp/disk-io.tmp |awk '{print $14}'`

/usr/bin/gmetric -t uint16 -n disk-util -v$UTIL -u '%'


31/54

memcached

w at i you


32/54

w at i youhave graphs but

no raw data ?GraphClick

http://www.arizona-software.ch/applications/g

raphclick/en/


33/54


34/54

application

usageUsage stats are just asimportant

as server stats!

Examples:

# of user registrations

# of photos uploaded

every hour

not a stra g t


35/54

not a stra g tline

th t


36/54

another not

straight line

b t t i ht


37/54

but straight

relationships!


38/54

measurement

examples


39/54

queries


40/54

disk I/O


41/54

What we know

nowwe can do at least 1500qps (peak) without:

- slave lag

- unacceptable avgresponse time

- waiting on disk IO


42/54

MySQL capacityfind ceilings of existing

h/wtie app usage to server

stats

find ceiling:usage ratio

do this again:

- regularly (monthly)- when new features arereleased

-

caching


43/54

caching

maximums

cac ing


44/54

cac ingceilings

squid, memcacheworking-set specific:

- tiny enough to all fitin memory ?

- some/more/all on disk ?

- watch LRU churn


45/54

churning full

cachesCeilings at:

- LRU ref age small

enough to affect hit

ratio too much

- Request rate largeenough to affect disk IO(to 100%)

squid requests


46/54

squid requests

and hits


47/54

squid hit ratio

LRU reference


48/54

LRU reference

age

response


49/54

responsetimes


50/54

What we know

nowwe can do at least 620req/sec (peak) without:

- LRU affecting hitratio

- unacceptable avgresponse time

- waiting too much ondiskIO


51/54

not full caches

(working set smaller thanmax size)

- request rate largeenough to bring network or

CPU to 100%


52/54

deployment

d


53/54

Automated

Deploy ToolsSystemImager/SystemConfigurator-

http://wiki.systemimager.org

CVSup:

- http://www.cvsup.org

Subcon:

-http://code.google.com/p/subc
http://wiki.systemimager.org/http://www.cvsup.org/http://code.google.com/p/subcon/http://code.google.com/p/subcon/http://www.cvsup.org/http://wiki.systemimager.org/


54/54

questions ?

http://flickr.com/photos/gaspi/62165

http://flickr.com/photos/marksetchel

http://flickr.com/photos/sheeshoo/72

http://flickr.com/photos/jaxxon/1655

http://flickr.com/photos/bambooly/29

http://flickr.com/photos/colloidfarl

http://flickr.com/photos/sparktograp
http://flickr.com/photos/gaspi/62165296/http://flickr.com/photos/marksetchell/27964330/http://flickr.com/photos/sheeshoo/72709413/http://flickr.com/photos/jaxxon/165559708/http://flickr.com/photos/bambooly/298632541/http://flickr.com/photos/colloidfarl/81564759/http://flickr.com/photos/sparktography/75499095/http://flickr.com/photos/sparktography/75499095/http://flickr.com/photos/colloidfarl/81564759/http://flickr.com/photos/bambooly/298632541/http://flickr.com/photos/jaxxon/165559708/http://flickr.com/photos/sheeshoo/72709413/http://flickr.com/photos/marksetchell/27964330/http://flickr.com/photos/gaspi/62165296/

capacity planning for lamp mysql conf 2007

Documents