scaling php to 40 million uniques

Scaling PHP to 40 Million Uniques

Northeast PHP 2013Jonathan Klein@jonathanklein

Saturday, August 17, 13

Scaling PHP to 60 Million Uniques

Northeast PHP 2013Jonathan Klein@jonathanklein


Who Am I?


Who Am I?

• Performance Engineer at Etsy


Who Am I?

• Performance Engineer at Etsy• Boston Web Perf Meetup Organizer


Who Am I?

• Performance Engineer at Etsy• Boston Web Perf Meetup Organizer• Previously at Wayfair


Who Am I?

• Performance Engineer at Etsy• Boston Web Perf Meetup Organizer• Previously at Wayfair• Led team that converted to PHP


Slides, Links:jkle.in/nephp


codeascraft.com


Some Etsy Stats


Some Etsy Stats• 1.5 billion page views/month


Some Etsy Stats• 1.5 billion page views/month• Almost $1B in sales last year


Some Etsy Stats• 1.5 billion page views/month• Almost $1B in sales last year• Over 1M lines of PHP


Some Etsy Stats• 1.5 billion page views/month• Almost $1B in sales last year• Over 1M lines of PHP• 60M+ unique visitors/month


2012 Traffic


August

2012 Traffic


August

2012 Traffic

50% Higher


Holidays:~90M Uniques


A few others...


A few others...

• Search (Solr)


A few others...

• Search (Solr)• Gearman


A few others...

• Search (Solr)• Gearman• Redis


A few others...

• Search (Solr)• Gearman• Redis• Postgres (legacy)


Hardware (Supermicro)


Hardware (Supermicro)2x 8-core Intel E5-2960 CPUs



24GB of RAM



24GB of RAM

160GB SSD


Apache


Apache

• Apache 2.2


Apache

• Apache 2.2• Prefork MPM


Apache

• Apache 2.2• Prefork MPM• mod_php5


Apache

• Apache 2.2• Prefork MPM• mod_php5

StartServers 30MinSpareServers 30MaxSpareServers 60ServerLimit 60MaxClients 60MaxRequestsPerChild 0


PHP


PHP

• PHP 5.4


PHP

• PHP 5.4• Zend OPCache (3GB Memory Segment)


PHP

• PHP 5.4• Zend OPCache (3GB Memory Segment)• memory_limit: 128M


PHP

• PHP 5.4• Zend OPCache (3GB Memory Segment)• memory_limit: 128M• max_execution_time: 30


Optimizing PHP

http://www.slideshare.net/jnklein/northeast-php-high-performance-php




Optimizing PHP

• See last year’s talk





Optimizing PHP

• See last year’s talk• Use an Opcode Cache





Optimizing PHP


• xhprof





Optimizing PHP


• xhprof

• StatsD/Graphite





Optimizing PHP


• xhprof

• StatsD/Graphite

• Find hotspots





Optimizing PHP


• xhprof

• StatsD/Graphite

• Find hotspots

• Upgrade it





Static Arrays


Translationsclass language_de {

static $translations = array( "<hash>" => array("content" => 'Accessoires', "file" => "TAXONOMY"), "<hash>" => array("content" => 'Keramik', "file" => "CATEGORY"), "<hash>" => array("content" => 'Dekorieren', "file" => "CATEGORY"), "<hash>" => array("content" => 'Getaggt %s', "file" => "Foo.php"), "<hash>" => array("content" => 'Badezusatz', "file" => "CATEGORY"), "<hash>" => array("content" => 'Datum:{% $date %}', "file" => "bar.tpl")

... snip ...

);}


Atomic Deploys


Deploying PHP

http://codeascraft.com/2013/07/01/atomic-deploys-at-etsy/




Deploying PHP

• mod_realdoc - Apache module





Deploying PHP

• mod_realdoc - Apache module• incpath - PHP Extension





Deploying PHP

• mod_realdoc - Apache module• incpath - PHP Extension• A/B symlink swap





Deploying PHP

• mod_realdoc - Apache module• incpath - PHP Extension• A/B symlink swap• Avoid recompilation to opcodes





Upgrading PHP


PHP 5.4 vs. 5.3 - CPU

PHP 5.3

PHP 5.4


PHP 5.4 vs. 5.3 - Memory

PHP 5.3

PHP 5.4


Understand Framework

Overhead


http://systemsarchitect.net/performance-benchmark-of-popular-php-frameworks/




Frameworks are Harder to Scale


Memcached


Memcached

• Roughly same hardware as webs


Memcached

• Roughly same hardware as webs• More memory (48GB)


Memcached

• Roughly same hardware as webs• More memory (48GB)• Shard keys across servers


Memcached

• Roughly same hardware as webs• More memory (48GB)• Shard keys across servers• mctop - https://github.com/etsy/mctop


https://github.com/etsy/mctop

https://github.com/etsy/mctop

MySQL


MySQL

• MySQL 5.5 (http://jkle.in/nephp)


http://jkle.in/nephp


MySQL

• MySQL 5.5 (http://jkle.in/nephp)• ~20 master/master pairs




MySQL

• MySQL 5.5 (http://jkle.in/nephp)• ~20 master/master pairs• innodb




MySQL

• MySQL 5.5 (http://jkle.in/nephp)• ~20 master/master pairs• innodb• thread_cache=800




MySQL

• MySQL 5.5 (http://jkle.in/nephp)• ~20 master/master pairs• innodb• thread_cache=800• max_connections=2500




MySQL

• MySQL 5.5 (http://jkle.in/nephp)• ~20 master/master pairs• innodb• thread_cache=800• max_connections=2500• [Almost] no joins




Theoretically we can serve ~9000 req/sec


Static Content


~500 Million Images


Offloading Requests


Offloading Requests

• Multiple CDNs


Offloading Requests

• Multiple CDNs• Img hit rate > 90%


Offloading Requests

• Multiple CDNs• Img hit rate > 90%• CSS/JS hit rate ~100%


Offloading Requests

• Multiple CDNs• Img hit rate > 90%• CSS/JS hit rate ~100%• Edge serves ~7000 reqs/sec


What about misses?


Change Management


Feature Flags


// Owner: Username$server_config['foo']['bar'] = array( 'users' => array('jsmith', 'jdoe'),);

// Owner: Username$server_config['baz'] = [ 'enabled' => [ 'variant1' => 0, 'variant2' => 0, 'variant3' => 50 ],];

// Owner: Username$server_config['qux'] = array( 'enabled' => 100, 'data' => array( 'somedata' ));


if (Feature::isEnabled('foo.bar')) { // Code}


Ramp-up:1%, 5%, 25%, 100%


Deployinator


Schemanator


Recent Improvements


Sandy Bridge


Baseline Performance (median)


Small code change



Small code change

PHP 5.4



Small code change

PHP 5.4

Disabled Hyperthreading



Challenges


Options


Options

• Scale Horizontally (more shards)


Options

• Scale Horizontally (more shards)• Scale Vertically (SSD, bigger drives)


Options

• Scale Horizontally (more shards)• Scale Vertically (SSD, bigger drives)• Change architecture


Architecture Changes



• Ongoing conversation



• Ongoing conversation• Separate logical/physical shards



• Ongoing conversation• Separate logical/physical shards• Complex



• Ongoing conversation• Separate logical/physical shards• Complex• Server failure




• Migrating data




• Migrating data

• Consistency


Monitoring


Graphite


Stacked Search Timers


Ganglia


Nagios


Takeaways


K.I.S.S.


Use Proven Technologies


Understand Your Stack


Measure Everything


Work at Etsy ;-)


Connecthttp://www.meetup.com/Web-Performance-Boston/

www.etsy.com/careers

[email protected]

@jonathanklein


http://www.meetup.com/Web-Performance-Boston/

http://www.meetup.com/Web-Performance-Boston/

http://www.etsy.com/careers

http://www.etsy.com/careers

mailto:[email protected]

mailto:[email protected]

scaling php to 40 million uniques

Technology

php php

php http

php mod

scaling php

lines of php

uniques northeast php

apache apache

etsy stats