php tek 2008 : apc @ facebook
DESCRIPTION
TRANSCRIPT
Brian M. ShirePHP Tek 2008, Chicago
May 23rd 11:15am-12:15pm
About Facebook
A social utility to communicate & share information
5th most-trafficked website in the world. (source: comScore)
Over 70 million active users.
Customization.
Global collaboration is beneficial!
“Own the stack.”
Open Source
Facebook uses and modifies Open Source technology.....
Provides a significant PHP interpreter performance increase.
Open source allows collaboration, bug fixes and optimizations.
Alternative PHP Cache (APC)
“APC is a free, open, and robust framework for caching and optimizing PHP intermediate code.”
George SchlossnagleDaniel CowgillRasmus LerdorfGopal VijayaraghavanEdin Kadribasic
APC Developers:
Ilia AlshanetskyMarcus BörgerSara GolemonBrian Shire
Motivations for Optimization
http://www.flickr.com/photos/wwworks/2222523486
Save Energy!
Motivations for Optimization
Save Money!
http://www.flickr.com/photos/twon/1526777999
Motivations for Optimization
Better User Experience!
http://www.flickr.com/photos/dnorman/1539617436
Requests per second provide an overall measurement of performance gains.
Measurements made using Apache Bench (ab) with 1000 total requests, concurrency of 40.Executed on a dual Dual Core AMD Opteron 2.2Ghz, 8GB RAM.
The APC Advantage
APC User & File
APC User
PHP 2.4 rps
5.2 rps (x2)
22 rps (x9)
29.5 rps (x12)
Start Request
Compilation:
Convert PHP source
into opcodes
Execute opcodes
End Request
APC hook Opcode
Cache
StoreAPC hook
Miss
Hit
Fetch Opcodes
Start Request
Compilation:
Convert PHP source
into opcodes
Execute opcodes
End Request
PHP
Compilation With Opcode Caches
APC
1 <?php2 $output = ’Hello World’;3 if($_GET[‘exclaim’]) {4 $output .= ‘!’;5 } else {6 $output .= ‘.’;7 }8 echo $output;9 ?>
PHP Source
1 ASSIGN !0, ‘HELLO+WORLD’ 2 FETCH_R GLOBAL $1, ‘_GET’ 3 FETCH_DIM_R $2, $1, ‘exclaim’ 4 JMPZ $2, ->6 5 ASSIGN_CONCAT !0, ‘%21’ 6 JMP ->7 7 ASSIGN_CONCAT !0, ‘.’ 8 ECHO $0 9 RETURN 110 ZEND_HANDLE_EXCEPTION
Opcodes
Opcode output generated with the Vulcan Logic Dissassembler (VLD)by Derick Rethans (http://pecl.php.net/packages/VLD/)
From Code to Opcode
apc.php is a web based console.
Monitor apc.php
Located in apc source directory.
Monitor cache size, hit rate, and configuration.
For automated monitoring, use apc_cache_info() and apc_sma_info() with the $limited flag to reduce load.
Browsing of File and User caches.
Monitor apc.php to determine your memory usage.
Performs poorly when memory is at capacity.
Hints optimize hash lookup tables.
Maximum number of files or user entries.
Cache Size and Hints
apc.shm_segments = 1
Allocate extra memory for overhead.
Deleted items remain in the cache until no longer in use.
Facebook: 1.6GB in production, 6GB in development.
Facebook: over 3,000 files and 490,000 user variables.
apc.shm_size = 30
apc.num_files_hint = 1000
apc.user_entries_hint = 4096
Uses file locking operations
Faster alternative to file locks
Stable, not effecient
Architecture specific, Linux Kernel 2.6.x or later
Significant performance gain
Better alternative to Linux Futex Locks
Same performance gain
Stable, wide OS support
Ported from the PostgreSQL project
Runs in user space
Locking Mechanisms
File Locks
IPC Semaphore Locks
Pthread Mutex Locks
Linux Futex Locks
Spin Locks
Facebook: Spin Locks!
0
50
100
60 120 180
Fil
e
APC Locking Performance Comparison Under High Contention
systemuser
0
50
100
60 120 180
IPC
Sem
. systemuser
0
50
100
60 120 180
Fute
x
systemuser
0
50
100
60 120 180
Pth
read
systemuser
0
50
100
60 120 180
Spin
Time in Seconds
systemuser
APC Locking Performance Comparison Under High Contention
APC stats files to determine if they’ve been updated.Start Request
Compilation:
Convert PHP source
into opcodes
Execute opcodes
End Request
APC hook Opcode
Cache
StoreAPC hook
Miss
Hit
stat()Disabling updates will increase performance.
Requires restart or apc_cache_clear() to update.
CVS, SVN and rsync backdate modified times.
stat_ctime checks the creation time for updates.
To Stat or Not to Stat?
apc.stat = true
apc.stat_ctime = false
Facebook: apc.stat = FALSE!
Inspect System Calls Using strace
System calls are expensive!
Inspect them with strace:
> strace -e trace=file httpd -Xstat("/www/html/test.php", {st_mode=S_IFREG|0644, st_size=710, ...}) = 0getcwd("/usr/local/apache/bin", 4095) = 41chdir("/www/html") = 0open("/www/html/includes/1.inc", O_RDONLY) = 7stat("/www/html/includes/1.inc", {st_mode=S_IFREG|0644, st_size=16, ...}) = 0open("/www/html/includes/2.inc", O_RDONLY) = 8stat("/www/html/includes/2.inc", {st_mode=S_IFREG|0644, st_size=16, ...}) = 0open("/www/html/includes/3.inc", O_RDONLY) = 9stat("/www/html/includes/3.inc", {st_mode=S_IFREG|0644, st_size=16, ...}) = 0open("/www/html/includes/4.inc", O_RDONLY) = 10stat("/www/html/includes/4.inc", {st_mode=S_IFREG|0644, st_size=16, ...}) = 0open("/www/html/includes/5.inc", O_RDONLY) = 11stat("/www/html/includes/5.inc", {st_mode=S_IFREG|0644, st_size=16, ...}) = 0open("/www/html/includes/6.inc", O_RDONLY) = 12stat("/www/html/includes/6.inc", {st_mode=S_IFREG|0644, st_size=16, ...}) = 0open("/www/html/includes/7.inc", O_RDONLY) = 13stat("/www/html/includes/7.inc", {st_mode=S_IFREG|0644, st_size=16, ...}) = 0open("/www/html/includes/8.inc", O_RDONLY) = 14stat("/www/html/includes/8.inc", {st_mode=S_IFREG|0644, st_size=16, ...}) = 0open("/www/html/includes/9.inc", O_RDONLY) = 19stat("/www/html/includes/9.inc", {st_mode=S_IFREG|0644, st_size=16, ...}) = 0open("/www/html/includes/10.inc", O_RDONLY) = 20stat("/www/html/includes/10.inc", {st_mode=S_IFREG|0644, st_size=16, ...}) = 0chdir("/usr/local/apache/bin") = 0
Reducing System CPU
Example:
<?php
/* Include some really important libraries */include_once(‘/www/includes/1.inc’);include_once(‘/www/includes/2.inc’);include_once(‘/www/includes/3.inc’);include_once(‘/www/includes/4.inc’);include_once(‘/www/includes/5.inc’);include_once(‘/www/includes/6.inc’);include_once(‘/www/includes/7.inc’);include_once(‘/www/includes/8.inc’);include_once(‘/www/includes/9.inc’);include_once(‘/www/includes/10.inc’);
echo ‘ok’?>
stat("/www/html/test.php", {st_mode=S_IFREG|0644, st_size=710, ...}) = 0getcwd("/usr/local/apache/bin", 4095) = 41chdir("/www/html") = 0open("/www/html/includes/1.inc", O_RDONLY) = 7open("/www/html/includes/2.inc", O_RDONLY) = 8open("/www/html/includes/3.inc", O_RDONLY) = 9open("/www/html/includes/4.inc", O_RDONLY) = 10open("/www/html/includes/5.inc", O_RDONLY) = 11open("/www/html/includes/6.inc", O_RDONLY) = 12open("/www/html/includes/7.inc", O_RDONLY) = 13open("/www/html/includes/8.inc", O_RDONLY) = 14open("/www/html/includes/9.inc", O_RDONLY) = 19open("/www/html/includes/10.inc", O_RDONLY) = 20chdir("/usr/local/apache/bin") = 0
System Call Reduction With apc.stat=FALSE
stat("/www/html/test.php", {st_mode=S_IFREG|0644, st_size=710, ...}) = 0getcwd("/usr/local/apache/bin", 4095) = 41chdir("/www/html") = 0open("/www/html/includes/1.inc", O_RDONLY) = 7stat("/www/html/includes/1.inc", {st_mode=S_IFREG|0644, st_size=16, ...}) = 0open("/www/html/includes/2.inc", O_RDONLY) = 8stat("/www/html/includes/2.inc", {st_mode=S_IFREG|0644, st_size=16, ...}) = 0open("/www/html/includes/3.inc", O_RDONLY) = 9stat("/www/html/includes/3.inc", {st_mode=S_IFREG|0644, st_size=16, ...}) = 0open("/www/html/includes/4.inc", O_RDONLY) = 10stat("/www/html/includes/4.inc", {st_mode=S_IFREG|0644, st_size=16, ...}) = 0open("/www/html/includes/5.inc", O_RDONLY) = 11stat("/www/html/includes/5.inc", {st_mode=S_IFREG|0644, st_size=16, ...}) = 0open("/www/html/includes/6.inc", O_RDONLY) = 12stat("/www/html/includes/6.inc", {st_mode=S_IFREG|0644, st_size=16, ...}) = 0open("/www/html/includes/7.inc", O_RDONLY) = 13stat("/www/html/includes/7.inc", {st_mode=S_IFREG|0644, st_size=16, ...}) = 0open("/www/html/includes/8.inc", O_RDONLY) = 14stat("/www/html/includes/8.inc", {st_mode=S_IFREG|0644, st_size=16, ...}) = 0open("/www/html/includes/9.inc", O_RDONLY) = 19stat("/www/html/includes/9.inc", {st_mode=S_IFREG|0644, st_size=16, ...}) = 0open("/www/html/includes/10.inc", O_RDONLY) = 20stat("/www/html/includes/10.inc", {st_mode=S_IFREG|0644, st_size=16, ...}) = 0chdir("/usr/local/apache/bin") = 0
apc.stat = TRUE apc.stat = FALSE
Inline garbage collector removes entries from cache as soon as possible.
Spreads load on startup, only caches percent of requests.
The non-blocking write lock is preferred to setting slam_defense.
Read updated files after given delay to prevent loading incomplete files.
“Time to Live”
Entries still in use will be removed after the gc_ttl has expired.
Maximum time a cache entry can remain in cache.
Slam Defenses & TTL
apc.slam_defense = 0
apc.write_lock = 1
apc.file_update_protection = 2
apc.ttl = 0
apc.user_ttl = 0
apc.gc_ttl = 3600
Limits the maximum file size that will be cached.
A regular expression that excludes files from being cached.
Setting to zero causes files to only cache if they match apc.filters.
Filtering Files
apc.max_file_size = 1M
apc.filters = NULL
apc.cache_by_default = true
Monitor Upload Progress
Upload progress support
Update frequency in percent of file size or per byte count
Hidden form input name for apc user variable key
Prefix for the apc user variable key
<form> <input type=”hidden” name=”APC_UPLOAD_PROGRESS” value=”A86DCF0C”> ... </form>
Produces an APC user cache entry “upload_A86DCF0C” containing upload statistics.
The HTML Form:
apc.rfc1867 = On
apc.rfc1867_freq = 0
apc.rfc1867_name = APC_UPLOAD_PROGRESS
apc.rfc1867_prefix = upload_
API
array apc_cache_info ([ string $cache_type [, bool $limited ]] )array apc_sma_info ([ bool $limited ] )
bool apc_store ( string $key , mixed $var [, int $ttl ] )bool apc_add ( string $key , mixed $var [, int $ttl ] )
mixed apc_fetch ( string $key )
bool apc_delete ( string $key )bool apc_clear_cache ([ string $cache_type ] )
bool apc_compile_file ( string $filename )
Info. & Stats.
Insertion
Retrieval
Deletion
File Insertion
User cache stores PHP variables across multiples requests on a per server basis.
Primary commands for utilizing the user cache:
Optimize...Application configuration
Statistics such as site usage, request types, error conditions, timing
Nth tier cache in addition to memcache or other caching service
HTML or other output
Database backed values that only change rarely like product listings
Site behavior like new features, A/B tests, or rate controls
The User Variable Cache
bool apc_store ( string $key , mixed $var [, int $ttl ] )bool apc_add ( string $key , mixed $var [, int $ttl ] )
mixed apc_fetch ( string $key )
Insertion
Retrieval
Control site configuration and features via the user cache.
Datacenter 'B
web Servers:
Datacenter 'A'
Web Servers:
apc_sitevar.php
Spawns requests to multiple web servers
The site updates in the time it takes to make HTTP requests.
Updates are simple and fast for the Engineer.
Engineers Users
Controlling Site Behavior with "Sitevars"
Site-wide Server Variables: “Sitevars”
Facebook primes it’s cache before each server restart.
Ready to serve requests without delay due to compilation or updating cache values.
Handles immediate flood of requests with limited warm-up time.
Starts with same cache state limiting differences between servers.
Improve Startup with Cache Priming
Apache is restarted for major code pushes.
bool apc_store ( string $key , mixed $var [, int $ttl ] )
bool apc_compile_file ( string $filename )
Insertion
File Insertion
Serialize Cache Values
Stop Apache
Disallow connections via
iptables or load balancer
Deserialize and apc_store()
values.
Start Apache
apc_compile_file()
PHP source
Allow connections via
iptables or load balancer
Serialized User
Cache Values
Serialized User
Cache Values
Distributed to Web Servers
Master Restart Script
Single Control Server Multiple Web Servers
System
Apache
PHP & APC
Local Disk
APC Priming and Restart
APC Binary Dump (experimental patch)
Allows the abilty to copy the APC memory cache to disk, and reload at a later time.
Ability to load pre-compiled files into your cache.
Ability to load and run code without touching disk or keeping source code on file.
Does not currently support apc.stat=1 mode.
Only works between identical architectures (x86_64 vs. i386, little vs. big endian).
Works only between the same PHP versions.
Prime APC Cache and
apc_bindump_file("gzip://...")
Stop Apache
Disallow connections via
iptables or load balancer
apc_binload_file("gzip://...")
Start Apache
Allow connections via
iptables or load balancer
Binary File Binary FileDistributed to Web Servers
Master Restart Script
Single Control Server Multiple Web Servers
System
Apache
PHP & APC
Local Disk
APC Binary Priming and Restart
Lazy Loading
Significant time is spent copying opcodes from APC.
Only a small portion of the code is executed.
Delay copying until function is called.
Perceived as include() being slow.
Will soon be available as an experimental patch to APC and PHP.
include('library.php');
library_function();
exit();
<?php // Normal LoadingCopyALL
Functions
APCFunctions
ExecuteSingle
Function
include('library.php');
library_function();
exit();
<?php // Lazy Loading
APCFunctions
Copy Single
Function&
Execute
Installation via “pecl install apc”or source http://pecl.php.net/packages/APC/
Start with a basic apc.ini file:
apc.enabled=1apc.shm_size=100M
Install apc.php under the DocumentRoot, configure the USER and PASS variables.Monitor apc.php for usage and adjust configuration.
Try tuning some of the discussed settings to meet application needs.
Add some APC user variables.
Try spin locks.
Try the apc.stat=0 setting.
Measure changes in CPU usage and maximum requests per second.
Getting Started
Further Information...
I appreciate feedback/questions about APC, patches, this talk, and anything else! <[email protected]>
More implementation examples in Lucas Nealan's “Facebook Performance Caching”http://sizzo.org/wp/talks/
Slides, and patches will be available at http://tekrat.com
Acknowledgments
George SchlossnagleDaniel CowgillRasmus LerdorfGopal VijayaraghavanEdin Kadribasic
APC Developers:
Ilia AlshanetskyMarcus BörgerSara Golemon
Any Questions?
Q & A
Thank you!
