![Page 1: Surge 2010 - from disaster to stability - scaling my.opera.com](https://reader033.vdocuments.mx/reader033/viewer/2022051514/54830128b4af9f8a548b4953/html5/thumbnails/1.jpg)
from disaster to stabilitythe scaling challenges of my.opera.com
Surge 2010 – Version 3
![Page 2: Surge 2010 - from disaster to stability - scaling my.opera.com](https://reader033.vdocuments.mx/reader033/viewer/2022051514/54830128b4af9f8a548b4953/html5/thumbnails/2.jpg)
![Page 3: Surge 2010 - from disaster to stability - scaling my.opera.com](https://reader033.vdocuments.mx/reader033/viewer/2022051514/54830128b4af9f8a548b4953/html5/thumbnails/3.jpg)
2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 20101 10 50 257 205 430
8871,640
2,500
5,500ServerskUsers
1999
![Page 4: Surge 2010 - from disaster to stability - scaling my.opera.com](https://reader033.vdocuments.mx/reader033/viewer/2022051514/54830128b4af9f8a548b4953/html5/thumbnails/4.jpg)
2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 20101 10 50 257 205 430
8871,640
2,500
5,500ServerskUsers
2001
![Page 5: Surge 2010 - from disaster to stability - scaling my.opera.com](https://reader033.vdocuments.mx/reader033/viewer/2022051514/54830128b4af9f8a548b4953/html5/thumbnails/5.jpg)
2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 20101 10 50 257 205 430
8871,640
2,500
5,500ServerskUsers
2004
![Page 6: Surge 2010 - from disaster to stability - scaling my.opera.com](https://reader033.vdocuments.mx/reader033/viewer/2022051514/54830128b4af9f8a548b4953/html5/thumbnails/6.jpg)
2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 20101 10 50 257 205 430
8871,640
2,500
5,500ServerskUsers
2007
![Page 7: Surge 2010 - from disaster to stability - scaling my.opera.com](https://reader033.vdocuments.mx/reader033/viewer/2022051514/54830128b4af9f8a548b4953/html5/thumbnails/7.jpg)
2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 20101 10 50 257 205 430
8871,640
2,500
5,500ServerskUsers
2009
![Page 8: Surge 2010 - from disaster to stability - scaling my.opera.com](https://reader033.vdocuments.mx/reader033/viewer/2022051514/54830128b4af9f8a548b4953/html5/thumbnails/8.jpg)
the current beta
![Page 9: Surge 2010 - from disaster to stability - scaling my.opera.com](https://reader033.vdocuments.mx/reader033/viewer/2022051514/54830128b4af9f8a548b4953/html5/thumbnails/9.jpg)
the situation2007
![Page 10: Surge 2010 - from disaster to stability - scaling my.opera.com](https://reader033.vdocuments.mx/reader033/viewer/2022051514/54830128b4af9f8a548b4953/html5/thumbnails/10.jpg)
crashes every day
too many connections!!!
Team?
NFS volume of doom
![Page 11: Surge 2010 - from disaster to stability - scaling my.opera.com](https://reader033.vdocuments.mx/reader033/viewer/2022051514/54830128b4af9f8a548b4953/html5/thumbnails/11.jpg)
monitoring
![Page 12: Surge 2010 - from disaster to stability - scaling my.opera.com](https://reader033.vdocuments.mx/reader033/viewer/2022051514/54830128b4af9f8a548b4953/html5/thumbnails/12.jpg)
![Page 13: Surge 2010 - from disaster to stability - scaling my.opera.com](https://reader033.vdocuments.mx/reader033/viewer/2022051514/54830128b4af9f8a548b4953/html5/thumbnails/13.jpg)
![Page 14: Surge 2010 - from disaster to stability - scaling my.opera.com](https://reader033.vdocuments.mx/reader033/viewer/2022051514/54830128b4af9f8a548b4953/html5/thumbnails/14.jpg)
![Page 15: Surge 2010 - from disaster to stability - scaling my.opera.com](https://reader033.vdocuments.mx/reader033/viewer/2022051514/54830128b4af9f8a548b4953/html5/thumbnails/15.jpg)
![Page 16: Surge 2010 - from disaster to stability - scaling my.opera.com](https://reader033.vdocuments.mx/reader033/viewer/2022051514/54830128b4af9f8a548b4953/html5/thumbnails/16.jpg)
➔ Efficient filesystem cache
➔ "Dogpile effect" AKA stampeding AKA ...
➔ Persistent db + memcached connections
➔ Soft counters
➔ Profiling, profiling, …
many improvements since then
![Page 17: Surge 2010 - from disaster to stability - scaling my.opera.com](https://reader033.vdocuments.mx/reader033/viewer/2022051514/54830128b4af9f8a548b4953/html5/thumbnails/17.jpg)
code profiling[DML] time=1237308152, user=, url=/tinh_yeu_cua_anh_b88/blog/index.dml/tag/...,name=XWA::User, variable=active, type=module, elapsed=0.068473, host=my.opera.com[DML] time=1237308152, user=, url=/community/,name=XWA::User, variable=, type=module, elapsed=0.015935, host=my.opera.com[DML] ...
![Page 18: Surge 2010 - from disaster to stability - scaling my.opera.com](https://reader033.vdocuments.mx/reader033/viewer/2022051514/54830128b4af9f8a548b4953/html5/thumbnails/18.jpg)
top time-intensive modules
XWA::User::Sidebar 2024.919s (27.2%, 0.28 s/call)XWA::User 1778.445s (23.9%, 0.09 s/call)XWA::User::Journal 1121.224s (15.1%, 0.24 s/call)XWA::User::Album 321.522s ( 4.3%, 0.17 s/call)XWA::User::Journal::Search 223.477s ( 3.0%, 20.32 s/call)XWA::User::Comments 188.011s ( 2.5%, 0.05 s/call)XWA::Skins 180.486s ( 2.4%, 0.49 s/call)XWA::User::JournalArchive 159.525s ( 2.1%, 4.43 s/call)XWA::User::Posts 146.644s ( 2.0%, 0.45 s/call)XWA::User::Picture 141.324s ( 1.9%, 0.10 s/call)XWA::Albums 93.740s ( 1.3%, 2.04 s/call)XWA::Journals 92.390s ( 1.2%, 2.37 s/call)
![Page 19: Surge 2010 - from disaster to stability - scaling my.opera.com](https://reader033.vdocuments.mx/reader033/viewer/2022051514/54830128b4af9f8a548b4953/html5/thumbnails/19.jpg)
many improvements since then
➔ YSlow?
➔ The Expires header is your friend!
➔ Hot MyISAM tables converted to InnoDB
➔ MySQL Master/Master setup
➔ Jet Profiler
![Page 20: Surge 2010 - from disaster to stability - scaling my.opera.com](https://reader033.vdocuments.mx/reader033/viewer/2022051514/54830128b4af9f8a548b4953/html5/thumbnails/20.jpg)
![Page 21: Surge 2010 - from disaster to stability - scaling my.opera.com](https://reader033.vdocuments.mx/reader033/viewer/2022051514/54830128b4af9f8a548b4953/html5/thumbnails/21.jpg)
jet profiler
![Page 22: Surge 2010 - from disaster to stability - scaling my.opera.com](https://reader033.vdocuments.mx/reader033/viewer/2022051514/54830128b4af9f8a548b4953/html5/thumbnails/22.jpg)
scalability3
![Page 23: Surge 2010 - from disaster to stability - scaling my.opera.com](https://reader033.vdocuments.mx/reader033/viewer/2022051514/54830128b4af9f8a548b4953/html5/thumbnails/23.jpg)
1. avatars
![Page 24: Surge 2010 - from disaster to stability - scaling my.opera.com](https://reader033.vdocuments.mx/reader033/viewer/2022051514/54830128b4af9f8a548b4953/html5/thumbnails/24.jpg)
Avatars - 2007
75%/<user-name>/avatar.pl
/<user-name>/avatar.pl?xscale=8192 (!)
![Page 25: Surge 2010 - from disaster to stability - scaling my.opera.com](https://reader033.vdocuments.mx/reader033/viewer/2022051514/54830128b4af9f8a548b4953/html5/thumbnails/25.jpg)
my $sql = DBConnect('master');my %user = $sql->get( "SELECT a.blob, a.filename, FROM avatars a, users u WHERE u.user=? AND u.id=a.user", $user);$req->print( $user{'blob'} );
Avatars wtf!?
![Page 26: Surge 2010 - from disaster to stability - scaling my.opera.com](https://reader033.vdocuments.mx/reader033/viewer/2022051514/54830128b4af9f8a548b4953/html5/thumbnails/26.jpg)
Avatars - reloaded➔ Export to balanced fs (5 formats)
➔ Zero SQL queries
➔ Storage subsystem
➔ static.myopera.com was born
![Page 27: Surge 2010 - from disaster to stability - scaling my.opera.com](https://reader033.vdocuments.mx/reader033/viewer/2022051514/54830128b4af9f8a548b4953/html5/thumbnails/27.jpg)
resources(user uploads, binary blobs, ...)
Poolsor single servers
URLshttp://static.myopera.com/pool1/avatars/a4/754/a1b2c3d4e5f6.../<userid>_o.pnghttp://static.myopera.com/pool1/avatars/a4/754/a1b2c3d4e5f6.../<userid>_t.jpghttp://static.myopera.com/pool1/avatars/a4/754/a1b2c3d4e5f6.../<userid>_m.jpghttp://static.myopera.com/pool1/avatars/a4/754/a1b2c3d4e5f6.../<userid>_l.jpg
![Page 28: Surge 2010 - from disaster to stability - scaling my.opera.com](https://reader033.vdocuments.mx/reader033/viewer/2022051514/54830128b4af9f8a548b4953/html5/thumbnails/28.jpg)
+ x➔ Load
➔ Flexibility
➔ Static scales!
➔ HTTP::DAV
➔ Precomp URLs
![Page 29: Surge 2010 - from disaster to stability - scaling my.opera.com](https://reader033.vdocuments.mx/reader033/viewer/2022051514/54830128b4af9f8a548b4953/html5/thumbnails/29.jpg)
2. varnish
![Page 30: Surge 2010 - from disaster to stability - scaling my.opera.com](https://reader033.vdocuments.mx/reader033/viewer/2022051514/54830128b4af9f8a548b4953/html5/thumbnails/30.jpg)
![Page 31: Surge 2010 - from disaster to stability - scaling my.opera.com](https://reader033.vdocuments.mx/reader033/viewer/2022051514/54830128b4af9f8a548b4953/html5/thumbnails/31.jpg)
VarnishMost popular RSS feeds
My Opera frontpage
Opera Mini approval
Datacenter emergencies
![Page 32: Surge 2010 - from disaster to stability - scaling my.opera.com](https://reader033.vdocuments.mx/reader033/viewer/2022051514/54830128b4af9f8a548b4953/html5/thumbnails/32.jpg)
VarnishMost popular RSS feeds
➔ /desktopteam/blog/
➔ Friends, Groups API
➔ No cookies (remove req.http.cookie)
![Page 33: Surge 2010 - from disaster to stability - scaling my.opera.com](https://reader033.vdocuments.mx/reader033/viewer/2022051514/54830128b4af9f8a548b4953/html5/thumbnails/33.jpg)
VarnishMy Opera frontpage
➔ Danger, Will Robinson!
➔ Mangle cookies
➔ Accept-Language headers
![Page 34: Surge 2010 - from disaster to stability - scaling my.opera.com](https://reader033.vdocuments.mx/reader033/viewer/2022051514/54830128b4af9f8a548b4953/html5/thumbnails/34.jpg)
VarnishOpera Mini 5.0 approval
➔ Global coverage
➔ Traffic surge (5x peak, 2x over 24h)
![Page 35: Surge 2010 - from disaster to stability - scaling my.opera.com](https://reader033.vdocuments.mx/reader033/viewer/2022051514/54830128b4af9f8a548b4953/html5/thumbnails/35.jpg)
![Page 36: Surge 2010 - from disaster to stability - scaling my.opera.com](https://reader033.vdocuments.mx/reader033/viewer/2022051514/54830128b4af9f8a548b4953/html5/thumbnails/36.jpg)
IT NEEDS TO BE OUTTOMORROW
!!!
THERE WILL BE A
PRESS RELEASE !
![Page 37: Surge 2010 - from disaster to stability - scaling my.opera.com](https://reader033.vdocuments.mx/reader033/viewer/2022051514/54830128b4af9f8a548b4953/html5/thumbnails/37.jpg)
VarnishOpera Mini 5.0 approval
➔ Global coverage
➔ Traffic surge (5x peak, 2x over 24h)
➔ No problems!
![Page 38: Surge 2010 - from disaster to stability - scaling my.opera.com](https://reader033.vdocuments.mx/reader033/viewer/2022051514/54830128b4af9f8a548b4953/html5/thumbnails/38.jpg)
Opera Mini “countup” trafficSubmittedto Apple StoreMarch, 23rd
ApprovedApril, 12th
![Page 39: Surge 2010 - from disaster to stability - scaling my.opera.com](https://reader033.vdocuments.mx/reader033/viewer/2022051514/54830128b4af9f8a548b4953/html5/thumbnails/39.jpg)
VarnishDatacenter emergencies
![Page 40: Surge 2010 - from disaster to stability - scaling my.opera.com](https://reader033.vdocuments.mx/reader033/viewer/2022051514/54830128b4af9f8a548b4953/html5/thumbnails/40.jpg)
Datacenter emergencies
files.myopera.com
User Files Storage SAN
DC1
![Page 41: Surge 2010 - from disaster to stability - scaling my.opera.com](https://reader033.vdocuments.mx/reader033/viewer/2022051514/54830128b4af9f8a548b4953/html5/thumbnails/41.jpg)
Datacenter emergencies
files.myopera.com
User Files Storage SAN
DC1
DC2
LVS + Varnish servers
![Page 42: Surge 2010 - from disaster to stability - scaling my.opera.com](https://reader033.vdocuments.mx/reader033/viewer/2022051514/54830128b4af9f8a548b4953/html5/thumbnails/42.jpg)
~ 1Gbit/s! Varnish
![Page 43: Surge 2010 - from disaster to stability - scaling my.opera.com](https://reader033.vdocuments.mx/reader033/viewer/2022051514/54830128b4af9f8a548b4953/html5/thumbnails/43.jpg)
+ x➔ Load
➔ Flexibility
➔ Instant scaling
➔ Chainsaw!
➔ Purging
![Page 44: Surge 2010 - from disaster to stability - scaling my.opera.com](https://reader033.vdocuments.mx/reader033/viewer/2022051514/54830128b4af9f8a548b4953/html5/thumbnails/44.jpg)
3. geodns
![Page 45: Surge 2010 - from disaster to stability - scaling my.opera.com](https://reader033.vdocuments.mx/reader033/viewer/2022051514/54830128b4af9f8a548b4953/html5/thumbnails/45.jpg)
geodns
![Page 46: Surge 2010 - from disaster to stability - scaling my.opera.com](https://reader033.vdocuments.mx/reader033/viewer/2022051514/54830128b4af9f8a548b4953/html5/thumbnails/46.jpg)
+ x➔ Prototype 1 week
➔ Geo-scaling
➔ Redundant
➔ Accuracy
➔ No DC feedback
➔ Monitoring
![Page 47: Surge 2010 - from disaster to stability - scaling my.opera.com](https://reader033.vdocuments.mx/reader033/viewer/2022051514/54830128b4af9f8a548b4953/html5/thumbnails/47.jpg)
Next steps➔ Search (Solr?)
➔ Batch activity feed
➔ Real connection pooling
➔ … and on ...
![Page 48: Surge 2010 - from disaster to stability - scaling my.opera.com](https://reader033.vdocuments.mx/reader033/viewer/2022051514/54830128b4af9f8a548b4953/html5/thumbnails/48.jpg)
Remember!➔ Team spirit is important
➔ Another level of indirection...
➔ Keep it simple
➔ Keep a log
![Page 49: Surge 2010 - from disaster to stability - scaling my.opera.com](https://reader033.vdocuments.mx/reader033/viewer/2022051514/54830128b4af9f8a548b4953/html5/thumbnails/49.jpg)
the heroeshttp://my.opera.com/devblog/about/http://my.opera.com/devblog/
![Page 50: Surge 2010 - from disaster to stability - scaling my.opera.com](https://reader033.vdocuments.mx/reader033/viewer/2022051514/54830128b4af9f8a548b4953/html5/thumbnails/50.jpg)
any questions? ?
![Page 51: Surge 2010 - from disaster to stability - scaling my.opera.com](https://reader033.vdocuments.mx/reader033/viewer/2022051514/54830128b4af9f8a548b4953/html5/thumbnails/51.jpg)
handout download:
thanks!
http://tinyurl.com/surge2010-cosimo