My Opera meets Varnishvarnish

high performance web cachingcosimo@opera.com

What is Varnish?

varnish

backends● Caching reverse proxy, like Squid● Delegates memory mgmt to OS cache● Mainly developed at Linpro in Oslo

Two typical Varnish setups

varnish

backends

frontends

incomingrequests

• man vcl• VCL is compiled to C code• Injected into the running instance, without restart• Must define a backend or a director• VCL gives you several hooks:vcl_recv()vcl_hash()vcl_fetch()vcl_hit()vcl_miss()vcl_deliver()

VCL - Varnish Config Language

• In production beginning of October 2009• 1 old recycled machine, 2 Gb of disk allocated• Started serving avatars

1M+ requests per day before Unitehttp://my.opera.com/<username>/avatar.pl

• Soon after, added Desktop Team RSS (very popular!)• then user pictures, hundreds of thousands req/day• then Unite/ASD API requests

- friends of a user- groups of a user

• In total, 13,25% of all My Opera requests are «varnished»• Around 7,2M req/day

Varnish deployment in My Opera

• Still using Debian Etch?First Varnish instance was running v1.x from Etch.several years old, not good

• Experienced VIPs– ”Very Interesting Problems”– User X getting User Y's session– Random users getting admin powers. Nightmare!

• Theory: Varnish was caching response bodies that containedSet-Cookie: opera_session=<session_id>

Varnish deployment in My OperaProblems /1

• There wasn't any obvious configuration problem.Same config worked with 2.0.x from Backports.

• v2.0.{4,5} is highly recommended!

Varnish deployment in My OperaProblems /2

• We tried caching the frontpage of My Opera, but had to revert the change due to too many different custom layouts for Opera Mobile, Mini, IE, Firefox, etc...

• Maybe using clever vcl_hash() tricks we can achieve that too.

Varnish deployment in My OperaProblems /3

My Opera configuration

• Backendsingle backend machine, or load-balanced virtual server

• Director– simple round-robin or random weighted “balancing” logic– has basic connection retries mechanism– has basic backend health check

• If you already have an LVS, define a single BackendOtherwise, go for the Director

Backends and Directors

Backends and Directors

Define a backend

# Only hit the upload serversbackend myopera { .host = "upload.my.opera.com"; .port = "80";}

Backends and Directors

Define a director

director myopera round-robin { .backend {

.host = "b1.opera.com";

.port = "80"; } .backend {

.host = "b2.opera.com";

.port = "80"; } ...}

Backends and Directors

...and then use them

sub vcl_recv { ... set req.backend = myopera; ...}

sub vcl_recv {

set req.backend = myopera; set req.grace = 3m;

# URL patterns based cache. # Avoid possible mixups. if(req.http.host !~ "^my\.opera\.com$") { pass; }

vcl_recv() / 1

if (req.url ~ "^/community/users/avatar\.pl/[0-9]+$" || req.url ~ "^/.+/avatar\.pl$" || req.url ~ "^/.+/picture\.pl\?xscale=100$" || req.url ~ "^/desktopteam/xml/atom/blog/?$" || req.url ~ "^/desktopteam/xml/rss/blog/?$" || req.url ~ "^/community/api/users/friends\.pl\?user=.+$" || req.url ~ "^/community/api/users/groups\.pl\?user=.+$") { unset req.http.Cookie; unset req.http.Authorization; lookup;}

vcl_recv() / 2

... # Check for cookie only after always-cache URLs if (req.http.Cookie ~ "(opera_session|opera_persistent_)") { pass; }

# DANGER, Will Robinson! Caching the front-page # At this point, lots of Google Analytics cookies will go in. # No problem. It's stuff used by Javascript if (req.url ~ "^/community/$") { lookup; }

pass;}

vcl_recv() / 3

sub vcl_fetch {

set obj.http.X-Varnish-URL = req.url; set obj.grace = 3m;

if (obj.http.Set-Cookie) { set obj.http.X-Varnish-Cacheable = "no, set-cookie"; pass; }

if (req.request != "GET") { set obj.http.X-Varnish-Cacheable = "no, !GET"; pass; }

vcl_fetch() / 1

if (req.http.host !~ "^my\.opera\.com$") { set obj.http.X-Varnish-Cacheable = "no, !my.opera.com"; pass;}

if (req.url ~ "^/community/users/avatar\.pl/[0-9]+$" || req.url ~ "^/[A-Za-z0-9]+/avatar\.pl$" || ... ) { unset obj.http.Set-Cookie; set obj.http.X-Varnish-Cacheable = "yes, url"; set obj.ttl = 24h; deliver;}

vcl_fetch() / 2

vcl_hash()sub vcl_hash { # Default Varnish behavior set req.hash += req.url; set req.hash += req.http.host; # Have a different cached frontpage per language if (req.url ~ "^/community/$") { set req.http.X-FrontPage-Language = regsub( req.http.Cookie, "^.*?language=([^;]*?);*.*$", "\1" ); set req.hash += "lang:"; set req.hash += req.http.X-FrontPage-Language; } hash;}

• Developed a testing tool (varnish-test)– outputs a TAP stream and some debug info– works best if varnish is specially tuned

• Can quickly check if a test/production instance is performing correctly or having problems

• Invoked as a simple script:varnis h-tes t --profile=tes ts .url --hos t=b1

Testing Varnishhow to avoid nightmares...

# Frontpag e

/ N O_C OOK IE S V AR N IS H_C AC HED

/ N O_C OOK IE S V AR N IS H_N OT_C AC H ED Hos t: my.c n.opera .c om

/ N O_C OOK IE S V AR N IS H_C AC HED C ookie:lang uag e=it

# B log s

/des ktopteam/blog / N O_C OOK IE S V AR N IS H_N OT_C AC HE D

# Avatars

/c ommunity/us ers /avatar/817271 N O_C OOK IE S V AR N IS H_C AC HE D

/c ommunity/us ers /avatar/442 N O_C OOK IE S V AR N IS H_C AC HE D

/g raphic s /avatar.g if N O_C OOK IE S V AR N IS H_N OT_C AC HE D

Testing Varnishcaching test list

• We can specify exactly how the varnish instance should behave.– Production acceptance tests– Test new varnish versions, new OS distributions– Fine tune config changes quickly with no impact on production

• Midway through there's a request that logs in as a test user.From then on, we can verify what resources are cached whena user is logged in. Some resources should be cached in any case.

Testing Varnishcaching test list

Testing Varnishsample run

...ok 289 - Got response from backend for /community/ (from ...) ok 290 - Correct status line# Adding header [Cookie] => [language=it]# ----------# GET http://cache01.my.opera.com:6081/community/# Host: my.opera.com# ------------ok 291 - 2nd request: got response from backend for /community/ (from...)ok 292 - Correct status line# X-Varnish: 1211283813 1211283812# X-Varnish-Status: hit# X-Varnish-Cacheable: yes, language cookie# X-Varnish-URL: /community/ok 293 - URL '/community/' was handled correctly by varnish# cookie_header:ok 294 - URL '/community/' has correct cookies (or no cookies)1..294

All tests successful.

X-Varnish: 1211283813 1211283812X-Varnish-Status: hitX-Varnish-Cacheable: yes, language cookieX-Varnish-URL: /community/

• varnishlog– Reads shared memory log info and displays it– Full instance log, on My Opera, 1 day is about 15 Gb– You can get an emulated Apache-style access.log from it

• varnishncsa– Displays requests to Varnish as Apache access logs– Can read from an archived log by varnishlog

• varnishstat– Displays realtime stats (hit ratio, space allocated, connections,...)

Monitoring Varnishbuilt-in tools

• Munin plugins– Hit ratio– Requests rate– Backend traffic

• Nagios plugins– Nothing special, TCP connection to port 6081

Monitoring Varnishexternal tools

Monitoring Varnish

Monitoring Varnish

Monitoring Varnish

• My Opera front page caching• My Opera files server?• Working on a prototype thumbnail server

Next steps

• Redpill-Linpro website– http://varnish.projects.linpro.no– Bug tracking, documentation and community support– Users and developers mailing lists

• Commercial support and training– http://www.varnish-cache.com

References and more information

• At Opera, there's several teams using Varnish in production• If you want to know more, contact me: cosimo@opera.com

Questions?