scale your application while improving performance and lowering costs (svc203) | aws re:invent 2013

© 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.

Cache is King: Scale Your Application while Improving

Performance and Lowering Costs

November 15, 2013

Cash is King Cash is King

Cache is Cash

Httparchive

Content Breakdown

87% of Your Site Consists Of Static/Re-usable Content

56% of The Internet’s Content is NOT Cached Today

But Wait…

Cache is King

Source: httparchive.org

300K Tracked Websites Oct 15 2012-Oct 1 2013

~56% {

CACHE EVERYTHING

Browser Cache

Edge Cache

Web Cache

Application Cache

Database Cache

Image Source: cakeormistake.wordpress.com

http://cakeormistake.wordpress.com/2012/01/10/top-ten-2012-cake-trends/

Speed of Light ~300,000 Km/s

Image Source: wordlesstech.com

http://wordlesstech.com/2011/09/24/cern-scientists-break-the-speed-of-light/

~200,000 Km/s through fiber

Image Source: www2.wiringandcabling.com

http://www2.wiringandcabling.com/wiring-cabling-companies-in-cochran-fiber-optics-cat5-cat-6-and-structured-cabling-quotes.php

Download Performance

Round Trip: 13,026 km



Theoretical: 65 ms



Theoretical: 65 ms

Measured: 68 ms

Time to First Byte

68 ms

Client Server

Time to First Byte

68 ms

68 ms

Client Server

SYN

SYN/ACK

Time to First Byte

68 ms

68 ms

68 ms

Client Server 136 ms

SYN

SYN/ACK

ACK,

GET /image.jpg HTTP/1.1

Browser Cache

Browser Caching

Set max-age or expiry date in the headers.

HTML5 Application Cache.

Helps eliminate network latency.

But… Browser cache size is limited. (e.g. IE is 8-50M, Chrome is < 80M, Firefox is 50MB, etc.)

Browser Cache

Edge Cache

Time to First Byte

68 ms

68 ms

68 ms

Client Server 136 ms

SYN

SYN/ACK

ACK,


Bring the Bytes Closer to Your Users

68 ms

Client Origin

SYN

10 ms

SYN/ACK

GET /image.jpg HTTP/1.1 ACK,


CloudFront

Saves 68 ms

RTT

SYN

SYN/ACK

Time to first byte: 20 ms vs. 136 ms

Amazon CloudFront: 46 Global Locations

=

Edge Cache: Amazon CloudFront

Amazon S3

Elastic Load

Balancing

Mobile Clients

Browsers/Desktop

Clients

Tablets/Devices

Amazon CloudFront

Edge Location

Edge Cache

How do you decide what to cache?

• Static or Re-Usable Content

• Customized Content

• On-Demand and Live Video

• Dynamic or Unique Content

Cache Static or Re-Usable Content

HTTP/1.0 200 OK

Date: Mon, 19 Mar 201 12:51:28 GMT

Server: Apache

Last-Modified: Mon, 19 Mar 2012 07:15:25 GMT

Accept-Ranges: bytes

Content-Length: 1918

Cache-Control: max-age=86400

Content-Type: image/jpeg Vary: Accept-Encoding

Age: 16

X-Cache: Hit from cloudfront

Connection: keep-alive

Cache Customized Content

Customized Content

Cache Video Content

Caching Your Live Stream?

Source

HTTP

Cache

Edge

Paris

New York

Encoder

HLS, HDS,

Smooth Streaming

Hong Kong

Dynamic Content?

Zero TTL – cannot be cached! CloudFront can still help…

Dynamic Content?

Zero TTL – cannot be cached! CloudFront can still help…

TCP/IP optimizations for the network path

Keep-Alive Connections to reduce RTT

SSL Termination close to viewers

POST/PUT upload optimizations

Latency Based Routing

Low prices, same as static content delivery!

“We are excited to use CloudFront's new

POST, PUT, PATCH, and DELETE capabilities

to accelerate our RESTful APIs on Amazon

EC2. With these new HTTP methods we can

now take advantage of CloudFront’s global

footprint and optimized connections back to

our origin servers in AWS. Routing our

customers’ API requests via a CloudFront edge

location near them will help improve their

experience by minimizing packet loss and

upload latency. This will help provide a

streamlined experience for our customers.”

- Ilan Rabinovitch, Tech Lead, Site Reliability Engineering

Browser Cache

Edge Cache

Web Cache

You can’t have dessert,

until you have your dinner!

-An Experiment. https://secure.flickr.com/photos/stephen-oung/6319155216/

Experiment Premise

• Start with basic infrastructure. – Single m1.large EC2 instance running Amazon Linux

– Single non-Multi-AZ m1.xlarge RDS MySQL 5.6 instance

– Apache httpd-2.2.25

– PHP 5.3.27

– Drupal Commerce Kickstart 7.2

– EIP on Instance

• Throw a ton of traffic at it. – 8,000,000 queries, 40 per second, from 4 other instances.

• Profile it. – New Relic PHP agent

– webpagetest.org

Experiment Infrastructure

Web/App

Instance

Elastic IP Amazon RDS

DB Instance Clients

Results? (not pretty)

14.7 second responses

Mostly Static Data

Low RPM

Web Cache

Webserver or proxy caches would live between

your CDN/Users and your web tier and can offer

up increased cost performance via reducing

internal application and database load. Can also

offer up increased edge to origin speed for lots of

content.

Web Cache

Popular solutions: – Varnish

– Nginx

– Apache with mod_cache/mod_proxy

– Squid

– Perlbal

– Language/framework caches (i.e., APC, Zend)

Web Cache

Availability Zone

VPC Subnet VPC Subnet

Elastic Load

Balancing Cache

Instances

Web/App

Instances

Elastic Load

Balancing

Availability Zone


Elastic Load

Balancing Cache

Instances

Web/App

Instances

Elastic Load

Balancing

Internet

Gateway

Amazon

Route 53

Amazon

CloudFront

Customer

Traffic

Web Cache

• Opt for in-memory caching when possible.

• Pay attention to your cache hit/miss ratios. It could be a sign that you need to re-size the instances or re-size the number of nodes in your cache pool.

• Set smart TTLs so that you don’t affect new deploys or cache content for too long.

• Be smart about what cookies can burst cache and what cookies can’t. Don’t serve up other people’s content or stale dynamic pages.

Web Cache

How do you decide what to cache? – All logged out user pages

– Any completely static pages

– Traffic/log analysis

• Look at your web logs/CDN logs

• Find heavily hit pages

• Figure out how often they actually change

• Apply a TTL to that page to be cached

– Even 60 second TTLs could help drastically!

Browser Cache

Edge Cache

Web Cache

Application Cache

Application Cache

Application level caches for information such

as session data, temporary application data

such as cart information, and live aggregation

of data feeds.

Application Cache

Popular solutions: – Memcached

– Redis

– Cassandra

– Amazon DynamoDB

Application Cache

Availability Zone

VPC Subnet VPC Subnet VPC Subnet

Elastic Load

Balancing Cache

Instances

Web/App

Instances

Elastic Load

Balancing ElastiCache

Cache Node

Availability Zone


Elastic Load

Balancing Cache

Instances

Web/App

Instances

Elastic Load


Cache Node

Internet

Gateway

Amazon DynamoDB Session Handler for PHP

https://aws.amazon.com/sdkforphp/

Application Cache

Internet

Gateway

Availability Zone


Elastic Load

Balancing Cache

Instances

Web/App

Instances

Elastic Load

Balancing Squid Proxy

Instances Amazon

DynamoDB

Availability Zone


Elastic Load

Balancing Cache

Instances

Web/App

Instances

Elastic Load

Balancing Squid Proxy

Instances

Application Cache

Use Cases: – Session information

– Temporary data

• Cart info, metadata

– Rate limiting

• Fight abuse of APIs, spamming, functionality abuse

– Counters

• Views, Scores, Leader Boards

Application Cache

How do you decide what to cache?

– Ideally, you have to treat data you cache at this tier as

loss tolerant if working with in-memory caches.

– Session information that wouldn’t make sense in

cookies or in a true DB.

– Look at the kind of data your application is generating

and storing in a DB that it might not need to.

Browser Cache

Edge Cache

Web Cache

Application Cache

Database Cache

Database Cache

Reduce workload on database servers by

caching commonly requested information, or

any information that might not change

frequently (i.e., user info, listing info, product

info).

Database Cache

Popular solutions: – In-engine query caches

– Memcached

• On dedicated host

• On DB host (built in w/ MySQL 5.6)

– Redis

• On dedicated host

Database Cache

A word of caution: In-engine DB caches are often not recommended for

many use cases, as they can significantly impact the

performance of many databases. Depending on the

workload and dataset you have, an in-engine query

cache might not be a good idea for you. We recommend

off DB caches where possible.

Database Cache

Availability Zone


Elastic Load

Balancing Cache

Instances

Web/App

Instances

Elastic Load


Cache Node

Availability Zone


Elastic Load

Balancing Cache

Instances

Web/App

Instances

Elastic Load


Cache Node

Internet

Gateway

RDS Instance

Primary (M-AZ)

RDS Instance

Standby (MAZ)

Memcached – Code Sample function retrieveValue($query)

{

$queryId = md5($query);

if ($myValue = $memcache->get($queryId))

{

return $myValue;

}

else

{

$myValue = dbfetch($query);

$memcache->set($queryId, $myValue);

return $myValue;

}

}

MySQL 5.6 + Memcached

RDS MySQL supports version

5.6 with integrated Memcached

on the instance: – Part of the InnoDB engine

– Memcached running as part of MySQL

talks directly to data in InnoDB tables,

essentially turning MySQL into a fast

“key-value store”

– From the opposite view point, adds

persistence to Memcached

– Same Memcached API as standalone

https://dev.mysql.com/doc/refman/5.6/en/innodb-memcached-intro.html

AWS Marketplace & Partners Can Help

• Customers can find, research, buy

software.

• Simple pricing, aligns with Amazon

EC2 usage model.

• Launch in minutes!

• Marketplace billing integrated into your

AWS account.

• 1100+ products across 24+ categories.

Learn more at: aws.amazon.com/marketplace

Back To Our

Experiment!

https://secure.flickr.com/photos/toyochin/1382531438/

Experiment Infrastructure (original)

Web/App

Instance

Elastic IP Amazon RDS

DB Instance Clients

Results? (not pretty)

14.7 second responses

Mostly Static Data

Low RPM

Experiment Infrastructure (with cake)

• Added in ElastiCache Memcached cache.m1.large

• Added in Amazon CloudFront for static content – Tuned expires and cache headers in Apache

• Added in APC for PHP caching – Increased memory to 128Mb, no other changes

• Did nothing to the DB

• Drupal memcached & CDN modules

CloudFront

Experiment Infrastructure (with cake)

Web/App

Instance

Elastic IP

Amazon RDS

DB Instance

Clients

ElastiCache

Cache Node

Results? (AWESOME!)

response time ~1/2

~2x RPM

Repeat Page view from 15.8sec to 2.75sec

Easy as Pie!

https://secure.flickr.com/photos/linecon0/2654865842/

Toronto Star

• Canada’s largest daily newspaper

• Focused on metro Toronto

• 3.3 million monthly unique visitors

• Small in-house digital group,

supported by vendors & corporate IT

• Digital group run as “startup” within

corporate structure

• Tech stack includes Java, PHP,

Ruby, Python

mystar

Wheels.ca

Toronto.com

Thegridto.com

Why Cache?

• Performance

• Scalability

• Reliability

• Cost

Breaking News

Breaking News: Client View

Breaking News: Server View

The Caching Onion

DB

App

Reverse proxy

Edge

Edge – The Onion Skin

• Examples: Akamai,

CloudFront

• Static Asset caching

• Full site caching

• Origins

• Behaviors

• thestar.com, wheels.ca,

toronto.com, thegridto.com

Edge


Production - US East Region

Elastic Load Balancer

Dispatcher/ApacheAmazon Linux 64 Bit

C1 Medium


C1 Medium

Publish CQ/CRXAmazon Linux 64 Bit

C1-Xlarge

Master

Author CQ/CRXAmazon Linux 64 Bit

C1-Xlarge


C1 Medium

Availability Zone A Availability Zone DAvailability Zone B

Cloudfront

VPC

Standby


C1-Xlarge


C1-Xlarge


C1-Xlarge


C1-Xlarge


C1-Xlarge


C1-Xlarge

Thestar.com

architecture

Production - US East Region

Elastic Load Balancer


C1 Medium


C1 Medium


C1-Xlarge

Master


C1-Xlarge


C1 Medium

Availability Zone A Availability Zone DAvailability Zone B

Cloudfront

VPC

Standby


C1-Xlarge


C1-Xlarge


C1-Xlarge


C1-Xlarge


C1-Xlarge


C1-Xlarge

Whole Site Delivery

• Cache everything possible

• No server side cookies written,

only select pages pass query

strings

• Control caching granularly using

19 different rules

• We use a single origin, but Elastic

Load Balancing and multi-tiered

multi-AZ configuration on backend

• Planning a multi-region DR

architecture that will also leverage

Amazon CloudFront

Reverse Proxy / Web Accelerator

• Examples: Varnish, Nginx, Apache

• Serves “static” content

• Reduce load on app server

• wheels.ca, mystar, thestar.com, thegridto.com

Reverse proxy


mystar


Wheels.ca

App-level caching

• Examples:

Memcached, Reddis – ElastiCache now supports

both

• Session caching

• View caching

• mystar

App

App-level caching

Database caching

• Example:

Memcached

• Cache frequent SQL

queries

• Reduce DB server

load, response time

• mystar, wheels.ca

DB

Database Caching

Wheels.ca

Thanks!

Ping me

[email protected]

@stephenaevans

mailto:[email protected]

Please give us your feedback on this

presentation

As a thank you, we will select prize

winners daily for completed surveys!

SVC203

APPENDIX


Varnish Cache is a web application accelerator also

known as a caching HTTP reverse proxy. You install it

in front of any server that speaks HTTP and configure

it to cache the contents. Varnish Cache is really, really

fast. It typically speeds up delivery with a factor of 300

- 1000x, depending on your architecture.

- varnish-cache.org


Nginx (pronounced "engine x") is an open

source reverse proxy server for HTTP,

HTTPS, SMTP, POP3, and IMAP protocols,

as well as a load balancer, HTTP cache, and

a web server (origin server). The nginx

project started with a strong focus on high

concurrency, high performance and low

memory usage.

- Wikipedia

App-level caching Memcached

Free & open source, high-performance, distributed memory object caching system, generic in nature, but intended for use in speeding up dynamic web applications by alleviating database load.

Memcached is an in-memory key-value store for small chunks of arbitrary data (strings, objects) from results of database calls, API calls, or page rendering

App-level caching

Redis

Redis is an open source, BSD licensed, advanced

key-value store. It is often referred to as a data

structure server since keys can contain strings,

hashes, lists, sets and sorted sets. – Redis.io


“A content delivery network is a large distributed

system of servers deployed in multiple data

centers across the Internet. The goal of a CDN is

to serve content to end-users with high availability

and high performance.” - Wikipedia

http://en.wikipedia.org/wiki/Server_(computing)

http://en.wikipedia.org/wiki/Data_centers

http://en.wikipedia.org/wiki/Data_centers

http://en.wikipedia.org/wiki/Internet


“A web accelerator is a proxy server that

reduces web site access times. They can be

a self-contained hardware appliance or

installable software.” - Wikipedia

scale your application while improving performance and lowering costs (svc203) | aws re:invent 2013

Technology

cache static

cache video content

web cache webserver

browser cache size

html5 application cache

web cache popular solutions

dynamic content

unique content