Using CloudFront with S3at SMARTSTUDY

Amazon Web Service Korean User GroupPark Hyun-woo, a SCV of SMARTSTUDY / ez@smartstudy.co.kr

What is S3 and CF?

● S3 : Simple Storage Service○ Extreme durability.○ Multiple copies in selected region.

■ vs EBS : Multiple copies in single available zone.

● CF : CloudFront○ Content Delivery Network(CDN) service.○ Distribute from origin to AWS Edge Network

■ http://aws.amazon.com/cloudfront/#details■ About 30 edges in global. (only in Earth, lol)

Why choose these?

● We serve large multimedia contents to global market in mobile application era.

● We need redundant, fast and reasonable price content delivery service aims globally.

● We have not enough human resource to maintain various system.

Flash Movie

S3 Bucket( tokyo )

s3cmd

Package( encrypted )

Thumbnails

Video

npk

Transcoder

Gnash

FFmpeg

qtfaststart

Flash

Movie

CloudFront{

Edge Edge Edge Edge Edge

S3 Bucket( tokyo )

s3cmd

Package( encrypted )

Thumbnails

Video

Transcoder

Gnash

FFmpeg

qtfaststart

Flash

Movie

npk

How it works?

● Copy to S3○ ~100 files, just use Cyberduck or other GUI tool.○ Or, you have to use s3fs or s3cmd.○ If you need sync or invalidation, use s3cmd.○ We have ~30,000 files. Thus, s3cmd.○ We chose Tokyo bucket for faster uploading.

● s3cmd○ Awesome!○ Works on S3 and CloudFront.

How it works? (cont.)

● s3cmd○ Can do almost everything as like as AWS console.○ Support INI-style configuration file.○ And, magical 'sync' command. (yeah!)

● s3cmd sync○ Support glob-style exclude / include option.○ ... rexclude / rinclude for regular expression.○ Support automatic invalidation request.○ ... with rsync-like 'dry-run' option. (-n, --dry-run)

● s3cmd sync○ Basically, sync based on MD5 information.○ (!) Over 15MB files, using only filesize.○ Our 30,000 files, 60GB = 8 min to dry-run.

( env : i7 2500K + SATA HDD to bucket in tokyo )

● s3cmd sync --cf-invalidate○ Call invalidation API automatically.○ (!) Simultaneous invalidation is up to 3.○ (!) Free for ~1,000 files per month.○ Check it up by s3cmd cfinvalinfo

How it works? (cont.)

So, is it really?

FASTREDUNDANTNOT EXPENSIVE

{

It's fast enough.

8.0MB/s

It's very redundant.

BETTER THAN YOUR OWN

SERVER, EVER

And

NOT EXPENSIVE

* without transfer fee for deploying origin bucket to edges

Bill for

CloudFrontBased on

Total traffic

● 10TB/month

● $0.209/GB

= $2,090*

Bill for

Domestic CDNBased on

Peak traffic

● 350Mbps/@peak

● $6.00/Mbps

= $2,100

STILL EXPENSIVE?

* Photo from Linkedin page of Lee, Jung-in / Amazon Korea

For over 10TB,

RESERVEDCapacity Pricing

● Smaller○ Encode media with optimal options.○ Use dedicated option for each platform.○ Storage is cheaper than transfer fee!

● Static○ Do not deploy volatile objects to CloudFront.○ Invalidation needs ~10min, often.○ We're still using Nginx for them.

● Streamable○ Mobile networks are still slow.○ Do not make users to wait until finishing download.

Tips

● Gnash○ http://www.gnu.org/software/gnash/

● FFmpeg○ http://www.ffmpeg.org/

● qtfaststart○ https://github.com/danielgtaylor/qtfaststart

● npk○ https://github.com/lqez/npk

● s3cmd○ http://s3tools.org/

Link

Using CloudFront with S3at SMARTSTUDY

Amazon Web Service Korean User GroupPark Hyun-woo, a SCV of SMARTSTUDY / ez@smartstudy.co.kr