aws re:invent 2016: case study: how startups like mapbox, ring, hudl, and others use aws to punch...

© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

November 29, 2016

Case Study: How Startups like Mapbox,

Ring, Hudl, and Veristor Use AWS to

Punch Above Their Weight

STG204

Maps

Directions Geocoding

Mobile

Developer tools

Analysis

Serving maps at scale on AWS

• Built from the ground up using AWS since 2007

• Powers over 5,000 apps in categories ranging from social

to mobility

• Reaches more than 200 million users each month and

growing

Amazon S3 is our storage workhorse

Processing

EC2s

Amazon

S3

Mobile Apps

Satellites

Drones

Vehicles

Any Sensor

Mobile AppsAPI EC2s Amazon

CloudFront

AWS KMS + S3

• Encrypts data on Amazon S3 using a hardware encryption

device

• Data is automatically encrypted as it's written and

decrypted as it's read

Massively scalable

• Over 2 petabytes of street and satellite data stored in S3

• 70 billion requests to S3 each month

• Burstable - temporarily scale up during heavy data

processing and scale back down

• At times we've had 5 petabytes of data in S3 but can

scale that back to save money

Built-in cost saving features

• Infrequent access - save on data that you don't access

often - like raw data

• Reduced redundancy storage - save on data that you can

easily regenerate from raw data

• Combine these features for maximum cost savings

In short

• Integration with other AWS services

• Scalability

• Cost-saving features

Rob Hruska

Engineering Director

8+ petabytes

4,300,000,000+ objects

35 hours video/minute

145 k amateur + elite teams

Today

• Video Upload/Storage Pipeline

• Amazon S3 Transfer Acceleration

• Standard – Infrequent Access

Hudl Video Lifecycle

Browsers

Amazon

CloudFront

Hudl Uploader

Upload-Only

Bucket

Application

Video

Processing

Mobile Apps

Metadata / Object Locations (HTTP)

VideoAmazon

SQS

Permanent

Bucket

Amazon

CloudFront

Application

Video

Processing

Browsers

Mobile Apps

Metadata / Object Locations (HTTP)

Amazon

SQS

Permanent

Bucket

Hudl Uploader

Upload-Only

Bucket

Video

Hudl Video Lifecycle

Amazon

CloudFront

Application

Video

Processing

Browsers

Mobile Apps

Metadata / Object Locations (HTTP)

Amazon

SQS

Permanent

Bucket

Hudl Uploader

Upload-Only

Bucket

Video

Transfer

Acceleration

Hudl Video Lifecycle – Transfer Acceleration

Hudl Video Lifecycle

Browsers

Amazon

CloudFront

Hudl Uploader

Upload-Only

Bucket

Application

Video

Processing

Mobile Apps

Metadata / Object Locations (HTTP)

VideoAmazon

SQS

Permanent

Bucket

Browsers

Amazon

CloudFront

Hudl Uploader

Upload-Only

Bucket

Application

Video

Processing

Mobile Apps

Metadata / Object Locations (HTTP)

VideoAmazon

SQS

Permanent

Bucket

Hudl Video Lifecycle – Standard-IA

Finding an ideal Standard-IA transition lifecycle

Video

Views

Weeks After Created

Basketball Video Usage Over Lifetime

Standard-IA> 50% transitioned

Teams

N-N

Teams

N-N

Teams

N-N

Game

Video

Highlight

Video

Recruit

Video

Bucket Organization

Ring Products: Practical Uses for the IoT

Ring Neighborhoods: Network Effects in Practice

Wilshire Park study with LAPD:

Ring installed on 10% of homes

Burglaries down 55% for the

entire community in 6 months

Burglars want an easy hit and go

elsewhere if you’re home

Alarms are reactive, not proactive

Traditional systems don’t link up,

so protection ends at your door

Devices installed in nearly every country on Earth

Millions of connected apps and devices

Over 1 billion videos and rapidly increasing

High growth brings big challenges, even month-to-month

Ring Video Pipeline

Raw

Buckets

Final

(Standard)

S3 Logs

Amazon

CloudFront

Ring App(s)

AWS

Lambda

Viewers

Amazon

SQS

Owner(s)

Visitor

Application

Servers

Ring Device

GPU

Farm

Final

(IA)

Lifecycle

Transitions

Event

Triggers

Live Video

Intelligently Determining Class

Infrastructure Savings in Amazon S3

Global Concerns

Extreme Performance is Easy

S3 will automatically partition if you use good keys – or just add more buckets

Amazon CloudFront as a CDN for GET heavy loads and faster downloads

Faster uploads with Transfer Acceleration

TCP Window Scaling - without it, 64 KB window kneecaps long fat networks

TCP SACK is good for fast but lossy connections like mobile connections

examplebucket/2134857/data/start.png

examplebucket/2134857/data/resource.rsrc

examplebucket/2134857/data/results.txt

examplebucket/2134858/data/start.png

examplebucket/2134858/data/resource.rsrc

examplebucket/2134858/data/results.txt

examplebucket/2134859/data/start.png

examplebucket/2134859/data/resource.rsrc

examplebucket/2134859/data/results.txt

examplebucket/7584312/data/start.png

examplebucket/7584312/data/resource.rsrc

examplebucket/7584312/data/results.txt

examplebucket/8584312/data/start.png

examplebucket/8584312/data/resource.rsrc

examplebucket/8584312/data/results.txt

examplebucket/9584312/data/start.png

examplebucket/9584312/data/resource.rsrc

examplebucket/9584312/data/results.txt

S3 Scaling on H-Day

DevOps Software

Real-Time Splunk for JIRA Service Desk

Splunk Real-Time Search integration with JIRA

Service Desk for automated issue creation.

JIRA Service Desk links to Splunk Real-Time

search results for rapid troubleshooting.

Atlassian Cloud to Splunk

Forty8Fifty

Atlassian

Connect Add-on

AWS Benefits

Fault Tolerant

HA Infrastructure

Rapid AWS

Innovation

Supports Our

Innovation

Scalable for Growth Cost/Margin

Management

Thank you!

Remember to complete

your evaluations!