jeff barr senior web services evangelist jbarr@amazon
DESCRIPTION
Amazon Web Services: Building Highly Scalable Web Applications Institutional Web Management Workshop July 2007. Jeff Barr Senior Web Services Evangelist [email protected]. Today. Introduction Amazon Web Services overview Amazon EC2 Amazon S3 Amazon SQS Application Architecture Questions. - PowerPoint PPT PresentationTRANSCRIPT
Amazon Web Services:Building Highly Scalable Web Applications
Institutional Web Management WorkshopJuly 2007
Jeff BarrSenior Web Services [email protected]
Today
IntroductionAmazon Web Services overviewAmazon EC2Amazon S3Amazon SQSApplication ArchitectureQuestions
Who am I?
Software development background5 Years with Amazon:
Senior Developer Web Services Evangelist Senior Web Services Evangelist
Veteran of several startupsMicrosoft .Net TeamVisix Software co-founder
Amazon?
Consumers (Buy)
Associates, Sellers & Merchants (Sell)
Developers (Innovate)
Issues Facing Developers
70% of Web Development Effort is “Muck”: Data Centers Bandwidth / Power / Cooling Operations Staffing
Scaling is Difficult and Expensive: Large Up-Front Investment Invest Ahead of Demand Load is Unpredictable
Dream or Nightmare?
Slashdot/Digg/TechCrunch EffectRapid, unexpected customer demand/growth
Seasonal Spikes
More Seasonal Spikes
“Every year, we take the busiest minute of the busiest hour of the busiest day and build capacity on that, we built our systems to (handle that load) and we went above and beyond that.” *
-- Scott Gulbransen Intuit Spokesman
* http://news.com.com/2100-1038_3-6177341.html
Solution: Web-Scale Computing
Scale capacity on demandTurn fixed costs into variable costsAlways availableRock-solid reliabilitySimple APIs and conceptual modelsCost-effectiveReduced time to marketFocus on product & core competencies
10
Amazon Web Services Are...
A set of APIs and business models which give developer-level access to Amazon’s infrastructure and content:
Data As A ServiceAmazon E-Commerce ServiceAmazon Historical Pricing
Search As A ServiceAlexa Web Information ServiceAlexa Top SitesAlexa Site ThumbnailAlexa Web Search Platform
Infrastructure As A ServiceAmazon Simple Queue ServiceAmazon Simple Storage ServiceAmazon Elastic Compute Cloud
People As A ServiceAmazon Mechanical Turk
The Commercial Side
Standard licensing termsCommercially usableAggressive pricingMonthly credit card billingSelf-serve model:
Sign up as developerChoose services Agree to service licensesEnter payment infoStart coding
Web Services Billing
Infrastructure Services
Compute
Store Message
Elastic ComputeCloud
Simple Storage Service
Simple QueueService
Amazon Simple Storage Service
S3
Amazon Simple Storage ServiceAmazon Simple Storage Service
$.15 per GB per month
storage
• Object-Based Storage• 1 B – 5 GB / object• Fast, Reliable, Scalable• Redundant, Dispersed• 99.99% Availability Goal• Private or Public• Per-object URLs & ACLs• BitTorrent Support $.10 - $.18 per
GB data transfer
$.01 for 1000 to 10000 requests
Amazon S3 Concepts
Objects:Opaque data to be stored (1 byte … 5 Gigabytes)Authentication and access controls
Buckets:Object container – any number of objects100 buckets per account / buckets are “owned”
Keys:Unique object identifier within bucketUp to 1024 bytes longFlat object storage model
Standards-Based Interfaces:REST and SOAPURL-Addressability – every object has a URL
S3 SOAP/Query APIService:
ListAllMyBuckets
Buckets:CreateBucketDeleteBucketListBucketGetBucketAccessControlPolicySetBucketAccessControlPolicyGetBucketLoggingStatusSetBucketLoggingStatus
Objects:PutObjectPutObjectInlineGetObjectGetObjectExtendedDeleteObjectGetObjectAccessControlPolicySetObjectAccessControlPolicy
Amazon Simple Queue Service
SQS
Amazon Simple Queue ServiceAmazon Simple Queue Service
$.10 per 1000 messages
• Scalable Queuing• Elastic Capacity• Reliable, Simple, Secure
Inter-process messaging, data buffering, architecture component
$.10 - $.18 per GB data transfer
Amazon SQS Concepts
Queues:Named message containerPersistent
Messages:Up to 256KB of data per messagePeek / Lock access model
Scalable:Unlimited number of queues per accountUnlimited number of messages per queue
Amazon SQS At Work
SQS SOAP/Query APIQueues:
ListQueues DeleteQueueSetVisibilityTimeoutGetVisibilityTimeout
Messages: SendMessage ReceiveMessage DeleteMessage PeekMessage
Security:AddGrantListGrantsRemoveGrant
Amazon Elastic Compute Cloud
EC2
Amazon Elastic Compute CloudAmazon Elastic Compute Cloud
$.10 per server hour
• Virtual Compute Cloud• Elastic Capacity• 1.7 GHz x86• 1.7 GB RAM• 160 GB Disk• 250 MB/Second Network• Network Security Model
Time or Traffic-based Scaling, Load testing, Simulation and Analysis, Rendering, Software as a Service Platform, Hosting
$.10 - $.18 per GB data transfer
Amazon EC2 Concepts
Amazon Machine Image (AMI):Bootable root diskPre-defined or user-builtCatalog of user-built AMIsOS: Fedora, Centos, Gentoo, Debian, Ubuntu, Windows ServerApp Stack: LAMP, mpiBLAST, Hadoop
Instance:Running copy of an AMILaunch in less than 2 minutesStart/stop programmatically
Network Security Model:Explicit access controlSecurity groups
Inter-service bandwidth is free
Amazon EC2 At Work
StartupsCruxy – Media transcodingGigaVox Media – Podcast Management
Fortune 500 clients:High-Impact, S hort-Term ProjectsDevelopment Host
Science / Research:Hadoop / MapReducempiBLAST
Load-Management and Load Balancing Tools:Pound WeogeoRightscale
EC2 SOAP/Query APIImages:
RegisterImageDescribeImagesDeregisterImage
Instances:RunInstancesDescribeInstancesTerminateInstancesGetConsoleOutputRebootInstances
Keypairs:CreateKeyPairDescribeKeyPairsDeleteKeyPair
Image Attributes:ModifyImageAttributeDescribeImageAttributeResetImageAttribute
Security Groups:CreateSecurityGroupDescribeSecurityGroupsDeleteSecurityGroupAuthorizeSecurityGroupIngressRevokeSecurityGroupIngress
Sample Web-Scale Architecture
GigaVox Media
Web-Scale Architecture
GigaVox Economics
Implemented Amazon S3, Amazon EC2 and Amazon SQS in November 2006
Created an infinitely scalable infrastructure for less than $100 - building the same infrastructure themselves would have cost thousands of dollars
Reduced staffing requirements - far less responsibility for 24x7 operations
Q & A
Thank You!PointersPortalhttp://aws.amazon.com
Bloghttp://aws.typepad.com
EC2http://aws.amazon.com/ec2
S3http://aws.amazon.com/s3
Resource Centerhttp://aws.amazon.com/resources
Forumshttp://aws.amazon.com/forums
Jeff BarrSenior Web Services Evangelist