cloud computing in php with the amazon web services
DESCRIPTION
Cloud Computing in PHP With the Amazon Web Services. Jeff Barr, Senior Web Services Evangelist [email protected] @ jeffbarr on Twitter. Introduction. Based in Seattle Career path: Startups Microsoft Consultant to VCs and startups Amazon Web Services Lead AWS Blogger - PowerPoint PPT PresentationTRANSCRIPT
CLOUD COMPUTING IN PHP WITH THE AMAZON WEB SERVICES
Jeff Barr, Senior Web Services [email protected]@jeffbarr on Twitter
• Based in Seattle• Career path:
– Startups– Microsoft– Consultant to VCs and startups– Amazon Web Services
• Lead AWS Blogger• Author of
“Host Your Website in the Cloud”
INTRODUCTION
• Amazon / Cloud / AWS Intro• AWS Overview
– Programmable Infrastructure– S3, EC2, SDB, SQS
• Programming AWS in PHP– AWS SDK for PHP– Scalable Image Processing– Visualizing Infrastructure– Dynamic Infrastructure
• Wrapup / Q & A
SESSION OUTLINE
AMAZON’S THREE BUSINESSES
Consumer (Retail)Business
Tens of millions of active customer accounts
Seven countries: US, UK, Germany, Japan, France, Canada, China
SellerBusiness
Sell on Amazon websites
Use Amazon technology for your own retail website
Leverage Amazon’s massive fulfillment center network
Developers &IT Professionals
On-demand infrastructure for hosting web-scale solutions
Hundreds of thousands of registered customers
WHAT IS CLOUD COMPUTING?
First, think of your electricity service…
Power is available to you on-demand, you pay only for what you use…
…and you plug into a vast electrical grid managed by professionals to get you the lowest cost, most reliable power with much greater efficiency and safety than you could probably do on your own.
INTRODUCING AMAZON WEB SERVICES
AWS provides flexible, scalable, secure, and cost-effective IT infrastructure for businesses of all sizes around the world.
Compute power and storage is available to you on-demand, you pay only for the resources you use…
…running on scalable, reliable, and secure infrastructure operated by Amazon Web Services, based on the knowledge gleaned from over a decade of building efficient and dependable infrastructure for Amazon.com.
AMAZON WEB SERVICES
ComputeAmazon Elastic Compute Cloud (EC2)
- Elastic Load Balancing- Auto Scaling
StorageAmazon Simple Storage Service (S3)
- AWS Import/Export
Application
Content DeliveryAmazon
CloudFront
MessagingAmazon Simple Queue
Service (SQS)Amazon Simple Notification
Service (SNS)
PaymentsAmazon Flexible Payments Service
(FPS)
On-Demand Workforce
Amazon Mechanical Turk
Parallel Processing
Amazon Elastic MapReduce
MonitoringAmazon CloudWatch
DatabaseAmazon RDS
Amazon SimpleDBThird-Party Offerings
ManagementAWS Management
Console
ToolsAWS Toolkit for Eclipse
Java, PHP, Ruby, Python, .Net Developer
Centers
Isolated NetworkVirtual Private Cloud
Metering and Billing
Identity and Access Management
• All functionality accessed by APIs• Amazon and third-party libraries• Command-line tools• AWS Management Console• Third-party Tools
PROGRAMMABLE INFRASTRUCTURE
CODE TO LAUNCH AN EC2 INSTANCE
// Run an instance$EC2 = new AmazonEC2();
$Options = array('KeyName' => "Jeff's Keys", 'InstanceType' => "m1.small");
$Res = $EC2->run_instances("ami-db7b9db2", 1, 1, $Options);
• Scalable data storage in-the-cloud• Highly available and durable• Pay-as-you-go pricing:
– Storage: tiered $0.15/GB to $0.055/GB– Data Transfer Out: tiered $0.15/GB to $0.080/GB– Data Transfer In: Free until June 30, 2010– Requests: nominal charges
• Big and busy:– 152 billion objects– 150K requests/second
AMAZON SIMPLE STORAGE SERVICE (S3)
Prices and facts are current as of October 2010.
• Amazon EC2: on-demand compute power– Obtain and boot new server instances in minutes– Quickly scale capacity up or down
• Key features:– Support for Linux, Windows, and OpenSolaris– Supports all major web and application platforms– Deploy across Availability Zones for reliability– Elastic IPs provide greater flexibility– Persistent storage with Amazon Elastic Block Store– Monitoring (CloudWatch), Load Balancing, Auto-Scaling
AMAZON ELASTIC COMPUTE CLOUD (EC2)
EC2 PROGRAMMING INTERFACE (API)
• Images:– RegisterImage– DescribeImages– DeregisterImage– ModifyImageAttribute– DescribeImageAttribute– ResetImageAttribute
• Instances:– RunInstances– DescribeInstances– TerminateInstances– StopInstances– GetConsoleOutput– RebootInstances
• IP Addresses:– AllocateAddress– ReleaseAddress– AssociateAddress– DisassociateAddress– DescribeAddresses
• Keypairs:– CreateKeyPair– DescribeKeyPairs– DeleteKeyPair
• Security Groups:– CreateSecurityGroup– DescribeSecurityGroups– DeleteSecurityGroup– AuthorizeSecurityGroupIngress– RevokeSecurityGroupIngress
• Block Storage Volumes:– CreateVolume– DeleteVolume– DescribeVolumes– AttachVolume– DetachVolume– CreateSnapshot– DescribeSnapshots– DeleteSnapshot
• VPC:– CreateCustomerGateway– DeleteCustomerGateway– DescribeCustomerGateways– AssociateDhcpOptions– CreateDhcpOptions– DeleteDhcpOptions– DescribeDhcpOptions– CreateSubnet– DeleteSubnet– DescribeSubnets– CreateVpc– DeleteVpc– DescribeVpcs– CreateVpnConnection– DeleteVpnConnection– DescribeVpnConnections– AttachVpnGateway– CreateVpnGateway– DeleteVpnGateway– DescribeVpnGateways– DetachVpnGateway
AMAZON EC2 INSTANCE SPECS
Standard High-CPU High-Memory Cluster
Micro Small Large Extra Large
Medium
Extra Large
Extra Large
Double Extra Large
Quadruple Extra Large
Cluster Compute
Quadruple Extra Large
Bits 32 32 64 64 32 64 64 64 64 64
RAM 613 MB 1.7 GB 7.5 GB
15 GB 1.7 GB 7 GB 17.1 GB
34.2 GB 68.4 GB 23
Disk 0 160 GB 850 GB
1690 GB
350 GB 1690 GB 420 GB
850 GB 1690 GB 1690 GB
EC2 Compute Units
Burst to 2
1 4 8 5 20 6.5 13 26 33.5
Cores 1 1 2 4 2 8 2 4 8 8(Dual
Processor)
Firewall Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes
Linux Per Hour
$0.02 $0.085 $0.34 $0.68 $0.17 $0.68 $0.50 $1.20 $2.40 $1.60
Windows Per Hour
$0.03 $0.12 $0.48 $0.96 $0.29 $1.16 $0.62 $1.44 $2.88 N/A
Prices and facts are current as of October 2010.
• Simple, scalable storage solution for structured data– Provides core database functionality for data storage and
querying– No schema, no data modeling, no DBA– SQL queries
AMAZON SIMPLEDB
item description color material
123 Sweater Blue, Red
789 Shoes Black Leather
Store:PUT (item, 123), (description, Sweater), (color, Blue), (color, Red)
Query:SELECT * FROM Inventory WHERE material='Leather'
• Reliable, highly scalable, hosted queue for messaging• Build automated workflows for all applications• Coordinate multiple Amazon EC2 instances
AMAZON SIMPLE QUEUE SERVICE
Queue
Producer
Producer
Producer
Consumer
Consumer
GETTING STARTED WITH AWS
CreateDeveloper Account
Enter PaymentInformation
Sign Up for Desired Services
Retrieve Private and Public Keys
Build & Deploy Application
Monitor and Scale Application
MOVING RIGHT ALONG!
"Enough with the talk, show us some
code, Jeff!"
PROGRAMMING AWS WITH PHP
We’ll use the free AWS SDK for PHP libraries from http://aws.amazon.com/sdkforphp
SDK BASICS
Download and unpack Add directory to PHP’s include_path Add AWS keys to config.inc.php
define('AWS_KEY', 'J35NTGFCQOIUY3OMNSQQ');
define('AWS_SECRET_KEY', '99pizu2vVOK11rk9UAgWVj7PBGzWwertqJlgLV0c');
Include one file:
require_once('sdk.class.php');
SDK XML / PHP SIMPLEXML
• All methods return a SimpleXML Object:
[body] => SimpleXMLElement Object ( [Name] => sitepoint-aws-cloud-book [Prefix] => SimpleXMLElement Object () [Marker] => SimpleXMLElement Object () [MaxKeys] => 1000 [IsTruncated] => false [Contents] => Array ( [0] => SimpleXMLElement Object ( [Key] =>
images/2008_shiller_housing_projection.jpg
• Access as $Res->body->Contents[0]->Key
EXAMPLE 1 – SCALABLE PROCESSING PIPELINE
BUILDING A SCALABLE IMAGE PROCESSING PIPELINE
• Fetch a web page, store in Amazon S3• Parse page and extract links to images• Fetch first 16 images on page, store in Amazon S3• Render images as composite image
URL Queue
Fetch & Store Page
Parse Queue
Parse Page
Image Queue
Fetch Images
S3
Render Queue
Render Images &
PagesS3S3
SQS-BASED APPLICATION ARCHITECTURE
ARCHITECTURE ATTRIBUTES
• Simple– Each stage is easy to understand– AWS reduces low-level coding
• Scalable– Add more processes – Add more instances– Add more stages– Storage any amount of data
• Fault Tolerant– Messages remain in queues until processed – Messages reappear in queues if process dies
• Asynchronous– Each stage runs at its own speed– Build, test,run one stage at a time
DEFINE QUEUES
// Queuesdefine('URL_QUEUE', 'c_url');define('PARSE_QUEUE', 'c_parse');define('IMAGE_QUEUE', 'c_image');define('RENDER_QUEUE', 'c_render');define('FEED_QUEUE', 'c_feed');
CREATE QUEUES
// Create the SQS access object$SQS = new AmazonSQS();
for ($i = 1; $i < count($argv); $i++){ $Queue = $argv[$i];
$Res = $SQS->create_queue($Queue);
if ($Res->isOK()) { print("Created queue '${Queue}'\n"); }}
CREATE QUEUES & CHECK STATUS
$ ./create_queues.php c_url c_parse c_image c_render c_feed
$ ./crawl_queue_status.phpc_url c_parse c_image c_render----- ------- ------- --------0 0 0 0
LOCATE A QUEUE'S URL
// Create the SQS access object$SQS = new AmazonSQS();
// Locate the queue$QueueURL = FindQueueURL($SQS, URL_QUEUE);
POST MESSAGE TO QUEUE
// Create message $HistItem = array('Posted by ' . $argv[0] . ' at ' .
date('c'));
$Message = json_encode(array('Action' => 'FetchPage', 'Origin' => $argv[0], 'Data' => $argv[$i], 'History' => $HistItem)); // Post message $Res = $SQS->send_message($QueueURL, $Message);
if ($Res->isOK()) { print("Posted '${Message}' to QueueURL '${QueueURL}'\n"); }
RECEIVE MESSAGE FROM QUEUE
function PullMessage($SQS, $QueueURL){ while (true) { $Res = $SQS->receive_message($QueueURL); if ($Res->isOk()) { if (IsSet($Res->body->ReceiveMessageResult->Message)) { } else { sleep(1); } }}
RETURN RECEIVED MESSAGE
$Message = $Res->body->ReceiveMessageResult->Message; $MessageBody = $Message->Body; $MessageDetail = json_decode($MessageBody, true); $ReceiptHandle = $Message->ReceiptHandle;
return array('QueueURL' => $QueueURL, 'Timestamp' => date('c'), 'Message' => $Message, 'MessageBody' => $MessageBody, 'MessageDetail' => $MessageDetail, 'ReceiptHandle' => $ReceiptHandle);
SAMPLE PROCESSING STAGE
// Pull, process, postwhile (true){ // Pull the message from the queue $Message = PullMessage($SQS, $QueueURL);
if ($Message != null) { // Extract message detail $MessageDetail = $Message['MessageDetail']; $ReceiptHandle = $Message['ReceiptHandle']; $PageURL = $MessageDetail['Data']; … }}
DETAILED PROCESSING
// Fetch the pageprint("Processing URL '${PageURL}':\n");$HTML = file_get_contents($PageURL);print(" Retrieved " . strlen($HTML) . " bytes of HTML\n");
// Store the page in S3 $Key = 'page_' . md5($PageURL) . '.html'; if (UploadObject($S3, BOOK_BUCKET, $Key, $HTML,
S3_ACL_PUBLIC)) { // Get URL in S3 $S3URL = $S3->get_object_url(BOOK_BUCKET, $Key); print(" Uploaded page to S3 as '${Key}'\n");
DETAILED PROCESSING
// Form message to pass page along to parser $Origin = $MessageDetail['Origin']; $History = $MessageDetail['History']; $History[] = 'Fetched by ' . $argv[0] . ' at ' .
date('c');
$Message = json_encode(array('Action' => 'ParsePage', 'Origin' => $Origin, 'Data' => $S3URL, 'PageURL' => $PageURL, 'History' => $History));
// Pass the page along to the parser $Res = $SQS->send_message($QueueParse, $Message);
RENDER IMAGES
foreach ($ImageKeys as $ImageKey) { // Fetch the image print(" Fetch image '${ImageKey}'\n"); $Image = $S3->get_object(BOOK_BUCKET, $ImageKey);
// Convert it to GD format $ImageBits = ImageCreateFromString($Image->body);
// Copy it to proper spot in the destination print(" Render image at ${NextX}, ${NextY}\n"); ImageCopy($ImageOut, $ImageBits, $NextX, $NextY, 0, 0, ImageSx($ImageBits), ImageSy($ImageBits));
// Update position for next image $NextX += THUMB_SIZE + GAP_SIZE; if (($NextX + THUMB_SIZE) > $OutX) { $NextX = BORDER_LEFT; $NextY += THUMB_SIZE + GAP_SIZE; } }
EXAMPLE 2 – CREATE INFRASTRUCTURE GRAPH
PROCESSING FLOW
Get AWS Metadata
Build Object Model
Render Object Model as DOT
Graph
Render Dot Graph as PDF
GET AWS METADATA
// Create the service access objects$Service_EC2 = new AmazonEC2);$Service_S3 = new AmazonS3);$Service_SDB = new AmazonSDB);$Service_CF = new AmazonCloudFrontetKey);
// Fetch information about all of the EC2 objects$ResElasticIP = $Service_EC2->describe_addresses();$ResAvailabilityZones = $Service_EC2-
>describe_availability_zones();$ResInstances = $Service_EC2->describe_instances();$ResVolumes = $Service_EC2->describe_volumes();$ResSnapshots = $Service_EC2->describe_snapshots();
BUILD OBJECT MODEL
// EC2 Instancesforeach ($ResInstances->body->reservationSet->item as $ItemSet){ foreach ($ItemSet->instancesSet->item as $Item) { $InstanceId = (string) $Item->instanceId; $ImageId = (string) $Item->imageId; $State = (string) $Item->instanceState->name; $InstanceType = (string) $Item->instanceType; $AvailabilityZone = (string) $Item->placement->availabilityZone; $LaunchTime = (string) $Item->launchTime;
$Region->AddInstance(new Instance($AvailabilityZone, $InstanceId, $ImageId, $State, $InstanceType, $LaunchTime));
}}
SAMPLE OBJECT (EC2 INSTANCE)
class Instance{ var $State; var $ImageId; var $InstanceId; var $LaunchTime; var $InstanceType; var $AvailabilityZone;
function __construct(…) {} public function GetAvailabilityZone() {} private function GetLabel() {} function Render() {} function RenderEdges() {}}
RENDER OBJECT MODEL AS DOT GRAPH
function WriteNode($FP, $IndentLevel, $Node, $Label, $NodeType)
{ fwrite($FP, Indent($IndentLevel) . Quote($Node) . ' [' . 'label=' . Quote($Label) . ', ' . GetNodeStyle($NodeType) . "];\n");}
DOT TEXT
graph aws{ // Region subgraph "cluster_us-east-1" { label="Region us-east-1"; style=filled; fillcolor=yellow; color=black;
// Elastic IP Addresses subgraph "cluster_us-east-1_ips" { label="Elastic IP Addresses"; style=filled; fillcolor=darksalmon; color=black;
"75.101.154.199" [label="Elastic IP 75.101.154.199\nec2-75-101-154-199.compute-1.amazonaws.com", color=black, style=filled, fillcolor=wheat, shape=rect];
"us-east-1_phantom_ips" [label="", shape=point, style=invis]; }
RENDER DOT TEXT TO PDF
$ dot –Tpdf aws_meta.dot > ~jeff/public_html/aws_meta.pdf
FINISHED INFRASTRUCTURE GRAPH
EXAMPLE 3 – DYNAMICALLY INSTANTIATE INFRASTRUCTURE
EC2 Instance
10 GB EBS
Volume
100 GB EBS
Volume
Elastic IP
Address
RUN AN EC2 INSTANCE
// Create the EC2 access object$EC2 = new AmazonEC2(AWS_PUBLIC_KEY, AWS_SECRET_KEY);
// Run an instance$Options = array('KeyName' => "Jeff's Keys", 'InstanceType' => "m1.small");
$Res = $EC2->run_instances("ami-db7b9db2", 1, 1, $Options);
GET INSTANCE INFO
// Get the Id and Availability Zone of the instance$Instances = $Res->body->instancesSet;$InstanceId = $Instances->item->instanceId;$AvailabilityZone = $Instances->item->placement
->availabilityZone;
print("Launched instance ${InstanceId} " . "in availability zone ${AvailabilityZone}.\n");
ALLOCATE ELASTIC IP ADDRESS
// Allocate an Elastic IP address$Res = $EC2->allocate_address();if (!$Res->isOK()){ exit("Could not allocate public IP address.\n");}
// Get the allocated Elastic IP address$PublicIP = $Res->body->publicIp;print("Assigned IP address ${PublicIP}.\n");
ATTACH IP ADDRESS TO INSTANCE
// Associate the Elastic IP address with the instance$Res = $EC2->associate_address($InstanceId, $PublicIP);
if (!$Res->IsOK()){ exit("Could not associate IP address ${PublicIP} " . "with instance ${InstanceId}.\n");}
print("Associated IP address ${PublicIP} " . "with instance ${InstanceId}.\n");
CREATE 2 EBS VIRTUAL DISK VOLUMES
// Create two EBS volumes in the instance's availability zone
$Res1 = $EC2->create_volume(10, $AvailabilityZone);$Res2 = $EC2->create_volume(100, $AvailabilityZone);
if (!$Res1->isOK() || !$Res2->isOK()){ exit("Could not create EBS volumes.\n");}
// Get the volume Ids$VolumeId1 = $Res1->body->volumeId;$VolumeId2 = $Res2->body->volumeId;
ATTACH THE VOLUMES TO THE INSTANCE
$Res1 = $EC2->attach_volume($VolumeId1, $InstanceId, '/dev/sdf');
$Res2 = $EC2->attach_volume($VolumeId2, $InstanceId, '/dev/sdg');
if (!$Res1->isOK() || !$Res2->isOK()){ exit("Could not attach EBS volumes " . "${VolumeId1} and ${VolumeId2} " . "to instance ${InstanceId}.\n");}
Q & A
• Amazon Web Services: http://aws.amazon.com
• AWS Blog: http://aws.typepad.com
• twitter.com/jeffbarr
FOR MORE INFORMATION
Thanks!