Download - Cs264 intro-to-cloud-computing
CS264: Introduction to Cloud Computing
Justin RileySoftware Tools for Academics and ResearchersOffice of Educational Innovation and Technology
Massachusetts Institute of Technology
What is Cloud Computing Anyway?
Image Credit: http://tomlambert.com/cloud-computing-will-rule-the-world/
“Cloud computing” is a very fuzzy term in general
Often includes everything and the kitchen sink
Three broad categories:
Software as a Service (SaaS)Platform as a Service (PaaS)Infrastructure as a Service (IaaS)
Infrastructure as a Service (IaaS)
Hardware On Demand
Pay for what you use
Full root access – you control the OS and Software Stack
Ability to scale computing resources up and down
No dealing with racks, networks, power, cooling, housing, etc.
Amazon Simple Storage Solution (S3)
“... a simple web service interface that can be used to store and retrieve any amount of data, at any time, from anywhere on the web”
Read, write, and delete objects containing from 1 byte to 5 terabytes of data each.
Number of objects you can store is unlimited.
Each object stored in a 'bucket' and retrieved via a unique, user-assigned key
Amazon Elastic Compute Cloud (EC2)
Resizable Compute CapacityAs much as you need, when you need it. Scale up or down in minutes.
Complete Control via APICreate, scale, & manage instances programmatically.
Variety of Instance SizesCPU Power, Cores, RAM, Disk.
Wide Variety of Pre-built AMIs (Amazon Machine Images)Hit the ground running with minimal system building effort. Now: Linux, Windows, and OpenSolaris.
Secure & Flexible Network Security ModelFull control of access for each running instance.Keypair required for SSH access.
Amazon EC2 Instance Types
Micro Standard HighMemory
HighCPU
ClusterCompute
Cluster GPU
Micro Small Large Extra Large XL 2 XL 4 XL Medium Extra
Large 4 XL 4XL
Bits 32/64 32 64 64 64 64 64 32 64 64 64
RAM613 MB 1.7 GB 7.5
GB 15 GB 17.1 34.2 68.4 1.7 GB 7 GB 23 22
Disk420 GB 160 GB 850
GB 1690 GB 420 850 GB
1690 GB 350 GB 1690 GB 1690 GB 1690 GB
Virtual Cores 1 1 2 4 2 4 8 2 8 8
2 NVIDIA Tesla
“Fermi” GPUs
EC2 Compute Units
2 (Burst) 1 4 8 6.5 13 26 5 20 33.5 33.5
Firewall Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes
On-Demand Pricing
Linux Per Hour $0.02 $0.085 $0.34 $0.68 $0.50 $1.00 $2.00 $0.17 $0.68 1.60 2.10
Windows $0.03 $0.12 $0.48 $0.96 $0.62 $1.24 $2.48 $0.29 $1.16 N/A N/A
“Spot” Instances
Bid for unused AWS capacity
Prices controlled by AWS based on supply and demand
AWS can terminate Spot Instances without notice
Best approach to temporary requests for large numbers of servers
Default maximum = 100 servers (instead of 20 on-demand)
Amazon Machine Images (AMI)
AMIInstance
InstanceInstance
InstanceInstance
Contains an entire operating system and software stack that can be loaded onto one or more virtual machines
Amazon Elastic Block Storage (EBS) Persistent storage
Volume lifetime is independent of any particular EC2 instance.
General purposeRaw, unformatted, block device. Use from Linux, Solaris or Windows.
High performanceEqual to or better than local EC2 drive.
High reliabilityBuilt-in redundancy within availability zone.AFR (Annual Failure Rate) between 0.1% and 1%.
ScalableVolume sizes ranging from 1 GB to 1 TB. Easy to create, attach, back up, restore, and delete volumes.
Amazon Elastic Block Storage Pricing
EBS Volumes
$0.10 per GB-month of provisioned storage
$0.10 per 1 million I/O requests
No charge for mounting/unmounting volume
EBS Snapshots to Amazon S3
$0.14 per GB-month of data stored
$0.01 per 1,000 PUT requests (when saving a snapshot)
$0.01 per 10,000 GET requests (when loading a snapshot)
Amazon EC2 Regions and Availability Zones
EU West Region
Availability Zone A
Availability Zone B
US East Region
Availability Zone A
Availability Zone C
Availability Zone B
US West Region
Availability Zone A
Availability Zone B
Note: Conceptual drawing only. The number of Availability Zones may vary
Singapore
Availability Zone A
Availability Zone B
Notes on Using EBS Volumes
● EBS volumes can only be used with instances in the same availability zone they were created in
● Analogous to a virtual “pen drive”
● Can only attach a volume to one instance at a time.
Amazon Web Services Console
http://aws.amazon.com/console
Web-based management console for all AWS services
Elastic MapReduce
Easily launch Map/Reduce jobs on Amazon EC2
Uses Hadoop
Define Map/Reduce work flows either at command line or from AWS console
Mapper/Reducer code must be stored on S3
Input/output data stored on S3
Introducing StarCluster
Developed at MIT
Under active development
Open source
Web site: http://web.mit.edu/stardev/cluster/
Easy to install and use ($ easy_install starcluster)
Simplifies creation and management of EC2 clusters
Why StarCluster?
EC2 provides raw compute power
There’s work to be done to create a usable cluster:
Software installation
AMI creation
AWS / SSH key management and distribution
Persistent Disk Storage and File Sharing
Configuration management
Higher-level management (cluster vs. instance)
StarCluster FeaturesPrebuilt 32 and 64 bit AMIs
Launch a cluster of EC2 instances:
One command (“starcluster”) to rule them all
Passwordless SSH pre-configured
Security group for SSH access
Shared disk volume (NFS)
Preinstalled libraries (OpenMPI, NumPy, SciPy, etc.)
Easy to install, configure, and use
AWS Region
ClusterClusterCluster
StarCluster Architecture / Terminology
Client
EC2 or Desktop Running Linux
Master Disk
Node001
EC2
NodeN
EC2…
Config File
Master
EC2
Prerequisites
Client computer running Mac/Linux
AWS security credentials:
Access Key ID
Secret Access Key
Public Key (Keypair)
Cluster-aware application (something to run)
Edit .starcluster/config
AWS Credentials
Must match KEYNAME
Name and location of filedownloaded in last slide
Name of EC2 keypair
Additional Configuration Options
Cluster size
AMI for nodes
Node instance type
Master instance type
AMI for master
Start Cluster (Output - 1)
StarCluster - (http://web.mit.edu/starcluster)Software Tools for Academics and Researchers (STAR)Please submit bug reports to [email protected]
>>> Using default cluster template: smallcluster>>> Validating cluster template settings...>>> Cluster template settings are valid>>> Starting cluster...>>> Launching a 5-node cluster...>>> Launching master node...>>> Master AMI: ami-d1c42db8>>> Creating security group @sc-jb1...Reservation:r-edb9bd87>>> Launching worker nodes...>>> Node AMI: ami-d1c42db8Reservation:r-e1b9bd8b>>> Waiting for cluster to start...\
Start Cluster (Output - 2)
>>> Waiting for cluster to start...>>> The master node is ec2-50-16-41-160.compute-1.amazonaws.com>>> Setting up the cluster...>>> Using private key /home/ec2-user/keys-jbarr-us-east.pem (rsa)>>> Creating cluster user: sgeadmin>>> Using private key /home/ec2-user/keys-jbarr-us-east.pem (rsa)>>> Using private key /home/ec2-user/keys-jbarr-us-east.pem (rsa)>>> Using private key /home/ec2-user/keys-jbarr-us-east.pem (rsa)>>> Using private key /home/ec2-user/keys-jbarr-us-east.pem (rsa)>>> Configuring scratch space for user: sgeadmin>>> Configuring /etc/hosts on each node>>> Configuring NFS...>>> Configuring passwordless ssh for root>>> Configuring passwordless ssh for user: sgeadmin>>> Generating local RSA ssh keys for user: sgeadmin>>> Installing Sun Grid Engine...>>> Done Configuring Sun Grid Engine>>>
Start Cluster (Output – 3)
The cluster has been started and configured.
Login to the master node as root by running:
$ starcluster sshmaster jb1
or manually as sgeadmin:
$ ssh -i /home/ec2-user/keys-jbarr-us-east.pem \ [email protected]
When you are finished using the cluster, run:
$ starcluster stop jb1
to shutdown the cluster and stop paying for service
>>> start took 5.337 mins
Check Cluster Status<client>: starcluster listclustersStarCluster - (http://web.mit.edu/starcluster)Software Tools for Academics and Researchers (STAR)Please submit bug reports to [email protected]
-----------------------------jb1 (security group: @sc-jb1)-----------------------------Launch time: 2011-01-14T05:43:44.000ZZone: us-east-1cKeypair: keys-jbarr-us-eastCluster nodes: master running i-3fad6653 ec2-50-16-41-160.compute-1.amazonaws.com node001 running i-3bad6657 ec2-184-73-107-91.compute-1.amazonaws.com node002 running i-35ad6659 ec2-174-129-124-218.compute-1.amazonaws.com node003 running i-37ad665b ec2-50-16-32-211.compute-1.amazonaws.com node004 running i-31ad665d ec2-50-16-31-114.compute-1.amazonaws.com
Access Cluster
SSH to master node as root:
<client>: starcluster sshmaster mycluster
SSH to any given node:
<client>: starcluster sshnode mycluster node001
StarCluster AMIUbuntu-based (8.10, 9.04, 10.04)
Automatically installs/configures:
OpenMPI
Oracle Grid Engine (formerly Sun Grid Engine)
Other pre-installed libraries:
ATLAS
LAPACK
NumPy
SciPy
Using Sun Grid Engine
Run all commands on master, as user sgeadmin:
<client>: starcluster sshmaster mycluster
<master>: su – sgeadmin
Important commands:
qstat – Examine work queue
qsub – Submit work
qhost – List hosts in grid
Sun Grid Engine – Queue and Host Status
Check the queue status using qstat:
Check the host status using qhost:
Sun Grid Engine – Running Scripts
<master-sge>: qsub -V -cwd exercise.sh
Your job 9 ("exercise.sh") has been submitted
The argument “-V” is used to pass the current environment to the job once it's executed.
-#!/bin/bashecho -n "Hello from script running on host "hostnametime find /lib -type f -exec ls -l {} \;echo "Goodbye from script"
Other StarCluster Commands
listclusters
listinstances
createimage
createvolume (EBS)
listvolumes
showconsole
Stop Cluster
AWS charges accrue as long as the cluster is running!
Easy to start, easy to stop, so be parsimonious.
To stop the cluster:
<client>: starcluster stop jb1StarCluster - (http://web.mit.edu/starcluster)Software Tools for Academics and Researchers (STAR)Please submit bug reports to [email protected]
Shutdown cluster jb1 (y/n)? y>>> Shutting down i-3fad6653>>> Shutting down i-3bad6657>>> Shutting down i-35ad6659>>> Shutting down i-37ad665b>>> Shutting down i-31ad665d>>> Removing cluster security group @sc-jb1<client>:
Creating EBS Volumes (made easy)
$ starcluster createvolume 100 us-east-1a
Automagically handles:
Launching instance in specified zone
Creating and attaching an EBS volume to the instance
Partitioning/formatting the EBS volume
Creating a Custom AMI
Create a custom AMI (image):
Launch instance of AMI
Install and configure desired libraries, tools, apps
$ starcluster createimage i-9c9c9c myimg myimgbucket
StarCluster Plugin System
Specify your own custom install routines
Executed after default cluster setup routines
Plugins currently exist for:
Hadoop (MapReduce), ipcluster (Ipython cluster),MPICH2, and more
Example Code (ubuntu.py):
Example Config: