cloudmesh feb-2016 2 (1)

PowerPoint Presentation

CloudmeshAn Introduction

Gregor von [email protected]

[email protected], http://cloudmesh.github.io/client/

1

ResourcesDocumentationhttp://cloudmesh.github.io/client/

Source RepositoryCloudmesh_base: https://github.com/cloudmesh/baseCloudmesh_client:https://github.com/cloudmesh/client

[email protected], http://cloudmesh.github.io/client/2

Independent StudyI accept a limited number of talented students for independent studies. Projects: extending cloudmesh or other interesting cloud related workEffort: The usual effort 3 credit hours = 12 hours weekly at minimumWhy? E.g. 36 hours per week /3 courses = 12 hoursWill not accept students that have more than 9 credit hours per week.Grading: Based on working projectJob: No. You must at least have done an independent study with A or A+ grade.


OutlineMotivationHybrid CloudsPrevious EffortsIntroductionCurrent FocusMany OSClient FocusedLayersAPI, Shell, PortalCloud UsageOpenstackAzureAWSComet

HPC UsageQueuesExperimentsCoordinationSimple workflowPlatformsLaunchersVirtual [email protected], http://cloudmesh.github.io/client/4

MOTIVATION


Hybrid CloudCloud Computing has many different facets and offers the ability to leverage IaaS, PaaS and SaaS on resource and services offered by others.However, some application require due to privacy concerns, cost, or functionality the use private clouds or a mix hosted at different providers.This leads us to the term hybrid cloud.Definition:A hybrid cloudis acomputing environment which uses a mix of on-premises, privatecloudand third-party, publiccloudservices with orchestration between the two platforms. It can offer bare metal, traditional HPC as a service, or virtualized resources offered in a cloud.von Laszewski, [email protected], http://cloudmesh.github.io/client/6

Federated CloudsCapacity Federation

Technology Integration

[email protected], http://cloudmesh.github.io/client/7AWSAzureRackspaceHP CloudHybridCloudFutureSystemsChameleonCloud.HybridCloudIaaSPaaSCloud, ClusterSaaSBareMetalaaS aaS

Technology Robustness

Technology Federation

[email protected], http://cloudmesh.github.io/client/8AWSAzureHPEucalyputusHybridCloud

AWSAzureOpenStackEucalyputusHybridCloud

Federated CloudsMotivationPriceAvailabilityFault ToleranceCapacityResource LimitationsFeatures within the cloudHybrid CloudsIndependence:Avoid vender lock in

RequirementsAccessibleEase of useIntegratedFlexibleSupport multiple user community typesEnduserAdministratorDevelopers


Gartners Analysis


http://www.zdnet.com/article/microsoft-and-google-rise-while-ibm-sinks-in-gartners-latest-magic-quadrant-for-cloud-providers/

IaaSAmazon Web Services (AWS)Bulk of market share in the IaaS market, 10 times more cloud IaaS compute capacity in use than the aggregate total of the other 14 providers Gartner studied.MicrosoftAzure Infrastructure Services, second place in market share, with more than twice as much cloud IaaS compute capacity as the other providers Gartner studied (excluding AWS).GoogleThe Google Cloud Platform combines Compute Engine (its IaaS offering), App Engine (an application PaaS or aPaaS offering) and complementary services like object storage and a Docker container service.CenturyLinkGlobal communications service provider CenturyLink acquired Web hoster Savvis in 2011 and pure-play cloud IaaS provider Tier 3 in 2013 and merged them to create its CenturyLink Technology Solutions business unit, which now offers CenturyLink Cloud (CLC).VMwareVMware has traditionally focused on virtualization technologies, but made its vCloud Air cloud IaaS service generally available in 2013.IBM (SoftLayer)IBM acquired SoftLayer in 2013, and by 2014 it migrated its existing SmartCloud Enterprise cloud IaaS customers to SoftLayer and shut SmartCloud down.RackspaceRackspace offers a multitenant, OpenStack-based public cloud and three flavors of hosted private cloud: vCloud Director-based and VMware-virtualized, Microsoft Cloud OS-based and Hyper-V virtualized or OpenStack-based and KVM-virtualized. It also offers an on-premises private cloud.

HP and Rack space have since stopped serving public clouds and focus on specialized customers.

http://www.cio.com/article/2947282/cloud-infrastructure/top-cloud-infrastructure-as-a-service-vendors.html


Academic cloud providersFutureSystems.orgComet Cloud (not OpenStack)CloudLabChameleonCloud.orgCybera.org (Canada)UBuffalo Cloud (Eucalyptus)RedCloud (Cornell)Many many more [email protected], http://cloudmesh.github.io/client/12

Need for managed cloud services?Hybrid cloud brokerage - managing services you use on clouds.Centralized visibility and control.Optimising cost/performance and staying compliant.Hybrid CloudsOn Premise - eg: OpenStackPublic Clouds - eg: AWS, Azure, SoftLayerCredential management & Access control across multiple clouds.Policy based cataloguing & orchestration.


CLOUDMESH


INTRODUCTIONCloudmesh is a project to easily manage virtual machines and bare metal provisioned operating systems in a multi cloud environment. With Cloudmesh, it is possible to use Azure, Amazon Web Services, and other clouds such as chameleon.Cloudmesh also provides an interface to manage thousands of jobs on High Performance Computing (HPC) clusters, such as XSEDEs comet supercomputer


How cloudmesh fills the gap?Matches cloud consumer needs with multiple providers services.Researchers can use one platform to manage their clouds.Orchestrates provisioning and allocation of cloud resources.Local copy of your cloud data is created, so new clouds with similar configurations can be created easily.Default attributes allow easy control of cloud artefacts.


Previous EffortFocus was on portal for FutureGrid that accesses hybrid clouds within FutureGrid Portal while also managing the users credentialsCons: Portals are time consuming to developManaging credentials for others is dangerousPros:Baremetal provisioning before others did itIt actually worked

Effort is known as cloudmesh


Current Effort: Cloudmesh ClientKnown under the code name cloudmesh clientFocus on user installable and hosted serviceFocus on command shell first, than portal.The portal is also hosted on the client machineEasy installation while eliminating Unix specific components such as LDAPTarget OS right from the start: OSX, Linux, WindowsProvide most useful featuresProvide access to SDSC comet


Cloudmesh Client and PortalSimplify accessQueueing SystemsContainersIaaS

ParallelismTaskComputeData

(GPU)XSEDEHPCCloudsCometJetstreamBridges

19


Cloudmesh Client and PortalData RepositoryAccessCommand Shell & LineAPIRESTPORTALComputeVMs, Containers, Batch ChoreographyGroups, Workflows, PaaS

20

$ cmcm> register remotecm> default cloud=kilocm> vm boot


Integrated Access InterfacesIntegrated State Management!E.g. you start a VM in the commandline, you can see it in the Web interfaceCommunication through a NoSQL database


InstallWe recommend python 2.7.10pip 8.0.2

Will install on LinuxOSXWindows

See:http://cloudmesh.github.io/client/ Do not just do what we present here on the slides

pip install cloudmesh_client

One line install with curl

In case we find resources for it (good student projects):rpm, dpg, ..dmgexe

22


Overview: ShellShell$ cmcm> register remotecm> default cloud=kilocm> vm boot

Yes that is it! Defaults for images and flavors can be set

Command Linecm register remotecm default cloud=kilocm vm boot

Integration of the cybera (Alberta, Canada) took 5 minutes

23


Basic UsageCommandDescriptioncm helphelpcm manmanual pagescm script.cmexecute cm commands in script


Cloud CommandsCloudCommandDescriptioncm image listlist imagescm flavor listlist flavorscm vm listlist vmscm vm bootboot vmcm vm boot cloud=kiloboot vm on cloud kilocm default cloud=kiloset default cloud to kilocm select imageselect interactively the default image (not implemented yet).cm select flavorselect interactively the default flavor (not implemented yet).cm select cloudselect interactively the default cloud (not implemented yet).


Shell CommandsCommandDescriptioncm color onsets the shell colorcm color offswitches off the colorcm refresh onautomatic refresh from the cloudscm refresh offdata is only read from the database. Useful for managing thousands of VMs or limit your access to the cloud.var a=xyxdeclares a variablevar username=cloudmesh.profile.usernamereads the variable from the cloudmesh.yaml filevar time=nowgets the time and store it in the variable time


HPC CommandsCommandDescriptioncm helpHelpcm hpc queue info about the queue cm hpc infoinformation about the queues on the HPC resourcecm hpc run uname -aruns the command unamecm hpc run listprints the ids of previously run jobscm hpc run listprints the ids of previously run jobscm hpc run list 11prints the information regarding the job with the id 11


PaaS ManagementVirttual Cluster$ cm cluster n 10 group=myCluster

Hadoop$cm hadoop n 10 group=myHadoop

28


Launchers: commands via GUI

29Specific LaunchList of Launchers


Homeworkhttp://cloudmesh.github.io/client/exercises.html

Install CloudmeshStart vms on one cloudStart groups of vms


QuestionsGregor von [email protected]@gmail.com, http://cloudmesh.github.io/client/31

cloudmesh feb-2016 2 (1)

Internet