cloud standards in the real world: cloud standards testing for developers

Open Cloud Standards In The Real World:

Cloud Standards Testing For Developers

A Project Of the National Science Foundation Cloud and Autonomic ComputingIndustry/University Cooperative Research Center At Texas Tech University

Alan Sill, Ph.D Site Director

July 21, 2014OSCON Open Cloud Day

First, some preliminaries…

What it often looks like when developers encounter standards committees.

OSCON 2014 Open Cloud DayJuly 21, 2014

What it ought to look like:

(Taken from an actual Cloud Plugfest.)


A New Research Effort


NSF CAC Current Participants:


CAC Industry Members:

Technical Partnerships:

Other CAC University Sites:

Others to

come!

Texas Tech University Other CAC Sites

…(Pending)(Existing)

New NSF Award —April 2014

Background: Large-Scale Academic Computing


In Top500!!3rd in Texas and Big 12!!In Top 100 academic institutions in the world !

Supports high-quality academic research across many fields of science, business data analysisand engineering.

TTU High Performance Computing Center:

Background: Big Data


2014:! Phones: 100+ Gigabytes! Science and Business: 100s to 1000s of Petabytes

Experience:

Related CyberInfrastructure Projects

Experience In Global-Scale Computing


The Role of Standards for Risk Reduction and Inter-operation in XSEDE

XSEDE: The Next Generation of US Supercomputing Infrastructure

Cloud and grid standards now power some of the largest academic supercomputing infrastructures in the world!

Blacklight Shared Memory 4k Xeon cores

!Darter 24k cores !Nautilus Visualization Data Analytics !Keeneland CPU/GPGPU

Stampede 460K cores w. Xeon Phi >1000 users Upgrade in 2015

Wrangler Date Analytics

Trestles IO-‐intensive 10k cores 160 GB SSD/Flash !Gordon Data intensive 64 TB memory 300 TB Flash Mem

Open Science Grid High throughput 124 sites

Blue Waters Leadership Class

SuperMIC 380 nodes – 1PF (Ivy bridge, Xeon Phi, GPU)

Over 13 million service units/day typically delivered as of 2014 across all XSEDE supercomputing sites (about 3 million core hours/day), totaling about 1.6 billion core hours per year

Yellowstone Geosciences

US National Cyberinfrastructure

Promote an open, robust,

collaborative, and innovative ecosystem

Adopt, create and

disseminate knowledge

Extend the impact of cyber-

infrastructurePrepare the current and

next generation

Provide technical expertise and

support services

Collaborate with other CI groups

and projects

NSF SupercomputingResources


FutureGrid* CS testbed

Maverick Visualization Data Analytics

Comet “Long Tail Science” 47k cores/2 PF High throughput

ACI-‐REF Campus sharing

Grids

Credit: Irene Qualters, US National Science Foundation

XSEDE Services Layer: Simple services combined in useful ways

–Resource Namespace Service 1.1 –OGSA Basic ExecuKon Service –OGSA WSRF BP – metadata and noKficaKon –OGSA-‐ByteIO –GridFTP –JSDL, BES, BES HPC Profile –WS Trust Secure Token Services –WSI BSP for transport of credenKals –… (more than we have room to cover here)

Examples – (not a complete list)

Bottom line: XSEDE represents a demonstration of the effective use of standards to provide interoperability within large-scale supercomputing cyberinfrastructure.

Used between supercomputers and

all the way to university campuses!

Distributed Across 124 Sites Open Science Grid currently consists of over 124 geographical sites, operating on a wide variety of computing systems

600k - 800k jobs/day!

Virtual Organizations, July 18th 2011

Science VOs on the Open Science Grid• Astrophysics • Biochemistry • Bioinformatics • Earthquake Engineering • Genetics • Gravitational-‐wave physics • Mathematics • Nanotechnology • Nuclear and particle physics

… and many others!

www.egi.euEGI-InSPIRE RI-261323 www.egi.euEGI-‐InSPIRE RI-‐261323XSEDE/EGI BoF - XSEDE 2013

EGI international presence numbers

Storage Value (yearly increase)

Disk (PB) 235 PB (+69%)

Tape (PB) 176 PB (+32%)

Value (yearly increase)

CPU cores 361,300 across 53 countries (1.44 M job/day)

Standards-based international collaboration

EGI Federated Cloud: a successful standards-based international federated infrastructure

Credit: David Wallom Chair EGI Federated Cloud Task Force

Members •70 individuals •40 institutions •13 countries

Stakeholders •23 Resource Providers •10 Technology Providers •7 User Communities •4 Liaisons

Technologies •OpenStack •OpenNebula •StratusLab •CloudStack (in evaluation)

•Synnefo •WNoDeS

BSC

CNRS

LMU

OeRC

Masaryk

TUD

IFAE

Cyfronet

100%IT

CESNET

RADICAL

SRCE

DANTE

FZJ

GRNET

GWDG

STFC

SARA

KTH

INFNFCTSG

EGI.eu

Imperial

CESGACETA

IFCA

IGI

IPHC

IN2P3

SZTAKI

IISAS SixSq

Standards •OCCI (control) •OVF (images) •X.509 (authN) •CDMI (storage - under development)

(Updated July 2014)

An international standards-

basedfederated

cloud infrastructure:"

In production,

in use and growing!

www.egi.euEGI-InSPIRE RI-261323

Federated Cloud architecture

Open to new members: Join as user, or as an IaaS/PaaS/SaaS service provider: http://go.egi.eu/cloud

Cloud hypervisor (e.g. OpenStack, OpenNebula, EmotiveCloud, Okeanos…)

Cloud site academic/commercial

Domain specific services in Virtual Machine Images

FedCloud User interfaces

Standards used to enable federation • OCCI: VM Image management • OVF: VM Image format • X509: Authentication • (CDMI: Storage)

FedC

loud

Ope

ration

interfaces • Information system (BDII)

• Monitoring (SAM) • Accounting (APEL) • AAI (Perun)

Virtual organisations

• GLUE2: Resource discovery and Description

• Others in development

Federation monitoring

http://go.egi.eu/cloud

Examples of CAC@TTU

Involvement In Cloud Standards Testing

US NIST SAJACC Working GroupAlan Sill, TTU and Eugene Luster, R2AD, co-chairs

NIST Cloud Computing and MobilityForum and Workshop

March 25, 2014

SAJACC Phase II Use Case Definition and Testing Project for Cloud Standards, APIs and Cloud Software

Product Implementations

http://collaborate.nist.gov/twiki-cloud-computing/bin/view/CloudComputing/SAJACC

NIST SAJACC Public Process


http://collaborate.nist.gov/twiki-cloud-computing/bin/view/CloudComputing/SAJACC

European Commission “Standards & Interoperability for e-Infrastructure Implementation” Initiative

SIENA (2010-‐2012) was a Support Ac*on funded by the European Commission under FP7 (2007-‐13) CapaciTes programme. This project created the ongoing “CloudScape” conference series and led to other related projects, such as CloudWatchHub and the CloudScout effort aimed at enhancing the adopTon of cloud methods by SMEs.

A coordinated effort...

...towards the delivery of a futuree-‐Infrastructures Roadmap....

...aligned with the needs of European

and naVonal iniVaVves and the evolving world.

IEG – Industry Expert Group

Partners

Interna*onal Experts

REB – Roadmap Editorial Board

SLG – Special Liaison Group

Standards Development Organisa*ons (SDOs)

Distributed Compu*ng Infrastructure (DCI) Ini*a*ves

Consolidated Cloud Compu*ng Workshops

23

Open-access academically hosted Cloud

Interoperability Testbed project

CAC@TTU Project Areas

Target Cloud Standards-Related Organizations:

• It is often said that there are “too many standards organizations”. This is a lot like saying there is “too much software”.

• Each has its own area of specialty, its own contributor base, and its own method of funding to develop its work products.

• CAC will study products and effectiveness of each of these organizations.


…

Initial CAC@TTU Project Areas1. Product and Standards Testing

• Cloud Performance Testbed • Cloud Standards Testbed • Cloud Interoperability Testbed • Cloud Security Testbed <— (Future)

2. Design Labs • Storage Design and Testing Lab • Network Design and Testing Lab

3. Developer Events • Cloud Plugfest Series • Participation in technical partner events • Organization of and participation in conferences


p CAC@TTU is new! n More coming…

• Cloud Standards Testbed!• Cloud Performance Testbed!• Cloud Interoperability Testbed!• Cloud Tester Benchmark Suite

Core Technology Efforts

*

* (In cooperation with The Aerospace Corporation and other CAC partners)

Continuing series…

!Oriented towards

REAL DEVELOPMENT

!Past events

co-sponsored by many open source and

standards-related organizations

including

OGF, DMTF, SNIA, OASIS, ETSI, OCEAN

and OW2

Developer-oriented in-person standards and software testing series

Easy to get involved and join in events as developers or project

researchers

Cloud Plugfest 11 just completed!

http://cloudplugfest.org

Cloud Plugfest Developer Series:


More events in planning pipeline.


CACCloud Standards

Testing and Implementation Projects

Testing Cloud-Oriented Use Cases§ Builds on results of publicly accessible documents and forums for general

interoperability testing, in addition to internal ones that may be set up by companies and US agencies.

§ Up to now these have mostly been approached piecemeal. § CAC is carrying out an organized project within the center to extend these

techniques into tools that can be used for formal support of cloud conformance, interoperability and requirements acceptance testing on an ongoing basis.

§ This project builds on previous public efforts such as SAJACC.

§ Our plan is to work with and contribute to public processes in open settings while creating conditions in which our expertise and value can be applied for center members to cloud standards and product testing.


Current Standards Being Tested§ Cloud Data Management Interface (CDMI) from the Storage Networking

Industry Association (SNIA). (Reference implementation available.) § Open Cloud Computing Interface (OCCI) from the Open Grid Forum

(OGF). (Many implementations available.) § Cloud Infrastructure Management Interface (CIMI) and Open

Virtualization Format (OVF) from the Distributed Management Task Force (DMTF). (CIMI implementations just becoming available; OVF implementations well established.)

§ Topology and Orchestration Specification for Cloud Applications (TOSCA) from the Organization for the Advancement of Structured Information Standards (OASIS). (Several implementations available.)

§ Others under active evaluation.


Texas Tech Leadership Team

Alan Sill, Ph.D Site Director

Yong Chen, Ph.D Associate Site Director

Ravi Vadapalli, Ph.D Associate Site Director

Experienced university-based researchers carrying out projects with real-world applicability that draw on decades of experience and the best, newest techniques. Publication oriented, but available for practical member-driven projects. Full university support from TTU and other participating CAC university sites.


v

COMPUTER SCIENCE

Rattikorn Hewett, Ph.D.

Chairperson in the Department

of Computer Science

Areas of expertise » Security and privacy in cloud environments» Intelligent data management & understanding» Automated web service composition

Sunho Lim, Ph.D.

Assistant professor in the

Department of Computer Science

Areas of expertise » Wireless networks and mobile computing» Mobile data management» Mobile software» Network security

Mohan Sridharan, Ph.D.



Areas of expertise » Learning and estimation in Big Data domains» Stochastic machine learning» Robotics and AI

Noe Lopez-Benitez, Ph.D.

Associate professor in the


Areas of expertise » Cloud computing» Parallel and distributed computing» High-performance computing

Yong Chen, Ph.D.



Areas of expertise » Cloud computing storage systems» Data management and data centers» Cloud programming and software environments» Cloud platform architecture and virtualization

Eunseog Youn, Ph.D.



Areas of expertise » Large-scale machine learning» Feature selection» Text mining» Bioinformatics

FACULTY MEMBERSHIGH PERFORMANCE COMPUTING CENTER

SCHOOL OF LAW

Victoria Sutton, Ph.D.

5FZQ�<MNYąJQI�-TWS�UWTKJXXTW�in the School of Law

Areas of expertise » Biosecurity» Emerging technologies» Energy» Constitutional law

Many University Researchers, Deep Expertise

INDUSTRIAL ENGINEERING

Timothy Matis, Ph.D.


Department of Industrial

Engineering

Areas of expertise » Operations research» Stochastic processes» Queueing theory» Ad-hoc communication networks» Conceptual learning theories and virtual

learning environments

Areas of expertise » Mixed-integer programming, theory and computation

» Nonlinear programming, theory and computation

» Operations research probabilistic methods» Applied management science

MATHEMATICS & STATISTICS

Philip W. Smith, Ph.D.

Senior director, High Performance

Computing Center

Areas of expertise » Optimization » Design and implementation of parallel and

serial codes» Code optimization » Numerical analysis and linear algebra

MECHANICAL ENGINEERING

Fazle Hussain, Ph.D.

Professor in the Department of

Mechanical Engineering

Areas of expertise » Fluid Flow, turbulence, & transport phenomena » Microseismology and inverse scattering » Modeling cancer tissue growth and

targeted drug delivery» Atmospheric boundary layers and wind

turbine aerodynamics

Siva Parameswarn, Ph.D.

Professor in the Department of

Mechanical Engineering

Areas of expertise » Modeling HF oscillatory ventilators» Aerodynamics of ground vehicles» Wake development behind wind turbines®�+LZPNUPUN�LMÄJPLU[�YVVM[VW�^PUK�[\YIPULZ

PHYSICS

Ismael Regis de Farias Jr., Ph.D.


Department of Industrial Engineering

Alan Sill, Ph.D.

Adjunct Professor in the

Department of Physics

Areas of expertise » High performance computing» Data-intensive scaled resource management» Large-scale distributed computing» Standards development

Experienced university-based researchers carrying out projects with real-world applicability that draw on decades of experience and the best, newest techniques. Publication oriented, but available for practical member-driven projects. Full university support from TTU and other participating CAC university sites.


Many University Researchers, Deep Expertise

FACULTY MEMBERS

RAWLS COLLEGE OF BUSINESS

Peter H. Westfall, Ph.D.

5FZQ�<MNYąJQI�-TWS�5WTKJXXTW�TK�Statistics in the Rawls College of

Business Administration

Zhangxi Lin, Ph.D.


Department of ISQS

Areas of expertise » Advanced analytics» Business intelligence» Statistics» Biostatistics» Bioinformatics

Areas of expertise » Big data analytics» Business intelligence» Information systems» Electronic commerce

Mayukh Dass, Ph.D.

Assistant Professor of Marketing

in the Rawls College of Business

Areas of expertise » Brand analytics» Business strategy analysis» Dynamic environment modeling» Network analysis

PETROLEUM ENGINEERING

Marshall Watson, Ph.D., P.E.

Roy Butler Chair and chairman

of the Bob L. Herd Department

of Petroleum Engineering

Mohamed Soliman, Ph.D., P.E.

George P. Livermore Chair and

Professor in the Bob L. Herd Dept.

of Petroleum Engineering

2��7FąVZQ�&\FQ��5M�)�


Bob L. Herd Department of

Petroleum Engineering

Ravi Vadapalli, Ph.D.

Adjunct Professor in the

Department of Petroleum

Engineering

Areas of expertise ®�9LZLY]LZ�KLÄUP[PVUZ�HUK�HWWSPJH[PVU» Production optimization » Coal bed methane» Unconventional reservoir analysis

Areas of expertise » Fracturing» Reservoir engineering» Well test analysis» Conformance

Areas of expertise ®�4PJYVòIVYLOVSL�KYPSSPUN®�*6�ò,69�HUK�JOLTPJHS�ÅVVKPUN» Reservoir simulation» Advanced well completion

Areas of expertise » High performance computing» Reservoir modeling» Cancer radiotherapy informatics» Cloud and autonomic computing

Many University Researchers, Deep ExpertiseExperienced university-based researchers carrying out projects with real-world applicability that draw on decades of experience and the best, newest techniques. Publication oriented, but available for practical member-driven projects. Full university support from TTU and other participating CAC university sites.


Links For Further Information and To Join:

§ NSF Cloud and Autonomic Computing Center main site: http://nsfcac.org

§ CAC@TTU information and membership materials:http://www.depts.ttu.edu/cac/

§ Cloud Plugfest developer series: http://cloudplugfest.org § NSF I/UCRC main site: http://www.nsf.gov/eng/iip/iucrc/ § Cloud standards organization compilation:

http://cloud-standards.org § NSF CAC@TTU contact email: [email protected]


http://nsfcac.org

http://www.depts.ttu.edu/cac/


http://www.nsf.gov/eng/iip/iucrc/

http://cloud-standards.org

mailto:[email protected]

cloud standards in the real world: cloud standards testing for developers

Technology

open cloud standards

oscon open cloud day

federated cloud infrastructure

federated cloud architecture

sites open science grid

international standards

actual cloud plugfest

grid standards