building a real world public cloud from the ground up · building a real‐world public cloud from...

[email protected]| 20121112

Building a real‐world public cloudfrom the ground upg p

Vangelis KoukisVangelis [email protected] Coordinator, okeanos Project

GRNET DeIC konference 2012 1


Outline okeanos ? okeanos ?

Rationale

Design – Platform ‐ Features

Unity ‐ Automation

Opensource Upcoming Opensource – Upcoming

Greek Research and Technology Network DeIC konference 2012 2


What is okeanos?

‘okeanos’ is Greek for ‘ocean’.

Oceans capture store and deliverOceans capture, store and deliverenergy, oxygen and life around the planet.



Si li iSimplicity



ComputeVirtual MachinesComputeVirtual Machines

NetworkVirtual Ethernets

StorageVirtual DisksSto age

S i

tua s s

Vi l Fi llSecurityVirtual Firewalls



Fl ibiliFlexibility



2x5x

1x8x



okeanos service

Goal: Production‐quality IaaS

B t i D t Al h >1600 VM / >1000 Beta in Dec, current Alpha: >1600 VMs / >1000 users

Target group: GRNET’s customers

direct: IT depts of connected institutions

indirect: university students researchers in academia indirect: university students, researchers in academia

Users manage resources over

a simple, elegant UI, or

a REST API, for full programmatic control


a REST API, for full programmatic control


okeanos features

Compute/Network Service: Cyclades

File Storage Service: Pithos+

Image Service: Plankton Image Service: Plankton

Identity Service: Astakos

Volume Service: Archipelago



R i lRationale



How it all started

Need for easy, secure access to GRNET’s datacenters

U f i dli i li i User friendliness, simplicity

Scalable to the thousands

#VMs, TBs, users (Pithos: 10k)

running within GRNET’s AAI Federation

Resell or build your own? Resell or build your own?

IaaS cloud provider, vendor, or own infrastructure?


It all depends on your needs


Build on commercial IaaS?

Commercial IaaS

Amazon EC2 not an end user service Amazon EC2 not an end‐user service

Need to develop custom UI, AAI layers

Vendor lock‐in

Unsuitable for IT depts

• persistent, long‐term servers

• custom networking requirements

GRNET has invested heavily in its core network

8000k f d k fib


> 8000km of dark fiber


Bring vendor into datacenter?

Hypervisor lock‐in

I t k l ti it bl f bli l d? Is a turn‐key solution suitable for a public cloud?

Building public clouds is an ongoing process

Manageable by GRNET’s operation

Integrated into the rest of the infrastructure Integrated into the rest of the infrastructure

Scaling to thousands of users

Build on existing know‐how

Gain know‐how build own IaaS reuse for own services


Gain know how, build own IaaS reuse for own services


What about opensource?

OpenStack, Eucalyptus, OpenNebula

Need a mature opensource core to build around

Maturity, production‐readiness? Maturity, production readiness?

proven in production environments, predictable

Extensibility?

Flexibility? Flexibility?

Upgradeability, maintainability?



D iDesign



okeanos design decisions

Reuse existing components

Build on Google Ganeti Build on Google Ganeti

target commodity hardware

release to the community as opensource


release to the community as opensource


okeanos design principles

No need to make the world

No need to support everything

Service developed and maintained by 10‐15 peoplep y p p

Start from the architecture…

…then discover, combine, reuse the right components

And for everything that’s not already available And for everything that s not already available

Do it yourself!



Jigsaw puzzle

Synnefo

custom cloud management software to power okeanosg p

Google Ganeti backend

VM cluster management: physical nodes, VMs, migrations

OpenStack APIs: Compute API v1.1, Object Storage APIp p , j g

with custom extensions whenever necessary

Th thi t th Then everything comes together

UI, Networking, Images, Storage, Monitoring, Identity


management, Accounting, Billing, Clients, Helpdesk


Why Ganeti?

No need to reinvent the wheel

Scalable proven software infrastructure Scalable, proven software infrastructure

Built with reliability and redundancy in mind

Combines open components (KVM, LVM, DRBD)

Well‐maintained, readable code

VM cluster management in production is

serious business

reliable VM control, VM migrations, resource allocation

handling node downtime software upgrades


handling node downtime, software upgrades


Why Ganeti?

GRNET already had long experience with Ganeti

provides 280 VMs to NOCs through the ViMa service provides 280 VMs to NOCs through the ViMa service

involved in development, contributing patches upstream

Build on existing know‐how for okeanos Build on existing know‐how for okeanos Common backend, common fixes

reuse of experience and operational procedures

simplified, less error‐prone deployment



Pl fPlatform



Software Stack

REST API

Multiple users,multiple resources Synnefo

Multiple VMson cluster Ganeti

SingleVM KVM



l f iPlatform Design

user@home admin@homeWeb Client CLI Client Web Client 2

user@home admin@home

OpenStack Compute API v1.1 GRNET Proprietary

GRNETdatacenter

Synnefo cloud management software

Deb

ianGoogle Ganeti

KVM

Direct Outof Band Access

KVM

VirtualHardware



FFeatures



i l hi iVirtual Machine Actions

My_Windows_desktop

S C lStart Console

Reboot

Shutdown Destroy



IaaS – Compute (1)

Virtual Machines

d b KVM powered by KVM

• Linux and Windows guests, on Debian hosts

Google Ganeti for VM cluster management

accessible by the end‐user over the Web oraccessible by the end user over the Web or

programmatically (OpenStack Compute v1.1)




User has full control over own VMs

C Create

• Select # CPUs, RAM, System Disk

• OS selection from pre‐defined or custom Images

• popular Linux distros (Fedora, Debian, Ubuntu)

• Windows Server 2008 R2

Start, Shutdown, Reboot, Destroy, , , y

Out‐of‐Band console over VNC for troubleshooting




REST API for VM management

OpenStack Compute v1 1 compatible OpenStack Compute v1.1 compatible

3rd party tools and client libraries

custom extensions for yet‐unsupported functionality

Python & Django implementation

Full‐featured UI in JS/jQuery

UI i j h API li UI is just another API client

All UI operations happen over the API



k ( i l h )IaaS – Network (Virtual Ethernets)

IInternet

Private Network 1

Private Network 2Private Network 2

P i N k 3Private Network 3



IaaS – Network ‐ Functionality

Dual IPv4/IPv6 connectivity for each VM

E l tf id d fi lli Easy, platform‐provided firewalling

Array of pre‐configured firewall profiles

Or roll‐your‐own firewall inside VM

Multiple private virtual L2 networks Multiple private, virtual L2 networks

Construct arbitrary network topologies

e.g., deploy VMs in multi‐tier configurations

Exported all the way to the API and the UI


Exported all the way to the API and the UI


U iUnity



Images

Spawn

my own Ubuntu

Freezey



Custom Images: snf‐image

Untrusted images

H h id d d Host cannot touch user‐provided data

Resize fs, change hostname, change passwords, inject files

Split design

f h snf‐image‐host

snf‐image‐helper

All customization in helper VM



OpenStack Object Storage API

Block storage Block storage

Content‐based addressing for blocks

f l ll f bl k Every file is a collection of blocks

Web‐based, command‐line, and native clients

Synchronization, deduplication

An integral part of okeanos An integral part of okeanos User files, Image registry for VM Images

Goal: use common backend with Archipelago


Goal: use common backend with Archipelago


Images

Spawn

my own Ubuntu

Freezey



Images Storage

Clone

Ubuntu + user data

Snapshot



Images – Golden Image



IaaS – Storage



IaaS – Storage

VolumeRADOS

Volume Composer

Maps

object I/O Monitor nodes

ArchipelagoStorage


Object Storage nodes


IaaS – Storage

VolumeRADOS

Volume Composer

Maps

object I/O Monitor nodes


Storage nodes


IaaS – Storage (1)

First‐phase deployment

System provided and custom user Images System‐provided and custom user Images

Redundant storage based on DRBD

VMs survive physical node downtime or failure

Currently under testing Currently under testing

Reliable distributed storage over RADOS

C bi d i h f f h i l i Combined with custom software for snapshotting, cloning

Dynamic virtual storage volumes



IaaS – Storage (2)

Multi‐tier storage architecture

D di d S N d (SSD SAS d SATA ) Dedicated Storage Nodes (SSD, SAS, and SATA storage)

OSDs, e.g., for RADOS

Custom storage layer: Archipelago

h l bl k lmanages snapshots, creates clones over block pools

OS Images held as snapshots

VMs created as clones of snapshots



I iIntegration



Support services

Identity: Astakos

P id h b f k Provides the user base for okeanos

Once authenticated, the user retrieves a

common auth token for programmatic access



A iAutomation



./kamaki$ /kamaki$ ./kamakiUsage: kamaki <group> <command> [options]…

‐‐api=API API can be either openstack or synnefoapi API API can be either openstack or synnefo‐‐url=URL API URL‐‐token=TOKEN use token TOKEN

…

Commands:flavor info get flavor detailsflavor list list flavors

…image create create imageimage delete delete image

$ ./kamaki server shutdown 101 ‐‐url=http://localhost:8000/api/v1.1t k 1234527db2


‐‐token=1234527db2…


./kamaki$ ipython$ ipython

In [1]: from kamaki.client import ClientIn [2]: c = Client('http://localhost:8000/api/v1.1', "1234527db2…")In [2]: c Client( http://localhost:8000/api/v1.1 , 1234527db2… )In [3]: c.list_flavors()…In [4]: i = c.list_images()[ ] _ g ()In [5]: i[5]{u'created': u'2011‐06‐09T00:00:00+00:00',u'id': 7,u'metadata': {u'values': {u'OS': u'windows',

u'size': u'11000'}},u'name': u'Windows',u'progress': 100,u'status': u'ACTIVE',u'updated': u'2011‐09‐12T14:47:12+00:00'}

I [6] t (' i 1' 3 5)


In [6]: c.create_server('mywin1', 3, 5)


Si hSights



Live Demo

Prepare and upload Image from local template VM

Spawn compute cluster to run MPI app

Make local modifications and repeat Make local modifications and repeat

… What if it was over a 3G connection?

Time needed to upload 1GB Image file? Time needed to upload 1GB Image file?

Time needed to prepare and spawn virtual nodes?



I lInternals



Deployment

W b S

DB

Web Server REST API

API ServerSQLui

SQL

API Server

api aai

Logic RAPI

f di t h

GanetiQ

snf‐dispatcher

GanetiMaster

QueueGaneti node

KVMsnf‐gnt‐eventd


…snf‐gnt‐hook


Upcoming goals Credit based resource allocation Credit‐based resource allocation

Abstract away the Ganeti backend, replace with backend

connector behind the MQ

Release to community as reference implementation of

OpenStack Compute v1.1

Support live modification of VMs in Ganeti pp

Snapshots, clones in storage layer

i d i i i i li i i Dramatic decrease in VM initialization time

Support workloads with 100s of ephemeral VMs


• e.g. for scientific computation, MPI jobs


U iUpcoming



Current and Upcoming features

Now: Alpha2

C b i Pi h Common user base, custom user images on Pithos+

short‐term: Synnefo v0.12, Betay

Ultra‐lightweight VMs on Archipelago with RADOS backend

medium‐term

Volumes: clonable / snapshottable / attachable disks/ p /

Network and storage hotplugging


Upcoming beta in fully populated datacenter


Opensource

Synnefo: Cyclades / Pithos+ / Astakos y y / /

https://code.grnet.gr/projects/synnefo

https://code.grnet.gr/projects/pithos

https://code.grnet.gr/projects/astakos

kamaki

https://code.grnet.gr/projects/kamaki


pip install or apt‐get install everything!

http://okeanos.iohttp://okeanos.io


Thank You!

Questions?Questions?



Asynchronous design DB contains All state needed to handle API queries DB contains All state needed to handle API queries

no need to reach the backend

Ganeti GetInstanceInfo() is a proper job too slow Ganeti GetInstanceInfo() is a proper job, too slow

Two distinct paths, effect and update

Effect changes to VMs when servicing API requests to modify VM state

issue commands to Ganeti backend, over RAPI

ACK reception of request to user

Update DB, when interesting things happen user or admin initiated


Queue notifications to Message Queue, over AMQP


Synnefo deployment

W b S i

DB

Web Server REST API

API ServerSQL

ui

SQL

API Server

api aai

Logic RAPI

f di t h

GanetiQ

snf‐dispatcher

GanetiMaster

QueueGaneti node



…snf‐gnt‐hook


The “effect” Path Reception of API request to modify VM state (e g Reception of API request to modify VM state (e.g.,

PUT /servers over HTTP)

API enforces access rights and policy

Ganeti knows no cloud users or access rights

Need to translate from Openstack Compute to backend

ops (e g CreateInstance())ops (e.g., CreateInstance())

Asynchronous request processing

Return HTTP 202 Accepted

it’s up to the API client to poll for completion



Synnefo deployment

W b S i

DB

Web Server REST API

API ServerSQL

ui

SQL

API Server

api aai

Logic RAPI

f di t h

GanetiQ

snf‐dispatcher

GanetiMaster

QueueGaneti node



…snf‐gnt‐hook


The “update” path May run at any time May run at any time

Completely decoupled from “effect” path

Design goal:

G ti d i f t b f t d Ganeti admins free to bypass frontend

Synnefo adapts

Synnefo logic triggered on backend events

Ganeti operation progressing in the queue Ganeti operation progressing in the queue

Synnefo hook running inside Ganeti

H k i h i VM’ lif l


• Hooks run at various phases in a VM’s lifecycle


Synnefo deployment

W b S i

DB

Web Server REST APIui

API ServerSQL

SQL

API Server

api aai

Logic RAPI

f di t h

GanetiQ

snf‐dispatcher

GanetiMaster

QueueGaneti node



…snf‐gnt‐hook


The Ganeti event daemon Ganeti master manages job queue Ganeti master manages job queue

Jobs pass Queued, Waiting, Running, end up in Canceled,

Success, Error.

Need a way for Synnefo to monitor job progressy y j p g

Synnefo‐specific solution: Ganeti event daemon

Passively monitor the Ganeti job queue

Notifications over AMQP on job progress

Synnefo logic listens to Message Queue, updates DB

inotify()‐based mechanism no code changes to Ganeti


inotify()‐based mechanism, no code changes to Ganeti


The Synnefo hook in Ganeti Different phases in a VM’s life le Different phases in a VM’s lifecycle

{pre, post} – {add, start, stop, reboot, modify}

Run Synnefo‐specific hook in post‐*

Pushes VM configuration notifications to MQ

e.g., NIC setupg , p



IaaS – Network ‐ Implementation

Custom modifications to Ganeti

IP pool management for the public network IP pool management for the public network

Custom‐written DHCP server over NFQUEUE

Custom interface handling scripts

Enforce VM networking configuration Enforce VM networking configuration

Private Networks

Alpha: pre‐provisioned bridges to 802.1Q VLANs

Later on: MAC‐prefix based filtering


Later on: MAC prefix based filtering


Synnefo deployment

W b S i

DB

Web Server REST APIui

API ServerSQL

SQL

API Server

api aai

Logic RAPI

f di t h

GanetiQ

snf‐dispatcher

GanetiMaster

QueueGaneti node



…snf‐gnt‐hook


Reconciliation with Ganeti What if the MQ is do n and messa es are lost? What if the MQ is down, and messages are lost?

Ganeti is the Single Source of Truth for VM state

Reconcile DB state asynchronously

On success notification for a Ganeti GetInstanceInfo() op

Triggered periodically, e.g., using crongg p y, g , g

or even by the administrator,

i t i t i f llrunning gnt‐instance info manually


building a real world public cloud from the ground up · building a real‐world public cloud from...

Documents