HUAWEI TECHNOLOGIES CO., LTD. Page 1Huawei Confidential

Issues ofOpenStack multi-region modeChaoyi Huang ( joehuang@huawei.com )

Last edited May 28, 2015

HUAWEI TECHNOLOGIES CO., LTD. Page 2Huawei Confidential

Nova

Cinder

OpenStackRegion1

GlanceNeutron

Ceilometer

KeyStone

Horizon

Nova

Cinder

OpenStackRegion2

GlanceNeutron

Ceilometer

API Req. API Req.

Token Req.

OpenStack mutli-region mode: each site will be deployed with independent OpenStack instance (including Nova,Cinder,Neutron, Ceilometer, Glance, Glance sometimes will be shared for multi-site). KeyStone service is shared for same credential to access multi-region, and Horizon functions as the unified portal for the multi-region cloud.

Site1 Site2

HUAWEI TECHNOLOGIES CO., LTD. Page 3Huawei Confidential

Quotas, tenants: Missing global view across regions Global quota issue including quotas for Nova, Cinder, Neutron... Issue for soft/hard quotas based on current usage for all region for this tenant Issue for each of the individual projects managing quotas provide a global view

Issues from production multi-site / multi-region cloud:Vancouver session moderated by Comcast: https://etherpad.openstack.org/p/YVR-ops-multi-siteOPNFV multisite project: https://etherpad.opnfv.org/p/multisite_usecase_collection

vApp 5

vApp 1vApp 2

vApp 6

OpenStackvApp 8

vApp 4vApp 3

vApp 7OpenStack

Nova Quota Cinder Quota Neutron Quota Nova Quota Cinder Quota Neutron Quota

Issue: discrete quota control

HUAWEI TECHNOLOGIES CO., LTD. Page 4Huawei Confidential

Resource utilization across regions: Missing global view across

regions Issue for a user being able to management all his virtual resources from one centralized management

interface, at least to have a summarized view of the total resource capacity and the live utilization of his

virtual resources, for example

Issues from production multi-site / multi-region cloud:Vancouver session moderated by Comcast: https://etherpad.openstack.org/p/YVR-ops-multi-siteOPNFV multisite project: https://etherpad.opnfv.org/p/multisite_usecase_collection

vApp 5

vApp 1vApp 2

vApp 6

OpenStackvApp 8

vApp 4vApp 3

vApp 7OpenStack

VM… Volume… Network, Port… VM… Volume… Network, Port…

Issue: discrete resource utilization

HUAWEI TECHNOLOGIES CO., LTD. Page 5Huawei Confidential

Missing global view for tenant level metering data. Issue for a user being able to access all the logs and indicators produced by multiple

OpenStack instances, in a centralized way

Issues from production multi-site / multi-region cloud:Vancouver session moderated by Comcast: https://etherpad.openstack.org/p/YVR-ops-multi-siteOPNFV multisite project: https://etherpad.opnfv.org/p/multisite_usecase_collection

Ceilometer Ceilometer

Metering…Sample…

Issue: discrete metering data

Metering…Sample…

HUAWEI TECHNOLOGIES CO., LTD. Page 6Huawei Confidential

Neutron: No global view for tenant level IP address space management Issue of clone security groups across regions Issue of establishing tenant level L2/L3 networking across region for application high availability across

OpenStack instances

Issues from production multi-site / multi-region cloud:Vancouver session moderated by Comcast: https://etherpad.openstack.org/p/YVR-ops-multi-siteOPNFV multisite project: https://etherpad.opnfv.org/p/multisite_usecase_collection

Neutron vApp Standby

Neutron

Network CIDR… Network CIDR…

Security Group… Security Group…

APP networking across OpenStack instance…For heartbeat, states replication….vApp

master

HUAWEI TECHNOLOGIES CO., LTD. Page 7Huawei Confidential

Nova: Lack of clone ssh keys from one region to another

Issues from production multi-site / multi-region cloud:Vancouver session moderated by Comcast: https://etherpad.openstack.org/p/YVR-ops-multi-siteOPNFV multisite project: https://etherpad.opnfv.org/p/multisite_usecase_collection

Nova Nova

KeyPair… KeyPair…

HUAWEI TECHNOLOGIES CO., LTD. Page 8Huawei Confidential

Glance: Lack of replicate images across regions

Issues from production multi-site / multi-region cloud:Vancouver session moderated by Comcast: https://etherpad.openstack.org/p/YVR-ops-multi-siteOPNFV multisite project: https://etherpad.opnfv.org/p/multisite_usecase_collection

Glance Glance

Upload image… Upload Imgae…

HUAWEI TECHNOLOGIES CO., LTD. Page 9Huawei Confidential

Cinder: Lack the function like volume replication across regions for Geo-site

Redundancy/Disaster Recovery

Issues from production multi-site cloud:Vancouver session moderated by Comcast: https://etherpad.openstack.org/p/YVR-ops-multi-siteOPNFV multisite project: https://etherpad.opnfv.org/p/multisite_usecase_collection

Cinder CinderVolume

Volume replication for Geo-site Redundancy/Disaster Recovery ….

Volume

HUAWEI TECHNOLOGIES CO., LTD. Page 10Huawei Confidential

Cascading OpenStack

…

Tenant 1 Virtual Resources

Tenant 2 Virtual Resources

Tenant x Virtual Resources

Cascaded OpenStack 2Cascaded OpenStack 1 Cascaded OpenStack y

OpenStack API OpenStack API OpenStack API

OpenStack cascading solution – best matched multisite solution: Integrate multiple OpenStack instances from multi-site into one unified cloud for tenant’s resources distributionhttps://wiki.openstack.org/wiki/OpenStack_cascading_solution

Tenant 2Tenant 1 Tenant x

OpenStack API

• global view of tenant quota• global view of tenant resource utilization• global view of tenant metering data• tenant level global IP address space

management• clone security groups across regions• L2/L3 networking across regions• KeyPair replication• Replicate images• Volume replication• …..

Service Orchestrator x

Service Management

Service Orchestrator 1

…

HUAWEI TECHNOLOGIES CO., LTD. Page 11Huawei Confidential

Cascading OpenStack

…

Tenant 1 Virtual Resources

Tenant 2 Virtual Resources

Tenant x Virtual Resources

Cascaded OpenStack 2Cascaded OpenStack 1 Cascaded OpenStack y

OpenStack API OpenStack APIOpenStack API

Service Orchestrator x

Service Management

Service Orchestrator 1

OpenStack API OpenStack APIOpenStack API

… …

Cloud over clouds by OpenStack cascading: the cascaded OpenStack instance even can be running inside Amazon AWS or Windows Azure, and then being integrated into a cloud through the cascading OpenStack. This is cloud over clouds.

HUAWEI TECHNOLOGIES CO., LTD. Page 12Huawei Confidential

Service Orchestrator x

Service Management

Service Orchestrator 1

CascadingOpenStack(Tenant 1)

…CascadingOpenStack(Tenant 2)

CascadingOpenStack(Tenant x)

…

Tenant 1 Virtual Resources

Tenant 2 Virtual Resources

Tenant x Virtual Resources

Cascaded OpenStack 2Cascaded OpenStack 1 Cascaded OpenStack y

https://tenant1.OpenStack/ https://tenant2.OpenStack/ https://tenantx.OpenStack/

OpenStack API

OpenStack API

OpenStack API

OpenStack APIOpenStack API OpenStack API OpenStack API

OpenStack APIOpenStack API

Fully distributed cascading layer: cascading OpenStack layer can be fully distributed if each small piece of cascading OpenStack only serve limited tenants. Each tenant can still has globe view for his distributed resources. Unlimited scalability can be achieved through this horizontal split.

… …

HUAWEI TECHNOLOGIES CO., LTD. Page 13Huawei Confidential

How to do that: just adding driver/agent/plugin each service to treat underneath OpenStack instance as its own backend. The cascading OpenStack becomes control layer only

Neutron Server

OVSAgent

Cinder API

Rabbit-MQ

Cinder Volume

Cinder Volume

Nova API

RabbitMQ

Nova Compute

Nova Compute

Libvirt Driver

Nova Driver

LVM Driver

Cinder Driver

KVM Nova LVM Cinder

RabbitMQ

OVS

L3Agent

Linux Router

Scheduler Scheduler

Neutron Agent

Neutron

Magic happens by just considering OpenStack as its own backend !

Glance Image1: Loc1: NFS Loc2: Glance1 Loc3:Glance2

Glance1

Glance2

Image1: Loc1: Ceph

Image1: Loc1: Ceph

Ceilometer API

hBase

StoreStore

CeilometerCeilometer

Nova as hypervisor Cinder as block storage Neutron as networking device Glance as Img. Location Ceilometer as store

** Architecture simplified for illustration only

HUAWEI TECHNOLOGIES CO., LTD. Page 14Huawei Confidential

Also solve OpenStack scalability at least 10 times level

OpenStack …

…

1 2 100

1 2 1000

OpenStack

…1 2 1000

OpenStack

…1 2 1000

OpenStack

Max. 100 data centers

Max. 100k physical server

nodes

Max. 1 million VMs

OpenStack API OpenStack API OpenStack API

OpenStack API

Scalability inside one data center,

multi-data centers or multi-sites

Test report: http://www.slideshare.net/JoeHuang7/test-report-for-open-stack-cascading-solution-to-support-1-million-v-ms-in-100-data-centers

HUAWEI TECHNOLOGIES CO., LTD. Page 15Huawei Confidential

More information:

Wiki: https://wiki.openstack.org/wiki/OpenStack_cascading_solutionPoC Source Code: https://github.com/stackforge/tricircle