The Road to Enterprise-Ready

OpenStack Storage as Service

Today’s Presenters

Sean Cohen Jon Bernard Flavio Percoco

Principal Product ManagerRed Hat

Senior Software EngineerRed Hat

Senior Software EngineerRed Hat

2

THANK YOU

High Availability

DIVIDERHigh Availability

● High Availability of APIs/Services Goals

○ All services that power the OpenStack APIs should be always on, and able to always respond even during failure and massive stress.

○ Provide protection against HW & SW SPOFs (Single Points of Failure)

DIVIDERHigh Availability

● Where are we today?○ There are cases where the volume is left in

unrecoverable state and it is not possible to delete the volume without administrator’s intervention

○ If a cinder-volume node dies during volume create request processing for example, the volume will be in unresolved state.

DIVIDERHigh Availability

● Where are we today?○ Cinder volume service run in Active/Active state

which is not safe■ Non-Atomic state transitions in the API may end in

race conditions.■ Consistent replications of these nodes are currently

not possible.

DIVIDERHigh Availability... new in Kilo

● Cinder - iSCSI Multipath■ nova-compute support multipath for iSCSI volume data

path. However, some arrays only respond to discovery with a single portal address, even if secondary portals are available.

■ Cinder now can return multiple iSCSI paths information so that the connector can attach volumes even when the primary path is down.

■ Cinder side was completed in Kilo, while the Nova enablement work is still ahead.

Instance Migration

Horizon - Migrate all instances from host

● Allow administrators to migrate all instances from host marked for maintenance via Horizon in a “Push button” fashion as available in command line.

● “Migrate instances” button allow administrators to use simpler way of preparing host for maintenance actions in Horizon○ Useful in upgrades scenarios.○ Test/perform manual disaster recovery.

https://blueprints.launchpad.net/cinder/+spec/cinder-state-enforcer

DIVIDERThe Road to Active-Active

● Cinder State Enforcer ○ Long standing work to improve Cinder volume’s states

management and reliability, and to improve failure tolerance.

○ In order to mitigate the concurrent resource access problems in Cinder, work was done in the last cycles to refactor the concept of a lock to be a set of allowed and disallowed state transitions (instead of acquiring local filesystem locks in the manager processes) by implementing a new `enforcer` model.

DIVIDERThe Road to Active-Active

● Active / Active cinder-volume○ Effort in Liberty cycle to address issues around:

■ Local file locks in cinder-volume - need to enhance the lock reporting to Nova based on the volume active state.

■ DB accesses in drivers - need to be minimized or limited at all.

■ Nova is inspecting internal state of cinder volumes to determine if it can take an action, rather than properly delegating the attach/detach work.

DIVIDERThe Road to Active-Active

● Task Flow for managing create volumes tasks○ There are few corner cases where the volume is left in

unrecoverable state and it is not possible to delete the volume without administrator’s intervention.

○ The improvements of state management can get us a step closer to Active-Active safe operations.

○ The road to Active-Active should be spreading the use of state management (taskflows) to to cover Cinder operations beyond volume creation tasks

DIVIDERVolume Management

● Cinder - Attach a single volume to multiple hosts○ In order to support Hypervisor/Application clusters level, a

single volume would need to be exported to multiple host. ○ The patch that adds the multiattach flag to volumes was

merged in Cinder during the Kilo release.○ However we are still missing the Nova and python-

cinderclient patches to provide support for multiple attachments.

DIVIDERVolume Management

● Volume Migration - Retype initiated

○ One of the biggest problems around volume migration is the confusion around volume migration and retype.

■ Volume Retype will trigger a migration only if a user has requested it (not by default)

DIVIDERVolume Management in Liberty

● Volume Migration with file I/O instead of iSCSI attachment○ Currently when migrating a volume between two backends,

the copy_volume_data routine in the source volume's driver is executed to move the blocks from one volume to another. This routine assumes that both source and destination volumes can be attached locally (e.g. iSCSI)

○ Add the ability to migrate volumes of drivers that don’t support iSCSI such as Ceph RBD.

Business Continuity

Backup improvements

Incremental backup ● Cinder Backup API was extended to support snapshot based

backups, where the volume can remain online and in-use for the duration of the operation. (Swift or NFS target)○ The enhancement also included performing a backup from a

snapshots. ○ New cinder backup CLI was added: --incremental or --incr○ swift.py creates sha256 file for every backup to calculate deltas○ During restore, if a differential backup needs to be restored, the

restore process first restores the full backup.

Backup improvements

NFS & POSIX Backup • Cinder Backup has now support to use NFS/POSIX supplied

data repository as backup target with two new drivers in place.

Backup Support for Encrypted Volumes• The Cinder backup includes now a clone of the volume's

encryption key UUID so that the encryption key is available when the backup is restored.

Backup improvements

Nova - Support for quiescing file-systems during image snapshot• Using QEMU guest agent• With this new feature, users can create a snapshot image with

consistent file systems state while the instances are running (it requires QEMU Guest Agent to be installed in a KVM instance).

• Useful for taking a quick backup before installing or upgrading softwares / Can set to run automatically every night etc.

Backup improvements

Swift - Erasure Coding• Erasure coding is a storage policy designed to reduce

storage costs associated with massive amounts of data (by providing an option that maintains the same, or better, level of durability using much less disk space)

• Can be very useful when performing volume backup to a Swift object storage system, as backups are typically large compressed objects and are infrequently read once they have been written to the storage system.

DIVIDERBackup improvements… in Liberty

● Cinder - Scaling Backup Service○ Currently the Backup service must scale up rather than out.○ The Backup service and cinder volume drivers are coupled

so that all must run together on a single node - By breaking the coupling between backup service and volume drivers, it will allow the service to scale out.

● Swift - Fast Posting○ Where a POST to an object will trigger a container update.○ Allow for updating objects metadata through POST

semantics and still guarantee data consistency in the container.

Disaster recovery

Disaster Recovery

Cinder - Consistency groups enhancements• Added the ability to add/remove volumes from an existing

consistency group.• Added the ability to create a consistency group from an

existing consistency group snapshot.• Support creation of a cg_volume types table (to overcome the

limitation of the current solution which stores all volume type uuids in one column of the CG table).

Disaster Recovery… in Liberty The OpenStack snapshot mechanism allows you to create new images from running instances. This is very convenient for upgrading base images or for taking a published image and customizing it for local use. But what about external use?

Cinder - Import/Export snapshots● Allows to import volumes snapshot from one Cinder to

another.● Allows to import "non" openstack snapshots already on a

backend-device. Where, export snapshots should work the same way as export volumes.

Disaster Recovery… in LibertyCinder - Volume Replication V2● Things we’ve missed in V1:

■ Replication between Cinders ● Currently we have basic replication in a single

Cinder deployment.■ Consistency data replication

● Align CG design and volume-replication spec, one CG could support different volume-types, where the volume-type to decide which volume-replication is going to be created and added to CG.

DIVIDERDeployment & Rolling Upgrades

● Image Introspection○ A new task has been added to Glance’s v2, which makes

it possible for introspecting image’s metadata and populate it.

● Image Conversion○ A new task has been added to Glance’s v2, which makes

it possible for converting images on import.○ Useful to unify stored image types and use a type that

works better with the hypervisor and the storage backend.○ Current supported formats are: raw <-> qcow2

Deployment & Rolling Upgrades

DIVIDERDeployment & Rolling Upgrades

● Cinder DB Purge Utility○ Very long lived Openstack installations will carry around

database rows for years and years.

○ Operators need to have the ability to purge deleted rows, possibly on a schedule (cron job) or as needed before an upgrade, prior to maintenance.

○ The new utility allows you to clean up rows that are already marked as deleted of a certain specified age.■ The age is calculated as timedelta in days, which are

given at command line.

DIVIDERDeployment & Rolling Upgrades

● Implement force_detach to allow safe cleanup of stuck volumes○ For volumes stuck in 'attaching' or 'detaching' , there is no

safe way to cleanup that involves the backend storage. ○ Using python-cinderclient 'reset-state' will only change the

Cinder database, and may leave the volume exported to the compute host, and may leave an entry in Nova's database that prevents the volume from being re-used.

○ This also need to be addressed by the Nova side.

DIVIDERDeployment & Rolling Upgrades … in Liberty

● Cinder Objects ○ Supporting rolling upgrades by using versioned objects.○ These objects are isolated from the schema and contain

the required information for communications and operations.

○ These objects can be sent over RPC.○ Work started in Kilo.

DIVIDERDeployment & Rolling Upgrades … in Liberty

● Cinder Storage Policies - Standard Capabilities○ Goals:

■ Provide standard capabilities from drivers that the cloud administrator can specify from volume types.

■ Improve the visibility of what policies are possible with your storage solution via Cinder client and Horizon.

○ Capabilities need to be exposed to the admin from Cinder, so that Cinder is not limiting what storage backends can do. ■ These capabilities can include, but are not limited to QoS,

replication factor, bandwidth control, etc.

Security

DIVIDERSecurity

● Cinder - Private Volume Types○ With the new Cinder ability for defining private volume types, as

some volume types should only be restricted. ■ Private volumes for special needs where most users should

not be able to select these volumes.■ Volume types are public by default■ Private volume types can be created by setting the is_public

boolean field to False at creation time.■ Access to a private volume type can be controlled by adding

or removing a project from it.

DIVIDERSecurity… in Liberty

● Glance - Image Signing and Encryption○ Right now, there is no way to guarantee that image you asked Glance

for is the image you got in Nova.○ This feature has been discussed in the past and it looks like it’ll finally

happen.○ Image signing and encryption using Barbican as a key manager.○ The goal is to guarantee image’s integrity.

● Horizon - Volume Encryption

■ Support for volume encryption through Horizon is almost there. Some of the work is done but it was moved out of Kilo at the very end.

DIVIDERSecurity… in Liberty

● Swift - Encryption At Rest○ Currently objects are typically stored on disk as files in a standard

POSIX filesystem.○ Provide option for Swift operators to have objects stored in an

encrypted form.○ When disks reach end-of-life, they are discarded, and if not properly

wiped, may still contain data. ○ Swift will use AES in CTR mode with 256-bit keys, where the entire

object is encrypted as a single byte stream, as well as user metadata with the same key.

○ Swift will probably want a keymaster that stores things in Barbican at some point.

DIVIDERSecurity… in Liberty

● Swift - Composite Tokens & Service Accounts○ Composite tokens allow other OpenStack services to store data

in Swift on behalf of a client so that neither the client nor the service can update the data without both parties consent.

○ Example: ■ User requests that Nova save a snapshot of a VM. ■ Nova passes the request to Glance■ Glance writes the image to a Swift container as a set of objects. ■ The user cannot modify the snapshot without also having a valid token

from the service.■ Nor can the service update the data without a valid token from the

user.

Q & A