introduction to achieving performance and scale on...

Introduction to Achieving Performance and Scale on Large OpenStack Clouds

Alec Hothan, Principal Engineer

DEVNET-1693

• OpenStack Performance and Scale: challenges and existing tools

• A new generation of fully integrated tools to measure tenant network and storage at scale

• Design of these tools

• What they measure and how

• How to read results

• Free and open-source: install and use the tools yourself

Agenda

OpenStack Performance and ScaleTools and challenges

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 4DEVNET-1693

OpenStack Performance Metrics

Control Plane Metrics

Resource operations

Storage/Data PlaneTraffic metrics

Host system metrics

Networking

Compute Storage

tenants, userscpu, memory… networks

routers…

volumesimages…

Bare Metal


Measuring these metrics is challenging

Abundance of tools

Complex measuring conditions

Automation is key

ps, top, strace, ptrace, perf…

iperf, netperf, nuttcp…

fio, jmeter, siege, wrk, cb

OpenStack rally, tempest

Cloudbench, PerfKitBenchmarker…

OpenStack variants/integration

Cloud scale

Failure conditions

Repeatable

Consistent


Example: web services and user experience

Density

Throughput

Latency

HTTP servers per compute node

HTTP requests per second

Latency percentile requirements:

User perception…

90th %ile 99th %ile Max

Fast

< 0.1 sec

Noticeable

< 1 sec

Slow

< 10 sec


Latency histogram (each traffic generator)

Latency

histogram

min

90th %ile

99th %ile

max


Latency histogram aggregation

+N x latency histograms

(distributed)

Overall system latency

histograms


Latency histograms at cloud scale

Number of HTTP servers per cloud 1000

Number of users per HTTP server x 1000

Number of requests per user/sec x 1

Test duration in seconds x 100

Total number of requests 100M

Number of latency histograms to consolidate

(1000 users/HTTP traffic generator)1000

A new generation of performance/scale tools


Tools characteristics

Fully integrated

End to end

Accurate and precise

Tight integration with OpenStack

Automated

VM and user level metrics

All batteries included

< msec precision

Correct consolidation of metrics

Simple and AccessibleInstallation

Usage

Result interpretation


OpenStack tools for performance/scale

Control Plane API-level

Performance

Single Flow Performance

KloudBuster

Networking

Compute Storage

Rally

High Scale Performance

VMTP

✅

✅

✅ ✅

✅ ✅

(coming soon)NFV Data Plane✅


Neutron Tenant Network Flows

VMs can run on same or

different compute nodes

North-South flows (NAT)

East-West flows

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

Networking

Compute Storage

14DEVNET-1693

VMTP

• Single flow VM-level performance

• TCP

• UDP unicast and multicast

• ICMP

• Tenant network flows

• East-West and North-South

• Intra-node and inter-node

Validation and Performance Measurement of Network Flows

• Encapsulation and Neutron implementation agnostic

• Also supports

• Provider network

• SR-IOV NICs


VMTP Throughput Chart


KloudBuster

• Configure:

• Number of HTTP servers (1 per VM)

• Number of HTTP clients per server

• HTTP requests/sec

• VM placement policy

• Stages and orchestrates everything

• Simple runs

• Progression runs (HTTP servers)

• Real time latency monitoring

OpenStack Data Plane Performance at Scale

• Aggregates and consolidates all results

• Total throughput

• Micro-sec precision latency histograms• HDR (High Dynamic Range) histogram

library

Networking

Compute Storage


VM VM VM

VM

proxy

VMVM

KB

17DEVNET-1693

Data Plane Scale Overview (East-West 1 cloud)

N x http servers

N x (http traffic generators (m users))

East-West L3 traffic

N x m flows

staging

orchestration

results aggregation

Internal network

tenant


proxy VM VM VM VM

VM VMVM

VM

KB

18DEVNET-1693

Data Plane Scale Overview (North-South 2 clouds)

N x http servers

N x (http traffic generators (m users))

North-South traffic

N x m flows

staging

orchestration

results aggregation


Latency Chart: 1K to 20K HTTP Connections


Real-Time Latency Monitoring (vPC failure)One protected

interface going down

Latency stacked

iso-percentile

scrolling layers


KloudBuster

• Configure:

• VM count

• Cinder volumes (block storage) or ephemeral disks

• Storage workload definition

• Stages and orchestrates everything

• Progression runs (VM count)

OpenStack Storage Performance at Scale

• Aggregates and consolidates all results

• Total throughput

• micro-second precision latency histograms (HDR)

Networking

Compute Storage


KloudBuster Storage Scale Test Overview

ephemeral

or block storagestorage network

tenant networkstaging

orchestration

results aggregation

N x Fio testers


Generated Charts (NFVi profile/Ceph backend)

progressions

concurrent VMs

Latency iso-percentile

line charts

workload results tabs


High IOPs profile (61K IOPs random read Cinder)


High Throughput profile (3,743 MB/s seq read Cinder)

Installation and User Interfaces


Installation and User InterfacesVMTP KloudBuster

Docker Container docker pull berrypatch/vmtp docker pull berrypatch/kloudbuster

PyPI installation (pip) pip install vmtp pip install kloudbuster

VM Image (REST/Web server) http://storage.apps.openstack.org/images/kloudbuster_v6.qcow2

git clone git clone https://github.com/openstack/vmtp.git git clone https://github.com/openstack/kloudbuster.git

CLI ☑️ ☑️

REST ☑️

Web UI ☑️

27DEVNET-1693


What’s next

• Meet us at the Cisco Live! DevNet OpenStack Demo Zone

• Video demos of VMTP and KloudBuster

• Contact: [email protected]

• VMTP: https://github.com/openstack/vmtp

• KloudBuster: https://github.com/openstack/kloudbuster

29DEVNET-1693

mailto:[email protected]


Complete Your Online Session Evaluation

Don’t forget: Cisco Live sessions will be available for viewing on-demand after the event at CiscoLive.com/Online

• Give us your feedback to be entered into a Daily Survey Drawing. A daily winner will receive a $750 Amazon gift card.

• Complete your session surveys through the Cisco Live mobile app or from the Session Catalog on CiscoLive.com/us.

30DEVNET-1693

CiscoLive.com/Online

http://ciscolive.com/Online

http://ciscolive.com/Online

http://ciscolive.com/us

http://ciscolive.com/us


Continue Your Education

• Demos in the Cisco campus

• Walk-in Self-Paced Labs

• Lunch & Learn

• Meet the Engineer 1:1 meetings

• Related sessions

31DEVNET-1693

Please join us for the Service Provider Innovation Talk featuring:

Yvette Kanouff | Senior Vice President and General Manager, SP Business

Joe Cozzolino | Senior Vice President, Cisco Services

Thursday, July 14th, 2016

11:30 am - 12:30pm, In the Oceanside A room

What to expect from this innovation talk

• Insights on market trends and forecasts

• Preview of key technologies and capabilities

• Innovative demonstrations of the latest and greatest products

• Better understanding of how Cisco can help you succeed

Register to attend the session live now or

watch the broadcast on cisco.com

Thank you

introduction to achieving performance and scale on...

Documents