Tempest: Integrated OpenStack Testing

Matthew Treinishmtreinish@kortar.org

May 12, 2014

OpenStack is Complicated

Testing before Tempest

I Every project had its own unit tests

I Some projects had functional tests

I There was nothing to test the whole picture

What is Tempest?

I The official OpenStack integration test suite

I Only black-box testing, only interacts with OpenStack REST APIs

I Over 2300 tests covering every incubated and integrated project

I Used as the gating tests for every integrated project

Tempest GrowthI Started in late Diablo to solve the lack of integrated testingI Became a gating test suite in the Folsom cycleI Was one of the 5 most active projects in IcehouseI Testing with tempest is now a project graduation requirementI Being used in refstack as part of defcore validation

Growing Pains

I Huge continued increase in the number of projects and tests

I The common code to handle all the APIs becomes more complex

I Managing individual tests becomes very difficult

I Total run time continues to grow

I Tempest configuration becomes more involved to handle all the services andconfigurations

Design Principles

I Should be able to run against any OpenStack Cloud from devstack to a publiccloud

I Everything explicit

I Only use public interfaces (REST APIs)

I Should be self cleaning

I Self Testing

Basic Tempest Architecture

API Tests

I Directly test the OpenStack REST APIs

I Uses a unified tempest REST client

I Fall into 2 categories positive and negative

I Negative tests are starting to be automatically generated from api schemas

Scenario Tests

I Through path tests of functionality

I Designed to simulate a typical end user action workflow

I Use official python-*clients

I Ideally will involve integration points between projects

CLI Tests

I Test the official client CLIs

I Quick sanity checks of functionality in an OpenStack environment

I Only perform read-only operations

I Mostly a matter of convenience for gating

Stress Tests

I Use tempest to generate load on a cloud

I Dedicated scenarios for generating load

I Can specify any existing tempest test to also be a stress generator

I All scenario tests are automatically added as stress tests

I Run stress tests serially or in parallel looping for a set period of time

3rd Party and Unit Tests

I 3rd Party TestsI Tests for in-tree non-OpenStack APIsI Currently only tests Nova’s EC2 implementation

I Unit TestsI Tempest eventually reached a size where its operations were too complex to take for

grantedI Started late in Havana because of bugs in wrapper scripts ”super passing” the gateI Unit testing coverage was expanded significantly during Icehouse

Parallel Tempest

I Tempest is capable of running it’s tests in parallel

I More closely simulates real load on a cloudI Greatly improves speed of running tests:

I Serially gate runs took 1 hr with <1000 testsI Current gate runs are 40min to 1 hr with 2300 tests

Issue with single user in parallel

Tenant Isolation and Parallel Execution

I Tenant isolation enables running each test class in isolation

I When enabled each test class creates an isolated user and tenant

I Execute tests in parallel at the class level

Branchless Tempest

I Starting with Icehouse tempest will no longer have stable branches

I For gating purposes we will use master tempest against stable branch proposedcommits

I All proposed tempest commits will run against master on the other projects andthe currently supported stable releases (Icehouse forward)

I This should help us ensure greater api consistency between releases

Tempest Configuration

I Around 200 config options currently

I Intent of config file is to tell tempest what can be run

I What should be run is the domain of the test runner

I Sample config file documents every option

I Intended to be able to cover any OpenStack configuration

I Should have sane defaults (ie match project defaults)I To setup a basic config file:

I Configuring only basic auth and connection info should be neededI Also specify flavor and image ids to use for creating serversI List which services are available and any required configuration for those services

Future Work

I Expansion of autogenerated negative tests

I More Unit test coverage

I Expansion of feature flags in config file

I Configuration file validation and generation tooling

I Possibly a new service for running tempest

Where to get more information

I Tempest Repository

I Tempest Documentation

I IRC #openstack-qa channel on Freenode

Questions?