sfo15-tr6: server ecosystem day (part 6a)
TRANSCRIPT
Presented by
Date
Event
SFO15-TR6-A: Openstack Testing on ARM64
Clark LaughlinClark Laughlin
Wednesday 23 September 2015
SFO15
Agenda1. Introduction2. Progress since HKG153. Current testing effort
a. Why aren’t we running an Openstack 3rd-party CI setup?
b. openstack.linaro.org & current CIc. results
4. Issues and Next Steps5. Questions?
Openstack
Hopefully know what Openstack is.
If not…http://www.openstack.org/software/
Devstack
A set of scripts for deploying an Openstack developer environment. Makes many assumptions, and deploys a working, non-production Openstack deployment on a single node.
http://docs.openstack.org/developer/devstack/
Tempest
Integration test suite for verifying an existing Openstack deployment● API tests, scenarios (no benchmarking)● Running as part of a Devstack deployment is much
easier -- Devstack generates the (mostly) proper Tempest configuration
http://docs.openstack.org/developer/tempest/
Since HKG15...
Since HKG15...March 2015 September 2015
Distro/Version Ubuntu/Trusty Trusty Wily
Tests Run 713 1521 1522
Pass 470 (66%) 1431 (94%) 1432 (94%)
Fail 51 (7%) 23 (1.5%) 23 (1.5%)
Skip 192 (27%) 5 (0.3%) 5 (0.3%)
Ignored No analysis done 62 63
Not tested cinder, neutron
Since HKG15...
enabled cinder, neutron
switched to kernel 4.1.x
enabled missing CONFIG_*
virtio-scsi, cirros
Current Testing - Why Not What We Originally Planned?
We did go through the process of creating an Openstack 3rd-Party CI system as we had planned and presented at HKG15.
So why aren’t we using it?
Current Testing - Why Not What We Originally Planned?
● Requires more ARM64 nodes than we had available to scale
● Not enough personnel to support… we wanted lunch breaks and sleep
● Openstack’s third-party CI processes are not geared towards platform testing, which is what we wanted to accomplish
So, what ARE we doing?
● Nightly Tempest runs on ARM64 systems○ Jenkins Matrix Job:
■ branch: master, stable/kilo (latest released stable)
■ os distro: Ubuntu■ os version: trusty, vivid, wily
○ Devstack & Tempest deployed and run by LAVA job
So, what ARE we doing?
Results reported publicly:
http://openstack.linaro.org
Openstack.linaro.org: Overall Summary
Openstack.linaro.org: Historical Graph
Openstack.linaro.org: Single Test
Failing Tests
● Depending on the test combination, ~19-28 tests currently failing
● Many failures we have eliminated have been due to Tempest configuration or the cloud image we were using
● Causes of remaining failures have not been root-caused yet
Skipped Tests
● Expected:○ Depending on the test combination, ~60-90 tests are
expected to be skipped for various reasons○ Various reasons:
■ subsystem needed by test not installed, i.e. ironic, nova-network
■ tests disabled due to known Openstack bugs■ feature disabled, i.e. VNC console
Skipped Tests
● Unexpected:○ Currently, only about 5 tests are being skipped○ 3 are due to Tempest configuration settings that still need
to be changed○ 2 (EC2 tests) are due to the cloud image we are using not
having ami/aki/ari manifest
Issues And Next StepsIssue Next Step(s)
Publicising Results - since we are not running as an official 3rd-party CI, our results are not readily visible to developers
Continue work with Openstack CI and Third-Party Testing Teams to determine the best way to share results
Hardware - currently only utilizing APM Mustang boards
Two HP Moonshot enclosures with 40 m400 cartridges are now available. Work to integrate these into the test environment.
Will you provide your hardware to include in our test environment?
Issues And Next Steps #2Issue Next Step(s)
Testing is not against a production deployment with HA/redundancy/failover -- everything is running on a single node
This is standard practice for normal Openstack CI. However, we are looking into testing against more production-quality deployments.
Only testing on Ubuntu Incorporate additional OS distros / versions● Debian● CentOS
Failure count still > 0 Continue to root-cause remaining failures
PresentationFinished.
Any Questions?
Presentation Finished.
Any questions?