lca13: android infrastructure automation improvements
DESCRIPTION
Resource: LCA13 Name: Android Infrastructure Automation Improvements Date: 05-03-2013 Speaker: Tyler BakerTRANSCRIPT
ASIA 2013 (LCA13)
Tyler Baker – March 2013
LAVA Workshop @ LCA13 A LAVA Success Story
ASIA 2013 (LCA13)
www.linaro.org
Introduction
Tyler Baker – Former Android Platform Engineer at Intermec Started working with Android early 2008
Board bring ups on custom hardware designs
Boot loader porting
Development of BSP and device drivers
1D/2D Imaging solutions
System services and application development
RIL integration and debugging
System administrator for Gerrit, Jenkins, and LAVA
Android system internals course instructor at Marakana
Studied Computer Science and Applied Mathematics at Western Washington University
In my spare time I enjoy designing autonomous aerial drones, snowboarding, and fishing.
Located in Bellingham, WA
ASIA 2013 (LCA13)
www.linaro.org
Perspective
Working for an OEM designing and developing enterprise products has given me unique perspective
The intent of this presentation is to highlight the success I have had utilizing LAVA
Takeaway Points Continuous Integration Philosophy
What are the pain points associated with existing test frameworks
Why LAVA was adopted
How LAVA can be easily extended
How LAVA can be integrated into CI environments
Why everyone here should consider LAVA
ASIA 2013 (LCA13)
www.linaro.org
Continuous Integration Philosophy
Imagine CI as the LIM(x->0) of 1/x There is a vertical asymptote at x = 0
The function increases without bound
Think of true CI being the asymptote
Select a number close to zero, and I will select one that is closer
ASIA 2013 (LCA13)
www.linaro.org
Continuous Integration Philosophy Continued
Achieving true CI is very difficult and some say impossible There is always something else that can be automated
CI practices and tools are constantly evolving
New hardware platforms constantly drive CI software changes
Early stages of hardware and software development can be unstable
Boot loader CI?
What is the return on investment?
In my opinion, the best approach to CI is to evolve your strategy as the technologies changes
ASIA 2013 (LCA13)
www.linaro.org
Pain Points
Seemingly many companies build their own automated test framework and would like you to purchase it Intellectual property
Licensing
Closed Source
Paid support
Costly integration efforts
Technical debt incurred to modify the system
Integration with corporate IT infrastructure I.E (LDAP)
Scalability?
Extensibility?
Multi platform?
Reporting?
Dashboards?
KPI?
There is no perfect solution yet
ASIA 2013 (LCA13)
www.linaro.org
Linaro Automated Validation Architecture
I was first introduced to LAVA at ELC 2011
Began to add support for our rugged Android OMAP3 based handset shortly after
Why LAVA was adopted Flexible architecture
Compartmentalized
Easy to extend to meet the any need
Automated recovery of bricked devices
Well supported by LAVA team
Tests already integrated
Open Source
LAVA itself is an enabler
Contributing changes upstream reduces technical debt
ASIA 2013 (LCA13)
www.linaro.org
LAVA Extensions
CLI tools are wrapped, exposed as actions to the dispatcher
Very simple to extend the existing software once you understand entry points
Developers who write services or applications should also be able to take advantage of LAVA
The issues that needed to be solved Facilitate downloading, installing and uninstalling applications/system
services
Download and deploy files to the file system
Run commands on the client side over ADB
Run unit or functional tests and get the the results
ASIA 2013 (LCA13)
www.linaro.org
LAVA Extensions Continued
Extended lava-android-test Download and push files to a specific location on the file system
Install, uninstall, and reinstall applications
Run client side shell commands
insmod, chmod, reboot
Download, install, and run any Gtest or Junit binary with correct parse
Can be referenced in a job file
Was able to successfully test Android OTA updates
ASIA 2013 (LCA13)
www.linaro.org
LAVA and LDAP
To manage existing user accounts an extension was needed to allow LDAP authentication
Gerrit and Jenkins both support LDAP
LAVA server extension written in a single day
Used the OpenID authentication implementation as reference
Straight forward
LAVA can adapt to almost any need
ASIA 2013 (LCA13)
www.linaro.org
LAVA and Jenkins Integration
Having Jenkins submit a job to LAVA after a build is trivial It would be nice to have a plug-in, but not necessary
LAVA CLI tools work just fine
The issues When you submit a job in LAVA it is asynchronous operation
LAVA dashboard holds the test results
The developer must seek out the test results for his/her build
This violates the “rapid feedback loop” CI principle
How will LAVA be instructed to download the correct binaries from Jenkins in a dynamic fashion when the job file is static?
ASIA 2013 (LCA13)
www.linaro.org
LAVA and Jenkins Integration Continued
Extended lava-tool Created a function called “testjob-run”
It schedules the job
It blocks like a synchronous operation
It monitors the job progress
When the job completes, the results are fed to STDOUT
There is also a option to write the results out into xUnit format
Now the results can be visualized right from Jenkins!
Not only just graphs, but the same data you might find in a results bundle
This data can be pushed into an existing test results database
ASIA 2013 (LCA13)
www.linaro.org
LAVA and Jenkins Integration Continued
ASIA 2013 (LCA13)
www.linaro.org
LAVA and Jenkins Integration Continued
ASIA 2013 (LCA13)
www.linaro.org
LAVA and Jenkins Integration Continued
How are the job files being dynamically created? The job files used are “templates”
sed is used to replace “variables” in the LAVA job template
This allows the dynamic creation of Jenkins links in the job template submitted to LAVA
A bit of a hack, but so far been very robust
When the upstream build finishes, the modified job template(s) are downloaded to the downstream LAVA test job
These modified job templates are scheduled and results pushed up into Gerrit
ASIA 2013 (LCA13)
www.linaro.org
LAVA and Gerrit Integration
Jenkins can listen for Gerrit events to trigger builds Once triggered Jenkins will download the patch set, applied it to the
source tree and build
This will allow a developer build and test their code before ever merging
Since the test results are now being reported to Jenkins It is possible to use the Gerrit SSH API to “code review” the change set that got built and
tested
A thin bash wrapper parses the STDOUT from the lava-tool extension
The failed results get put into the code review message and -1 is applied to the change set
If all the results pass a +1 is applied to the change set
If the is job status is returned as “incomplete” then 0 is applied to the change set
In every case a link to the LAVA test bundle is included in the message
The feedback from the build and the test(s) will be reported all in one place
This is what CI is all about, giving the developer timely feedback about his/her changes
ASIA 2013 (LCA13)
www.linaro.org
LAVA and Gerrit Integration Continued
ASIA 2013 (LCA13)
www.linaro.org
LAVA and Gerrit Integration Continued
ASIA 2013 (LCA13)
www.linaro.org
CI Architecture Overview
ASIA 2013 (LCA13)
www.linaro.org
LAVA Canary Testing
Build machines can build boot loaders, kernel, modules, and Android file system in less than two minutes Change set applied
Dirty build, no CCACHE
18 minutes to build all components from a clean 4.2.1 tree
Quad Xeon processors with 48GB of RAM per processor
Building on a ramdisk
All units can recover from a bad kernel or file system
Once builds are smoked tested on the canary units, they are pushed out to larger populations as OTA updates
Commit, build, and receipt of test results takes just under 8 minutes This includes full OS build, downloading the build to LAVA server, flashing
full OS using fastboot, booting, running a quick smoke testing which includes monkey
ASIA 2013 (LCA13)
www.linaro.org
LAVA Multi Platform Support
Built a extension in a matter of days to support the Windows Mobile platform Flashing of the OS
Running tests
Reporting results
IOS support is being implemented
ASIA 2013 (LCA13)
www.linaro.org
Conclusion
Planning to upstream all the extensions mentioned in this presentation for the benefit of the community
LAVA may not do everything you want but can be easily extended
LAVA is easy to use
LAVA has a great team of developers
LAVA can be tightly integrated into a CI environment
I would personally like to see LAVA emerge as the gold standard for automated testing frameworks
ASIA 2013 (LCA13)
www.linaro.org
Thank you!
You can reach me at [email protected] or on IRC as tyler-baker
LAVA Hands On – Wednesday 4pm - 6pm
• Requirements
• Laptop
At least 4GB Ram
Virtualisation support (i.e. Intel i3-i7)
• Software
VirtualBox 4.1 or later (Linux, Windows and OS X
supportedhttps://www.virtualbox.org/
LAVA VM Image
http://deb.li/lavavm
• Limited space, but some spaces available
• E-mail [email protected] and turn up
More about Linaro Connect: www.linaro.org/connect/ More about Linaro: www.linaro.org/about/
More about Linaro engineering: www.linaro.org/engineering/
ASIA 2013 (LCA13)