acceptance testing in devops
DESCRIPTION
To support efficiently ATDD or BDD at the speed and scale required in Agile and DevOps projects, your acceptance criteria need to: - Be understandable by business experts, Product owners, developers (defining the business terminology) - Facilitates test authoring as well as maintenance (based on a semantic and not just text) - Can be transformed automatically into code for test executionTRANSCRIPT
Laurent PYCEO, Smartesting
[email protected] @py_laurent
www.smartesting.com
Acceptance testing in DevOps
Our long and painful journeytowards DevOps
Product: CertifyIt, eclipse plug-in in JAVA (Model-Based-Testing)
– Waterfall process
– 1 release every 6 months
– Few tests and no TDD
– 1 month (x5 people) to do release testing before deployment
– Poor quality impacting customers’ feedback and adoption
Overview of our development process 2004/06
3
Product: CertifyIt, eclipse plug-in in JAVA (Model-Based-Testing)
– First XP and then Scrum process
– We do TDD
– Continuous integration (code & unit test)
– 1 customer release every 3 months (1 operation release every 3 weeks)
– 1 man/month to do release testing before deployment
– Good feedback from customers!
Tasting agility end of 2006
4
Product: Zest, testing platform in the cloud
– Still Scrum process
– We do TDD and…
– …Acceptance Testing Driven Development (ATDD), 100% automated
– Acceptance tests are part of the CI process
– We do several deployments a day (≈10)
Now we are experiencing DevOps - 2012
5
Shorter iterations (1 to 4 weeks) lead to massive test automation completed by exploratory testing
Acceptance tests become the specification
Testing starts earlier in the development process: Shift left!
What we’ve learnt
Req Manageme
nt & Definition
Test Planning
Execution
Defect manageme
nt
What we’ve learnt
7
To achieve this level of speed (DevOps context), acceptance tests should be:
– Understandable to both developers and business experts to strengthen alignment and enable the ‘shift left’
– Maintainable to manage efficiently changes in requirements and keep pace with continuous deployment
– Automated to enable rapid execution and feed-back
ATDD with Business Domain Language and refactoring capabilities
ATDD & Refactoring
My view as product owner & tester:
Acceptance tests need to be continually reviewed and refactored
just like code!!!
ATDD & Refactoring
My view as product owner & tester:
Acceptance tests need to be continually reviewed and refactored
just like code!!!
Martin Fowler
Acceptance testing driven development
Shift left
PO developers testers Scrummaster
Scrum team
Acceptance testingAcceptance testingShift left
Product Backlog
Sprint Backlog Sprint 1 to 4 weeks
Product at the end of the iteration
Daily meeting
Acceptance Testing Driven Development (ATDD)
Acceptance test is a powerful artifact to improve communication
Test as the definition of ‘STOP’
Written prior to development by tester
Based on a business DSL (domain specific Language)
Confirmed with stakeholders
Mostly automated
Test in natural language
Test fixture
Code
Acceptance Testing Driven Development (ATDD)
Benefits:– Improve communication and collaboration between
project stakeholders
– Shared understanding of what a successful implementation means
– Better coverage of business expectations
– Faster feed back
Challenges:– New methodology that requires rigor and discipline
– Find the right balance between people/process/tool
Continuous acceptance testing
Key features:
– Define progressively your Action Words and your business domain language for test authoring
– Suggestion to promote the reuse of Action Word and avoid duplication
– Refactoring: when an Action Word is modified, impacts are performed automatically across the tests
– Optimization: inspection features enable to continuously optimize the acceptance tests
– Create scripts and accelerate test automation (Ruby, Java, XML…)
Integrations with:
Zest: create acceptance tests with DSL
15
Watir, Appium…
Collaboration through acceptance tests & DSL
16
Product OwnerValidate acceptance tests
TesterCreate acceptanceTests
DeveloperAutomate acceptancetests
• DSL• Refactoring capabilities
Define new business entities…
Define progressively your business terminology with Action Words. Enable collaboration based on acceptance tests.
…or define business entities right from the tests
Promote test steps into Action Words (what developers call extract function). This is refactoring!
Evils of duplication
A fundamental principle
Reuse, reuse and reuse Action Words!
Create and maintain consistent scenarios for your project
Suggestions
Analyse and optimize continuously your tests
When duplications are identified, refactoring options are suggested!
Add, remove, modify Action Words and Scenarios
Test refactoring enables to perform automatically impact analysis and test maintenance tasks
Add parameters to Action Word
Propagate automatically tothe scenarios
Generate scripts
Use of Action Words significantly decrease the cost of automation and accelerate the overall testing cycle
So your acceptance tests are…
UnderstandableDefinition of business domain tests enable better alignment of the team
MaintainableRefactoring and optimization
features dramatically accelerate maintenance
Can be automatedGood design based on Action Words streamlines the test
automation phase
• DSL• Refactoring capabilities
And remember
Acceptance tests need to be continually reviewed and
refactored just like code!!!
Give a tryZest-testing.com
Laurent PYCEO, Smartesting
[email protected] @py_laurent
www.smartesting.com