the testing pyramid
DESCRIPTION
The Testing Pyramid. Nancy Chacko. About Me. Working in software testing for over 10 years Director of Software Engineering in Test at Opower Founding member of DCAST (DC Agile Software Testing group). http:// meetup.com /d- cast. Objective. - PowerPoint PPT PresentationTRANSCRIPT
The Testing PyramidNancy Chacko
About Me Working in software testing for
over 10 years
Director of Software Engineering in Test at Opower
Founding member of DCAST (DC Agile Software Testing group)
http://meetup.com/d-cast
Objective
We will explore how to get testers and developers to think about testing in a holistic manner.
By working together to create the set of tests and determine where that testing belongs, we can increase efficiency, reliability
and reduce debugging time for failures in the future.
Overview Testing and test engineers in an agile environment
Layers of testing and the pyramid
Uniting dev and test
Pushing tests to the lowest layers
Analysis
Design
Build
Test
Maintain
Agile Testing Emphasis on rapid development forces a focus on testing
TDD is often practiced by developers Automation is key to running fast
Testers transitioning to agile can often feel lost If developers are testing, what is my role?
Teams are not used to working together – the divide of waterfall can continue
Layers of Tests Unit tests
Tests at the method level, often use mock objects
Integration/service/component tests Tests at interfaces, often use real objects and DBs
End-to-end automated tests GUI tests that need to be run for regression testing
Manual/exploratory/ad hoc/GUI tests Used to fully exercise UI and explore unknowns
Typical Divide Development typically owns tests at the code level
Unit tests Integration tests
Testers often own the tests at the UI level Manual tests
Teams sometimes work together to automate the UI tests End-to-end automated tests
Typical Divide
Unit Tests*
Integration Tests*
End-To-End Automated Tests*
Manual Tests
Dev Test
Unit Tests*
Integration Tests*
End-to-End Automated Tests*
Manual Tests
Unit Tests*
Integration Tests*
End-To-End Automated Tests*
Manual Tests
Dev & Test
http://watirmelon.com/2011/06/10/yet-another-software-testing-pyramid/http://blog.mountaingoatsoftware.com/the-forgotten-layer-of-the-test-automation-pyramidhttp://benhutchison.wordpress.com/2008/03/20/automated-testing-2-the-test-pyramid/http://www.informit.com/articles/article.aspx?p=1433906&seqNum=3http://blogs.agilefaqs.com/2011/02/01/inverting-the-testing-pyramid/
Test Pyramid
Unit Tests*
Integration Tests*
End-to-End Automated Tests*
Manual TestsCl
oser
to C
ode Closer to Custom
er
Uniting Dev and Test Lack of communication often leads to trouble
Duplication of tests Tests performed at an inefficient layer Bugs are found later than they could be
Need to break teams out of old mindsets and get them to see the other side of the picture
What a Tester Sees Usually sees a functional spec
Simple login form that shows informative errors
From facebook.com
Test cases• Email address not
associated/a password• Does the page work in IE6?
Test CasesEmail address associated with an account/no password Email address with multiple @
Email address associated/correct password Email address with no @
Email address associated/incorrect password Other malformed email addresses
Email address not associated/no password Layout of the screen
Email address not associated/a password Password with unusual characters
Email address with a +/correct password What happens when JS is disabled?
Is the email or password field still filled in after submitting the form that results in an error?
Layout of the screen when there are errors displayed
Does the page work in IE6? What happens when cookies are disabled?
No email address/no password No email address/a password
From http://www.thebraidytester.com/downloads/YouAreNotDoneYet.pdf
Michael Hunter’s “You Are Not Done Yet”
What a Front End Developer Sees<html>
<js onSubmit validate email and password are not null><form onSubmit login>
<input email><input password>
</form>
</html>
What a Back End Developer Seesdef login
if email is nilredirect to login_form, set error_message to “Please enter email”
if password is nilredirect to login_form, set error_message to “Please enter a password”
hash_password = hash(password)
db_password = query_db(email)
if db_password is nilredirect to login_form, set error_message to “Invalid email or password”
if hash_password != db_passwordredirect to login_form, set error_message to “Invalid email or password”
set cookie, redirect to logged_in_page
end
Test CasesEmail address associated with an account/no password Password with unusual characters
Email address associated/correct password Email address associated but password in DB is null
Email address associated/incorrect password Email address associated but password in DB is empty string
Email address not associated/no password Slow DB connection
Email address not associated/a password Hashed password with unusual characters
Email address with a +/correct password No email address
Dev Both TestHashed password with unusual characters
Email associated with an account/no password
What happens when cookies are disabled?
No email address Email associated/correct password Does the page work in IE6?
Email associated but password in DB is null
Email associated/incorrect password No email address/no password
Email associated but password in DB is empty string
Email not associated/no password Email address with multiple @
Slow DB connection Email not associated/a password Email address with no @
Email with a +/correct password Other malformed email addresses
Password with unusual characters Layout of the screen
What happens when JS is disabled?
Layout of the screen when there are errors displayedIs the email or password field still filled in after submitting the form that results in an error?No email address/a password
Uniting Dev and Test Simply consolidating lists can help reduce testing while
maintaining quality levels
May actually be able to increase quality – new perspectives spark new tests
Test Pyramid
Unit Tests*
Integration Tests*
End-To-End Automated Tests*
Manual Tests
Dev Test
Unit Tests*
Integration Tests*
End-to-End Automated Tests*
Manual Tests
Unit Tests*
Integration Tests*
End-To-End Automated Tests*
Manual Tests
Dev & Test
What Layer for What Test Once a united set of tests is created, next step is to decide
where to put the tests
Based on what you want to verify, try and find the layer that allows you to do this with minimal interference
Push Testing As Low As It Can Go!
Benefits of Lower Level Testing More efficient: Tend to run in less time – no cost of starting a
browser or user wait times
More reliable: GUI tests fail inconsistently more often than unit/integration tests
Easier debugging: less components are involved so root cause is faster to fine
Multiple Layers Some tests will belong in multiple places
Time to start thinking about what to verify! Often the same setup will need to be verified at multiple layers
for different reasons
Sometimes a little redundancy is not a bad thing
Email address not associated/no password
Unit/Integration Verify that the “Please enter
a password” error message is returned
GUI/Manual Verify that JS error message
is shown and no request is sent to the server
Verify that with JS disabled, the server error is well formatted on screen
How to pick a layer Think about what is actually being tested
Find the layers of code that are actually involved Requires a good understanding of the architecture Best done as a collaboration between dev and test
Push the testing as low as you can
The higher the layer, the harder you should think about how to group tests together
Test MatrixTest Case U I G M
Email address associated with an account/no password X X
Email address associated with an account/correct password X X
Email address associated with an account/incorrect password X X
Email address not associated with an account/no password X
Email address not associated with an account/a password X
Email address with a +/correct password X X
Is the email or password field filled in when an error screen is shown? X
Does the page work in IE6? X
Layout of the screen X
Email associated but password in DB is empty string X
Test Matrix Becomes the basis for testing
Can easily divide work to be done now without overlaps
Need to remember the verification may differ at the different layers
Document may only live for the life of a few iterations
A little redundancy is ok
Manual/Exploratory Testing Often undervalued in an agile environment
Very useful for helping to answer the unknowns: “Does the page work in IE6?”
May lead to new tests to automate Automation is for tests you want to continually run Exploration helps ensure the gaps between those tests work
Group Higher Level Tests Think about the true goal of the tests
Often, they will be layout related as opposed to core functions Don’t need to test 10 error messages, just the ones that push the
boundary cases Longest message, message with unusual characters …
Test Pyramid
Unit Tests*
Integration Tests*
End-to-End Automated Tests*
Manual Tests
In Practice Need a good understanding of the underlying architecture
Get you dev and testers working together
Bullet point the tests that you think are valuable
Determine the layer for each test
Aggressively group the higher level tests together
Don’t underestimate the benefits of exploratory testing
Divide and conquer the work!
Thank you!