201002 automation-1000 tests
TRANSCRIPT
-
8/6/2019 201002 Automation-1000 Tests
1/18
Automation - Getting through thefirst 1,000 tests
Test Types Covered: Functional Testing Fuzz Testing Load Testing Database Testing
By:
Jeremy Carey-dressler
-
8/6/2019 201002 Automation-1000 Tests
2/18
Overview
What is automation?
What you need
Product choices
How to design the system
Code samples
-
8/6/2019 201002 Automation-1000 Tests
3/18
What is Automation?
To replace or enhancehuman labor withmachines Wiktionary
Test Automation is meantto do a few things:
Perform tasks not possible oreasy to do
Works when testers are notavailable
Regression, Integration,acceptance testing
Find issues in the system
-
8/6/2019 201002 Automation-1000 Tests
4/18
Why Aren't Unit Tests Enough?
Only happy path or paths imagined at time ofwriting
Typically written by the developer
Does not verify things come together
Inputs dont change
Not what a user does unless you are selling a
API
Environment is ignored, data is simulated
Can't easily verify the view (particularly html)
-
8/6/2019 201002 Automation-1000 Tests
5/18
General Question For Automation
Consider the following:
How stable is the system or component (UI, DB,etc.)?
What are managements expectations? How long do you expect the product to last?
Who is writing the automation?
Is it a full time job?
How easy is it to create the automation? Are test hooks created?
How many custom controls exist in the app?
How stable is the test tool?
-
8/6/2019 201002 Automation-1000 Tests
6/18
Functional Automation
Typically UI based testing
Typically best bang for the buck Each test should be independent Well understood in both automated and manual
testing
Code should be so easy to understand that you
can run it manually 6 months later Rapid feedback
Never blind or faith based
One test boils down to verifying one thing
-
8/6/2019 201002 Automation-1000 Tests
7/18
Setting up for automation
Software (which should have hooks forautomated)
Available simulations of a customer environment
(VMs, etc) Installed software (unless it is planned to
automate the install)
Test Data Manual Tests
Budget / HR / Willing Management
-
8/6/2019 201002 Automation-1000 Tests
8/18
Record and Playback
Useful for...
Learning thesystem
Developing ideas Manual tests
Quick and dirtytests
Small test sets
Not useful for...
Long termautomation
Framework-based code
Large test sets
Changing codebase
-
8/6/2019 201002 Automation-1000 Tests
9/18
API vs All-in-one Tools
API Pros
Not tied to a toolset
Typically better forframeworks
More customizable
API Cons
No reporting Harder to run in
multipleenvironments
All-in-one Pros
Recorder, reportingand running all builtin
Single cost All-in-one Cons
Typically one wayto do things
Hard to developlarge set ofautomation
Recorder-based
-
8/6/2019 201002 Automation-1000 Tests
10/18
Open Source Options
Slick Test Developer
http://sourceforge.net/projects/slicktest/
WatiN
http://watin.sourceforge.net/ Watir
http://watir.com/
AutoIt
http://www.autoitscript.com/autoit3/
Nunit
http://www.nunit.org/index.php
http://watir.com/http://www.autoitscript.com/autoit3/http://www.autoitscript.com/autoit3/http://watir.com/ -
8/6/2019 201002 Automation-1000 Tests
11/18
xUnit
Setup
Class
Test (Method)
Clean up
Class
Test (Method)
When test fails by exception or assert, the testscleanup is called
-
8/6/2019 201002 Automation-1000 Tests
12/18
Framework, framework, framework
Spend time designing the back end
Consider how the UI is likely to change
Look at the cost/benefits in flexibility,readability, runtime speed, anddevelopment speed.
Hide UI implementation (Called a
description, UI Map, ID, etc)
Separate concerns such as Actions,Tests/Asserts, Steps and the actual test
-
8/6/2019 201002 Automation-1000 Tests
13/18
General Design Architecture
Establish hierarchy and levels of division for tests in aclass by page, module, test type or functionality
Define Page/UI to map a set of elements to a live UI
Define Steps which are a collection of Actions
Define Actions to do one single thing in the UI
Define Data to provide hardcoded, dynamic or random
data that will be used in testing. Define Test/Assert to verify elements
-
8/6/2019 201002 Automation-1000 Tests
14/18
General Design Architecture Details
Use Magic-words (5 or less) to make work easier
Example: Test.Steps.*, UI.Page.*, DBTest.Data, etc.
Allows for separate people to do tests and framework
If the UI does NOT change often, you can build up the Elements Example: UI.Admin.Settings.Save()
If the UI does change often, you are better off having a moreflat structure
Example: UI.AdminSettings.Save() This method works best with Functional or Database testing
-
8/6/2019 201002 Automation-1000 Tests
15/18
Load Testing
There are many forms of Load Testing: Performance / Capacity
Endurance
Stress Testing
Spike
Configuration
In most cases, 10-20 test cases capture 70+% of your user base'sactivities
Load Tests are less developed compared to functional testing
Expect them to cost more per test
Rarely will you need more than 100 cases for a single application
Some can get away with just one or two tests
-
8/6/2019 201002 Automation-1000 Tests
16/18
Fuzz Testing
Automation with random or pseudo-randominputs
Not intended to find functional bugs
You can perform this by API, UI, DB inputs, fileformats or network protocols
Send valid, mostly-valid, invalid or a mixture in
fuzz testing Can be mixed with other testing and can be
helpful for increasing code coverage
-
8/6/2019 201002 Automation-1000 Tests
17/18
Acknowledgments
McDonalds Image:http://www.flickr.com/photos/xclozano/2913341645/
Some advice inspired by http://www.kaner.com/testarch.html andhttp://www.io.com/~wazmo/papers/seven_steps.html
Unit Test Information: http://www.youtube.com/watch?v=Pq6LHFM4JvE
People from past and present QA groups I have worked with
Tools you may have seen today:
Microsoft Visual Studios
Sharp Develop
Slick Test Developer
WatiN WatiR
Nunit
Resharper
http://www.flickr.com/photos/xclozano/2913341645/http://www.kaner.com/testarch.htmlhttp://www.io.com/~wazmo/papers/seven_steps.htmlhttp://www.youtube.com/watch?v=Pq6LHFM4JvEhttp://www.youtube.com/watch?v=Pq6LHFM4JvEhttp://www.io.com/~wazmo/papers/seven_steps.htmlhttp://www.kaner.com/testarch.htmlhttp://www.flickr.com/photos/xclozano/2913341645/ -
8/6/2019 201002 Automation-1000 Tests
18/18
Code!
Practical code samples