the future of test automation worksoft, inc. © 2003 all rights reserved
TRANSCRIPT
The Future of Test AutomationThe Future of Test Automation
Worksoft, Inc.© 2003 All Rights Reserved
Agenda
Business Case for Test Automation– Why automate?
Test Automation Challenges– Common mistakes
Software Development Trends– Risk is increasing
Test Automation Trends– Technology is evolving
Future of Test Automation– Quality maturity
Business Case forTest Automation
Why Automate?
Save timeLeverage resourcesReduce costsIncrease coverage
Time
Repeatable tests– Multiple builds– Multiple versions
Unattended execution– Time to design more tests– Time to do manual testing
Concurrent testing– Write tests during requirements– Develop tests during coding
Resources
Reproducible tests– Audit trail of test results– Consistency from one tester to another
Higher volumes– Acquire test data from production– Generate data programmatically
Reduced rework– Catch errors early– Fewer emergencies, patches
Costs
Reduce failures in production– Loss of revenues, market share– Loss of customers, confidence– Loss of user productivity
Reduce support, rework costs– Costs to recover from errors– Costs to remedy defects– Costs to retest, redeploy
Coverage
Functionality increases over time– Inventory of existing features is cumulative– New features are added– Fixes must be verified– Ad hoc tests are needed
Do the math– Cycle times are shorter– Developers outnumber testers– A 10% software change requires 100% testing– A single release will have multiple builds
Features
Tests
Time#
Risk
Test Automation Challenges
Common Mistakes
You don’t have timeYou don’t have resourcesYou don’t understand costsYou can’t measure coverage
Time is Limited
Shorter cycle times– Schedules are market-driven, not effort-based
– Original schedule is always compressed
– Poor quality means longer cycles
Test automation takes time– It’s a development project
– 5X to 10X manual effort
– Ongoing maintenance effort
Resources are Limited
Recording tests takes manual effort– Tests have to be performed manually– You can’t start until software is ready
Scripting takes skill– Tools are programming languages– Automation is a development project
Implementation takes resources– Design, document tests– Develop framework, scripts– Execute, evaluate results
Costs Aren’t Fully Apparent
Cost of acquisition– Educate, evaluate, negotiate, license
Cost to implement– Planning, training, design, development,
documentation
– Requires development resources
Cost of ownership– Continuous maintenance, support
– Change, version control
– Training for new personnel
Coverage is Difficult to Measure
Requirements are missing, obsolete– Test cases are ad hoc– Test knowledge is organic
Regression tests are undefined– Existing functionality not captured– Past defects are forgotten
New features are informal– Poor or missing specifications– Casual or inconsistent hand-off
Software Development Trends
Risks are Increasing
Increased exposureDecreased cycle timesGreater functionalityMore complexity
Increased Exposure
Business rules exposed to users– Rules engines enable analyst involvement– More changes made faster
Applications exposed to customers– Web access becomes customer interface– Direct impact on credibility
Applications exposed to suppliers– Web services open applications– Direct impact on production
Decreased Cycle Times
Market-driven schedules– Software is competitive weapon– Customer, competition drive demands– Supply chain integration drives dates
Accelerated development techniques– Agile, extreme programming practices– Short interval, incremental deliveries– Cycles times of weeks or months
Greater Functionality
Component-based development– Incorporate new functionality quickly– Less development, more testing
Integrated applications– Massive ERP, CRM systems– Easy to configure, difficult to test
Integrated enterprises– Supply chain integration– Standards for development, none for test
Test Automation Trends
Technology is Evolving
Record/ReplayRecord/Replay/ScriptingData-Driven/Keyword Driven
FrameworkClass/Action Framework
Capture/Replay
Test cases executed manually and recorded– No leverage– No concurrent test development
Scripts are unreliable– Sensitive to timing, changes– No logic for error handling
Scripts are not maintainable– Unstructured, undocumented– Hard coded data means huge volumes
Example Script
# Add Accountset_window ("Account Manager", 15);edit_set ("txtName", “Worksoft");obj_type ("txtName","<kTab>");edit_set ("txtAddress", “123 Main Street");obj_type ("txtAddress","<kTab>");edit_set ("txtCity", “Dallaslist_select_item ("cbState", "Texas"); # Item Number 42;edit_set ("Edit", "Texas");edit_set ("txtZip", “99999");obj_type ("txtZip","<kTab>");button_press ("Save_2");
Record/Replay/Scripting
Test cases executed manually, recorded then modified– No concurrent test development
– Requires programming skills
Scripts are complex– Writing programs to test programs
– Code added for synchronization, logic, error handling
– Code added for external data
Scripts are difficult to maintain– Script code can exceed application code
– Turnover means obsolescence
Example Script# Add Account
set_window ("Account Manager", 15);if ( obj_exists “txtName != E_OK ){
gsStepStatus = "edit_Input: Edit object does not exists";return;}status = edit_set ( sAccountName );
}if ( status != E_OK ){
gsStepStatus = "edit_Input: Value could not be input";return;
}type("<kTab>");
Data-Driven/Keyword Driven Framework
Keywords recorded or generated, then modified– Script language specific– Requires programming skills
Scripts are complex, high volume– Complete framework must be developed– May be hundreds of action/keyword variations
Scripts are difficult to maintain– Added or modified for new functionality– Multiple changes for application UI changes
Keyword Framework Structure
Proprietary Tool
Business Processes
Execution ScriptsData Files
Configuration Managed Assets
Keyword Functions
Calls to BPs
Object Map
Example Test Case
Begin testcase Cust-AUD
Name Address City State Zip
Add Account Worksoft 123 Main Dallas Texas 77777
Update Account Acme AC 456 Elm Phoenix Arizona 88888
Delete Account Bay Brick 789 Shore San Jose California 99999
Class/Action Framework Document then execute
– Pre-written libraries, customer-extensible– Proven, supported application
Point and click development– Enforced structure, conventions– Add, manage data easily
Powerful framework– Synchronization, error handling, recovery– Standard and shared processes
Relational database– Automated impact analysis, maintenance– Multi-user, centralized maintenance, administration– All test assets can be housed in a database—no flat files
Open architecture– Single interface across multiple platforms, tools
Class/Action Framework Structure
Business Processes
Data (in DB)
Configuration Managed Assets
Proprietary Tool
Action FunctionsObject
Map
Business Processes and data are decoupled from tool
Application Map
Define Process
Add Application
Add Window
Select Object
Select Action
Input Value
Literal data
Variable data
Generic Functions
public function edit_Input ( sObjectId, sObjectName, sWindowId, sWindowName ){
if ( obj_exists ( sObjectId ) != E_OK ) {gsStepStatus = "edit_Input: Edit object does not exists";return;
} if ( GetStringArgument ( "Value", gsExpectedValue, sDefaultValue ) == FAILED )
{gsStepStatus = "edit_Input: Action parameter (Value) could not be retrieved";return;
}status = edit_set ( sObjectId, gsExpectedValue );
if ( status != E_OK ){
gsStepStatus = "edit_Input: Value could not be input";return;
}type("<kTab>");gbActionStatus = PASSED;return;
}
Specify Result on Pass, Fail
Specify Log Result
Add Data
Automated Documentation
Process / Window Step Description*Account Master
Press New CommandButtonInput Worksoft into Account Name TextBoxInput 123 Main into Account Address TextBoxInput Dallas into Account City TextBoxSelect Item Texas from Account State ListBoxPress Save CommandButton
*MessageBoxVerify Text is Equal To “Account Added”Press OK CommandButton
Automated Impact Analysis
Automated Maintenance
Extensive Reporting
Quality Maturity
Proprietary languages are being retired– Shift to native languages
Testability is becoming accepted– Shift to cooperative development
Automation is becoming automated– Tools are maturing
Increased management awareness– Risks are higher
Shift to Native Languages
Proprietary languages too costly– Skilled resources are scarce– 10X development costs– Deployment license costs– Not all tools support all platforms
Major vendors in transition– Mercury: WinRunner to QuickTest Pro/VBScript– Compuware: QA Run to Test Partner/VBA– Rational: Robot VB to Java
Shift to Cooperative Development
Development practices improving– Coding standards, conventions– Object properties, methods exposed– Unique, persistent, meaningful names
Development support enhanced– Code instrumentation more likely– Standards compliance more rigorous– Test APIs more common
Tools are Maturing
Next generation automation products– Certify– Unified TestPro, ABT, TestFrame
Automated unit testing – JUnit, NUnit– .TEST– AUTT
Testing designed into architectures– Test standards for Web services
Risks are Higher
Sarbanes Oxley raises stakes– CXO-level responsibility for financial accuracy– Duty imposed for internal controls– Potential civil, criminal liability
Customer, exterprise integration raises exposure– Defects are visible to outside world– Errors can impact entire supply chain
Build/Buy the Right Framework
Less development, more testing– Introduce development to testability– Focus test engineers on changes and exceptions– Empower business analysts to automate
Reduce cost of implementation, ownership– Less training, faster deployment– Less maintenance, support– Lower tool license costs
Introduce enterprise standards– Enable seamless end to end execution– Provide portability across tools, platforms– Leverage resources across applications