Think Youre Done Yet?

Its Automation, Not Automagic!or proving Einstein was right about insanity Presented byJim Hazen1IntroductionJim HazenVeteran of the Software Testing TrenchesExperience in software testing, both commercial and consulting work.Working with automation and tools since 1989.Implemented Automation Tools on DOS, OS/2, Windows and Web platforms.Experience with tools; including HP/Mercury, IBM/Rational, and other scripting languages.2Sage Words to RememberInsanity: doing the same thing over and over again and expecting different results. Albert Einstein

AgendaWhat is Test AutomationA Brief HistoryDifferent types of test automation, tools and usesAutomation Myths How to correct themResources & Skills neededSetting expectations for Management & OthersSummaryQ&A4What is Test Automation?5Its a Big Topic to Discuss

Bigger Than We Think!

Is It This?

8Or More Like This?

9Who Builds the Automation?10Some Are Seen This Way

11Most Look Like This Instead

12What Test Automation Really IsAutomation definitionWebsters the technique of making an apparatus, a process, or a system operate automaticallyIn Software Testing we use tools to drive our processes People only think of tools that execute a test as Automation . Other tools can be used for Test Design, Test Data Generation, Defect Tracking, Test Management, etc.Automation is using a tool during any part of the process of testing.What does it mean to make an apparatus or process or system operate automatically. It means setting it up so that it runs autonomously without human intervention during the operation. It means it runs by itself and does not require a human to interact with it to get the job done.

Typically people know of or think of automation as the execution of a test, or set of tests, via a tool. They see it as some type of magic monkey that will do their work for them. Execution of a test is only a small part of the process of testing. What about test design, test data generation, test execution management, and test results analysis and reporting? There are tools that automate those tasks/processes as well. They are test automation tools too.13Who Test Automators Really Are

All in all it is a dirty job! You have to get your hands into the application and work hard to get the tool to do what you want. 14A Brief HistoryTest Execution tools have been around a long time!Mainframe Home grown macro tools1, HyperionPC DOS AutoTester, SuperKey (macro tool)PC GUI Automated Test Facility (ATF), Rational Robot, QA Run, Winrunner, etc.PC Web QTP, SilkTest, Selenium, TestComplete, etc.Mobile MonkeyTalk, Robotium, etc.Mainframe based tools worked against static and linear logic based applications. The User Interfaces also were very static in their layout and behavior. Macros were easy to record and fields were simple to navigate to. The recorded keystrokes and tests were easy to replay because the applications were very straightforward in their logic and business actions.

DOS based tools also worked against static and linear logic based applications. So just like a mainframe app using Record/Playback was relatively successful. This in effect lulled some people into a false sense of security with automation at this time.

PC GUI based tools had a totally new animal to deal with. GUI applications allowed for additional keystroke and mouse interaction between the system and user. Also, the logic paths changed from linear to event driven. The interaction was not consistent in path logic or process. The application reacted to the users actions/inputs. Straight up R/P quickly failed.

Web based systems took everything to a new level, and now mobile adds a totally other dimension to the puzzle of getting automation tools to do their jobs. The level and amount of complexity has skyrocketed.15Other Types of Test Automation ToolsOther Test Execution ToolsxUnit (JUnit & NUnit), SOAPUI, FitNesse, etc.Test Design & Test Data GenerationHexawise, AllPairs, Tosca Test Suite, etc.Test Execution ManagementHP ALM, MS Team Foundation Server, TestLink, etc.Test Environment ManagementVMWare vCloud Director, etc. Defect Tracking & ReportingBugzilla, JIRA, etc.Other Test Execution ToolsUnit Testing, Services Testing, API, Load & Performance, etc.Test Design & Test Data GenerationTest scenario design and generationData generation via different methodsTest Execution ManagementManual and automated type testsTest Environment ManagementTest Lab configuration & maintenanceDefect Tracking & Reporting

16Automation Myths & Misconceptions17Old MythsYou can automate 100% of your tests Automation will cure all problems with testingAutomation tools are usable by anyoneRecord/Playback is all you needAutomation allows Management to cut test staffOne tool is all you needAutomation will speed up testing process

* Oldies, but goodies!!

18New MythsYou dont have to know how to programOpensource tools are freeCommercial tools are expensiveGUI level automation is a waste of timeCodeless and Scriptless toolsCommercial tools cant support agileDevelopers only need to focus on Unit Tests 19How to Correct Them100% Automation Certain tests shouldnt or cant be automated. 80/20 Rule; automate 20% of functionality used 80% of time.Automation solves all testing problems it helps, but it creates new problems too.Automation can be done by anyone Not everyone has the mindset, it is both learned and inherent (logic).Record/Playback is all you need Proven time & time again to be false, and to fail.100% Automation myth Use functionality maps, categorization and prioritization, and critical path as criteria to narrow scope and get focused tests. Common Usage Modeling Language (CUML) is a way to do it.

Automation Solves all testing problems it can help with some, but creates its own problems with purpose and maintenance.

Automation can be done by anyone Proven time and time again to be false. Not anyone can be a doctor or lawyer. Automation requires skills, knowledge and experience.

Record/Playback enough said.

One tool is all you need Not every job can be done with a hammer. You need a tool that is appropriate for the job at hand.

Automation speeds up testing The Illusion of Speed problem. Even automation is dependent upon synchronization and other timing issues in software. Also, a large amount of automated scripts will not run extremely fast if there are dependencies upon run order and data. And if the workload stack is only run on a couple, or single, machine you are throttled by the processing capacity of the setup. Put the workload stack on its side and divide up the workload and then distribute to multiple machines to run in parallel. Use Economies of Scale to get speed.

Automation can be created quickly A lot of time is required to write/create an automated test initially, and then additional time to maintain them. Expect a greater initial investment than in comparison to manual testing. The investment will pay back over time, but it is not instantaneous. Because automation is its own form of software development is will be constrained by the same rules as most other types of application development work.20How to Correct Them cont.One tool is all you need There isnt a one-size fits all.Automation speeds up testing process The Illusion of Speed problem. Both execution and creation.You dont need to know programming Anyone can Automate problem.Opensource tools are free Costs due to ramp up time (money) and skilled staff to do it (more expensive).2Commercial = Expensive Not really in grand scheme of things. 2You dont need to know programming Anything you do with a computer is some form of programming. You are giving the machine instructions to do its work. You need to understand process and logic to instruct a computer to do its job. It takes logic.

Opensource tools are free Initially the cost of the tool itself is zero, but the overall cost to implement and use them can be quite large due to time to build the framework and its functionality to match commercial tools. And the cost of staff can be higher due to scarcity of resource or skills to do the work. This can result in major hidden costs.

Commerical tools are expensive Initial costs maybe, but over the long run they are not.2

GUI level Automation is a waste of time Not really, Business scenario & End-to-End tests have to be done this way. Also, if you follow the Automation Pyramid method you still have a number of GUI tests to build. For example if GUI automation should only be 10% of your automation effort and you have automated 10,000 Unit Tests that means you will have 1,000 GUI tests. Better do it right from the beginning.

Commercial tools dont do Agile It isnt the tool that does agile, it is the person using it.

Developers only create Unit Tests In the Automation Pyramid you have Services and API tests, and the tools that drive this are more technical and/or require someone to build interfaces (Fixtures) for them. Developers are best suited to this job, or a skilled tester with programming skills/knowledge who can do the same.21How to Correct Them cont.GUI level automation is a waste of time Not really

You dont need to know programming Anything you do with a computer is some form of programming. You are giving the machine instructions to do its work. You need to understand process and logic to instruct a computer to do its job. It takes logic.

Opensource tools are free Initially the cost of the tool itself is zero, but the overall cost to implement and use them can be quite large due to time to build the framework and its functionality to match commercial tools. And the cost of staff can be higher due to scarcity of resource or skills to do the work. This can result in major hidden costs.

Commerical tools are expensive Initial costs maybe, but over the long run they are not.2

GUI level Automation is a waste of time Not really, Business scenario & End-to-End tests have to be done this way. Also, if you follow the Automation Pyramid method you still have a number of GUI tests to build. For example if GUI automation should only be 10% of your automation effort and you have automated 10,000 Unit Tests that means you will have 1,000 GUI tests. Better do it right from the beginning.

Commercial tools dont do Agile It isnt the tool that does agile, it is the person using it.

Developers only create Unit Tests In the Automation Pyramid you have Services and API tests, and the tools that drive this are more technical and/or require someone to build interfaces (Fixtures) for them. Developers are best suited to this job, or a skilled tester with programming skills/knowledge who can do the same.22How to Correct Them cont.Codeless/Scriptless tools Just a different UI. Youre still writing a test which is a set of instructions (code). Commercial tools dont do Agile Yes they do, its how you use the tool that counts.Developers only create Unit Tests Other tools require someone to build the fixtures/code to talk to the SUT.You dont need to know programming Anything you do with a computer is some form of programming. You are giving the machine instructions to do its work. You need to understand process and logic to instruct a computer to do its job. It takes logic.

Opensource tools are free Initially the cost of the tool itself is zero, but the overall cost to implement and use them can be quite large due to time to build the framework and its functionality to match commercial tools. And the cost of staff can be higher due to scarcity of resource or skills to do the work. This can result in major hidden costs.

Commerical tools are expensive Initial costs maybe, but over the long run they are not.2

GUI level Automation is a waste of time Not really, Business scenario & End-to-End tests have to be done this way. Also, if you follow the Automation Pyramid method you still have a number of GUI tests to build. For example if GUI automation should only be 10% of your automation effort and you have automated 10,000 Unit Tests that means you will have 1,000 GUI tests. Better do it right from the beginning.

Commercial tools dont do Agile It isnt the tool that does agile, it is the person using it.

Developers only create Unit Tests In the Automation Pyramid you have Services and API tests, and the tools that drive this are more technical and/or require someone to build interfaces (Fixtures) for them. Developers are best suited to this job, or a skilled tester with programming skills/knowledge who can do the same.23Why Failures HappenSource: IDT Survey385% Failure Rate! Too High!!24Einstein was right!We keep repeating history, why?Always a new group of management and testers.Companies always looking to do testing faster, cheaper and with less resources.Vendors and others who sell Automagic.Instant gratification mindset. Automation takes time.Not learning from past mistakes.Software not designed with testability in mind.Testers not involved early with decisions, not included by management.Resources for AutomationDedicated StaffSkilled & experienced staffDedicated Equipment/EnvironmentsReduce risk of competition of resource or teams stepping on each otherDedicated BudgetAutomation isnt cheap, plan appropriatelyDedicated Management SupportCant have any of the other 3 without itSkills for Test AutomationIts a different view and purpose for the script code.Creating software to test software; i.e., PROGRAMMING!Need basic understanding of programming methods.Understand different types of technology layersGUI, API, System, Web, Services, Database, etc.You need to know how to program in different languages when needed.C#, Java, VBScript, Jscript, XML, SQL, etc.

27Test Automation Skills Cont.Need to understand how to hook the software under test and interact with it, and at what level.API, or Object, or Service, or Protocol, etc.Understand how tool interacts with the software under test; what functions/methods are to use.GUI Object methods (functions)Ex: SwfWindow(LoginForm").SwfEdit(ID).SET JimNeed to understand how to pick apart the software and drive it programmatically. 28Axioms to rememberSMARTStandaloneTest/Script able to run by itself, minimize dependenciesModular/MaintainableScripts and code need to be modular in design and easy to maintainAppropriateBuild according to need for project, dont try to automate 100%ReusableBuild for reuse; keep script code compact and focused.Testable & TraceableMake it easy to debug your code, and be able to link together.Axioms to remember cont.SEARCH4,5SetupTest/Script needs to check for pre-test conditions and flag if needed.ExecuteThe logic and data needed to run the test itselfAnalyzeValidations, Error Handling, etc.ReportProgress, Error, Summation of run information (Logfile)CleanupTest/Script needs to cleanup (temp files, reset data, etc.) Help (or Home)

Setting Realistic ExpectationsManagement & Other Groups100% automation is an unreachable goal.6,7

Automation is its own form of software development.

Ramp-up time will be required, longer than you think.

Benefits realized over time, no immediate payback for investment.31Expectations cont.Management & Other GroupsNo tool supports all systems & technologies out of the box.

Not all testers can write scripts. Specialized resources are a must and need to be dedicated to the work.

Nothing is Free, there are always costs involved.

Dont believe the marketing hype! Dont buy the snake oil.8

There is no Magic!32Otherwise, the result

SummaryWhat is Test Automation A lot more than you think!History Lots of tools over the last 30 plus years.Different types of test automation tools Not just for execution of a test only.Automation Myths Old and New; lots of failures due to them.How to fix the problems Be realistic, dispel the myths.Resources and Skills needed for test automation Again, a lot more than you think. It is Programming!Setting expectations for Management & Others Without proper expectations you are painted into a corner.Remember Its automation,

Not Automagic!35Q&A

36Contact InfoJim HazenCompany: Freelance Contractor

Home email: calkelpdiver@gmail.comLinkedIn profile: http://www.linkedin.com/profile/view?id=2835000&trk=tab_pro

37ReferencesExperiences of Test Automation: Case Studies of Software Test Automation, Chapter 5, Graham & Fewster, Addison-Wesley, 2012 Test Automation - Let's Talk Business, Igor Gershovich, Connected Testing, 2006. Implementing Automated Software Testing, IDT Survey, Elfriede Dustin et al., Addison-Wesley, 2009.How We Test Software At Microsoft, page 187, Alan Page, Microsoft Press, 2009.How to Automate Testing: The Big Picture, Keith Stobie & Mark Bergman, 1992.Seven Steps to Test Automation Success, Bret Pettichord, http://www.io.com/~wazmo/papers/seven_steps.html, 2001.Success with Test Automation, Bret Pettichord, http://www.io.com/~wazmo/succpap.htm, 2001.Test Automation Snake Oil, James Bach, 1996

Additional References

Software Test Automation, Mark Fewster & Dorothy Graham, Addison-Wesley, 1999.Automated Software Testing, Elfriede Dustin et al., Addison-Wesley, 1999.Why does test automation often fail to deliver?, Ben Simo, http://www.questioningsoftware.com/2007/03/why-does-test-automation-often-fail-to.html, 2007.Important Considerations for Test Automation, http://bazman.tripod.com/auto.html.Im a tester, why should I care about unit tests?, Testing the Mind blog, http://testingthemind.wordpress.com/2012/02/09/im-a-tester-why-should-i-care-about-unit-tests/?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+stcfeedsbloggers+%28Testing+Feeds+-+Bloggers%29, 2012.Conquering Common Test Automation Challenges, Bob Crews, STPCon 2012 Fall, http://www.stpcon.com/Item/1070/, 2012.Rants of a Madman I dont care if its automation, its still code!, Mark Lawler, Blog, http://markslawler.wordpress.com/2012/07/30/rants-of-a-madman-i-dont-care-if-its-automation-its-still-code/, 2012.Automate your way to ineffective testing with my twelve step program, Jared Quinert, Blog, http://www.software-testing.com.au/blog/2007/03/09/automate-your-way-to-ineffective-testing-with-my-twelve-step-program/, 2007.

38Thank you for attending this session.

Please fill out an evaluation form.39