a reflection on test automation

8/6/2019 A Reflection on Test Automation

1/30

A Reflection on Test Automation

Automated software testing is nothing but automating the existing process of manual software testing. Itrequires a solid testing infrastructure and a thoughtful software testing life cycle, both supported and valuedby the management.

Test automation is not obviously the right thing to do without answering questions such as why, what, whenand how to automate. Moreover, it is an expensive process, contrary to what test tool vendors would like us tobelieve. It can take between 3 to 10 times longer to develop automated test suite than to create and execute

manual test cases. Costs of test automation include personnel to support test automation for the long term,dedicated test environment as well as the costs for the purchase, development and maintenance of tools.

The benefit of test automation comes only from running the automated tests on every subsequent release andafter carefully making a cost/benefit analysis beforehand. Test automation can provide valuable assistance if itis done by the right people, in the right environment and where it makes sense to do so. As a matter of fact, itis an addition to your testing process and does not replace manual testing nor does it enable you to downsizeyour testing department.

The Triangular Concept of Communication in Test Automation

As a professional tester, you know that everything does not always go smoothly on your project: applicationsnot delivered on time, test blocking defects, people insufficiently trained, unmotivated personnel, etc. Peopletend to take these problems into account when setting up project milestones in their test plan, but often oneaspect is overlooked: a detailed description of how the major parties involved in the project willCOMMUNICATE.

Be aware of the importance of good communication between users, developers and testers. Start thecommunication channel as soon as possible.

Gather information from the developers regarding regularly planned changes in the application anddevelop your automation scripts based on the development life cycle.

Have users (manual testers) and the test automation team use the same test designs to run tests andhave a standard way of filling out data. All agreements and procedures should be well-documented sonew people get the picture easily.

Avoid misunderstandings at a later stage by making sure both users and developers understand andagree upon well-defined business requirements for the system about to be developed.

Appoint one person (plus backup) on each team to serve as main contact.

A Perspective on Test Automation

Many people think that test automation is obviously the right thing and do not bother to state whatthey hope to get. The key is to follow the rules of software development when automating testing.This includes answering questions like why, what, when and how to automate.

To make any effective use of automated testing, a structured process of software testing mustalready be in place. Otherwise there is no real point in trying to automate something that does not

exist.

Why should we automate ?

The efforts of test automation are an investment. More time and resources are needed up front inorder to obtain the benefits later on. It should be understood that the benefits of test automationcome mainly from running the automated tests every subsequent release.

Some of the reasons why one would choose to automate tests include:


2/30

improve test coverage during regression test; prevent previous defects from reappearing in the new releases; speed up testing to accelerate releases; reduce costs of testing; ensure consistency;

improve the reliability of testing; testers can focus more on test-depth instead of repetitive work; Tests can be run over and over again with less overhead.

When should we automate ?

Developers nowadays can produce code faster and faster with more complexity than ever before.Advancements in code generation tools and code reuse are making it difficult for testers to keepup with software development. Test automation, especially if applied only at the end of the testingcycle, will not be able to keep up with these advances. Therefore automating at the early stagesshould be the best strategy. Like structured software testing, test automation has also itsplanning, designing, testing and implementation phases. The best approach is to integrate andsynchronize those different phases of test automation with the Implementation Modelof

structured software testing.

Test Automation Tips

When starting to use test automation tools, you are likely to experience more or less seriousproblems that even might endanger the final outcome of your project. This article will provide youwith some practical guidelines for test automation in order to avoid unnecessary problems. Someof them may just seem common sense, but we consider it worthwhile mentioning them here,since we have seen projects being jeopardised because one or more of them had not been takeninto account.

Do not try to automate every requirement. For some tests, the effort that should be spent

to set up automation is just too high. Test automation is software development, a test script is code. This does not only imply

that you need some people with programming experience in your test team, but also thatyour testers, operating to a certain extent as software developers, must be disciplinedand create the scripts according to predefined standards.

Use "general purpose" scripts for initialisation of test data, for navigation from one screenin the application to another, ... Also make sure that there exists only one version of thesescripts, instead of every tester developing his own script.

Separate the test data from the actions. Put the test data in separate data files. Thismakes it much easier to add or remove test data. This way of working also facilitates themaintenance of the script.

Make your scripts robust. Try to anticipate problems. If necessary, program your script insuch a way that it skips some tests rather than blocking the execution of your entire test

set. Make independent test scripts. Do not base your tests on the data generated in the

previous script(s), but make sure you always have control over your test data. Do not be scared off by these warnings. The same experience that revealed those

pitfalls, proved that a good automated test set really speeds up the turnaround time ofregression test execution. This allows you to focus on dynamic areas in your application.


3/30

What is Automation ?

Automated software testing means automating the existing process of manual software testing.This implies that a structured manual software testing process already exists. Automation is notan island unto itself. It requires a solid testing infrastructure and a thoughtful software testing life

cycle, both supported and valued by the management.

Automation is usually the fantasy of software testers, who are most of the time under pressure todo more testing in less time. Performing manual testing especially those labour-intensive tasks isusually unappealing thus automation seems to be the solution to make their job more simple andhelps them to meet the unrealistic schedules.

There is still a common myth in the testing community that test automation tools alone can bringsolutions to our software testing problems. Many people think that test automation is obviouslythe right thing and do not bother to state what they hope to get. This includes answeringquestions like why, what, when and how to automate.

Contrary to what test tool vendors would like us to believe automated testing is an expensive

process. Studies show that it can take between 3 to 10 times longer to develop automated TestSuite than to create and execute manual test cases. Costs of test automation include personnelto support test automation for the long term, dedicated test environment as well as the costs forthe purchase, development and maintenance of tools.

The benefits of test automation only come from running the automated tests every subsequentrelease and after carefully making a cost/benefit analysis beforehand, that is after making aninformed decision about what is the best for your situation.

Test automation can add a lot of complexity and cost to a test team's effort. In addition problemslike including unrealistic expectations, poor testing practices, a false sense of security,maintenance costs, and other technical and organisational problems might arise. But it can alsoprovide some valuable assistance if its done by the right people, in the right environment and

done where it makes sense to do so.

Test automation has its limitations. It does not replace manual testing, which will find moredefects than automated testing. As a matter of fact automated testing is an addition to yourtesting process and does not enable you to downsize your testing department.

Starting Test Automation - Introduction

If you start test automation, you should opt for a long-term strategic solution. The efforts madeare to be considered as an investment as more time and resources are needed up front in order

to benefit from it later on.Automation for one test cycle does not pay off. You and I know any release will take a number oftest cycles. That is why before venturing into test automation, realistic expectations should bedefined by means of investing thoroughly into planning. Furthermore, it is a "changemanagement" process and this requires management commitment.

In an initial stage, the development of a test automation strategy requires mapping out what is tobe automated, where in the testing life cycle, how it is going to be done, how the scripts will bemaintained and what will be the expected costs and benefits.


4/30

Firstly, ensure that a structured software testing process is already in place. If not, concentrate onimplementing a good structured software testing process in parallel. Without a good testingmethodology, test automation will never succeed.

During the planning phase specify your goals, scope, organisation, milestone plan, budget, testenvironment, roles and responsibilities for your automation. This preliminary stage also involves

the process of evaluation, selection and training of automated testing tools. Defect trackingprocedures and defect workflow should also be defined and documented.

In general, define the strategy on how you intend to implement and maintain your test automationprocess. Plan to achieve small successes and grow steadily. It's better to incur a smallinvestment and see the effort it really takes before trying to automate the whole regression testsuite.

It is important to implement a strategy that keeps the maintenance cost to a minimum.Maintenance costs are usually more significant for automation than for manual tests and if testscripts are not maintained for re-use, test automation will have little value.

Below, you will find a typical process for implementing a structured test automation framework

Figure 1. Structured Test Automation Process overview.


5/30

Starting Test Automation - Planning Phase

To Automate or not to Automate

An organisation might decide to introduce automation without analysing whether automation isappropriate or not. Expectations might be too high, as managers and software engineers quite

often consider automated testing as being the "silver bullet" for all quality related problems. It istherefore important to make a thorough investigation and analysis. Below, youll find someguidelines that will make it easier to decide if automation really is a solution to the problem.

Figure 2. The process of decision making.

At first, investigate if a structured testing process is already in place. If not, try implementing atesting process in parallel with automation. Before even starting to think of automation, astructured testing process should at least be in place. If a structured testing process is alreadypresent, an investigation should be done to find out whether automation is really the solution toyour problem (i.e. take into account the number of planned releases and test cycles). You willonly benefit from automation if more than two cycles or releases are planned, which is mostly thecase in reality.Acquiring managements commitment is also a crucial element to go ahead with the automationprocess. Assuming their expectations towards automated testing are realistic and if they areaware of the costs for introducing the tool, allocating appropriate staff and resources.

After acquiring management commitment a Test Automation Assessment report should beprepared to obtain the backing for the budget and the resource needed.

Measuring Test Automation - Regression Testing

There are several reasons why one would like to measure test automation, i.e. to know whether itwas a good investment, to monitor improvement, to evaluate and compare alternatives, to haveearly warning, to be able to make predictions, to benchmark against a standard, etc.

Before we plan to measure we should always know and understand our objectives. As someoneonce said: "Is fuel economy a useful measure for an automobile? Not if you want to know whetherit will fit into your garage!"


6/30

However, reducing costs does not mean shorter time to market. It can be advantageous to havehigh investment just to reduce the time to market, providing it gives the company a competitiveedge.

Cost benefit analysis should always reflect the customers business perspective and goals.

So what are we supposed to measure? An answer is not easy to find when it comes to testautomation. An important starting point is to know what your objectives are and to measure theattributes related to them.

Which attributes of test automation can we measure? Below listed are some of the attributes oftest automation that can be measured:

Maintainability

Definition : The effort needed to update the test automation suites for each new release. Possible measurements : The possible measurements can be e.g. the average work

effort in hours to update a test suite.

Reliability

Definition : The accuracy and repeatability of your test automation. Possible measurements : Number of times a test failed due to defects in the tests or in

the test scripts.

Flexibility

Definition : The ease of working with all the different kinds of automation test ware. Possible measurements : The time and effort needed to identify, locate, restore, combine

and execute the different test automation test ware.

Efficiency

Definition : The total cost related to the effort needed for the automation. Possible measurements : Monitoring over time the total cost of automated testing, i.e.

resources, material, etc.

Portability

Definition : The ability of the automated test to run on different environments. Possible measurements : The effort and time needed to set-up and run test automation in

a new environment.

Robustness

Definition : The effectiveness of automation on an unstable or rapidly changing system. Possible measurements : Number of tests failed due to unexpected events.

Usability


7/30

Definition : The extent to which automation can be used by different types of users(Developers, non-technical people or other users etc.,)

Possible measurements : The time needed to train users to become confident andproductive with test automation.

Measurements may be quite different from project to project and one cannot know what is best

unless one has clearly understood the objectives of the project.

For example, for software that is regularly changing, with frequent releases on many platforms,the important attributes will be ease of maintaining the tests and - of course - portability.

Figure 3. A highly maintainable, portable and usable automation suite.

Figure 4. Example of a highly flexible, robust but less usable and portable automation suite


8/30

Even though several attributes of automation can be measured, measuring them all is not reallynecessary. Begin measuring the few that are really important depending on the goals you want toachieve.

A few things to measure could be e.g. average time to automate a test case, total effort or timespent on maintaining the test suite, number of tests run per cycle or number of cycles completed

per release.

The Triangular Concept of Communication in Test Automation

As a professional tester, you know that not everything always goes smoothly on your project: theapplication isnt delivered on time, test blocking defects, people insufficiently trained, unmotivatedpersonnel, etc. People tend to take these problems into account when setting up projectmilestones in their test plan, but often one aspect is overlooked: a detailed description of how themajor parties involved in the project will COMMUNICATE. Hereunder we will investigate therelationship between three parties, namely: users, developers and testers. Setting up a propercommunication channel might take up some time but many of the problems mentioned can besolved or even avoided.

Figure 1. Basic Triangular Communication Channel - (seems obvious but often overlooked)

A tester is always in the midst of problems. Lets have a look at a few examples.

Test Automation Engineer Developer: Let us take a scenario where a new version of a web-based application is released for testing. Running your test automation scripts, you notice thatobject names have been altered without prior notification. Because you will have quite amaintenance task updating your scripts, a conflict may arise. If developers did inform theautomation team, maintenance could have been done before the release (at a time whenpractically everybody is just waiting to start testing).

User (Business) Automation tester: Imagine a complex application where users (businessexperts) have developed test designs coached by few professional testers. The idea behind thisapproach is to get a good quality of test designs mixing business knowledge with the expert useof testing techniques. Next to manual execution, the test designs will be used to automate the

testing. The latter makes the situation more complex when no standard procedures andtemplates are defined for designing the tests. Hence, when design and test data problemsdemand a solution, not only regular communication is at hand, but both teams (manual andautomation team) should work together as one.

User (Business) Developer: This concerns technical versus business knowledge. These peopleoften find it hard to communicate because they have different views on the application duringdevelopment. Even though the defined business requirements should be the common ground forcommunication, both parties usually misinterpret them. A testers task should be to ensure and


9/30

verify that users and developers have a common understanding regarding the requirements andto make sure the application becomes what it should be. This is not always an easy job.

I think most of you will admit to have encountered some of these situations.

So what should you do about it?

Being aware of the importance of a good communication between these groups is already a bigstep forward. Make sure you start the communication channel as soon as possible in order toavoid problems. Once certain processes have been set up (i.e. no proper communicationchannels), it can be a tough job to change them.Make sure you have regular meetings with developers (depending on the release schedule)regarding application alterations. Ensure to receive all necessary information regarding theregularly planned changes in the application. The idea here is to develop your automation scriptsbased on the development life cycle. If there is no real planning of the development life cycle,then it becomes difficult for an automation team to predict changes, causing an additional scriptmaintenance effort.Make sure that users (manual testers) and test automation team use the same test designs to runtests and that you have a standard way of filling out data. All agreements and procedures should

be well documented so that new people can easily get the picture. Ensure regular communicationin order to synchronise the needs of both teams.At the start of the project, both users and developers should understand and agree upon well-defined business requirements for the system about to be developed. If business rules are not yetdefined, a lot of misunderstandings may arise at a later stage (knowing that developers areworking on a very technical level).Make sure to have one person (plus backup) in each team who serves as the main contact point.This approach will make it a lot easier in case of problems. If the person cannot answer yourquestion immediately, he can consult his team in order to provide an answer.

"Key-phrase" Approach to Test Automation

Definition:

It is an application independent automation framework designed to process our tests. The testsare developed using a keyphrase vocabulary that describes the business processes of the AUT(Application Under Test) and that is independent of the test automation tool which is used toexecute them.

Why:

The advantage of this approach is that it enables us to develop automated tests that are businessdriven without worrying about the technicality which lays under it. Testers can develop their testsindependent from the technical scripts needed to run the test. They will be more concerned aboutwhat the test case does instead of how it does it. Tests became more descriptive, more businessrelated and more independent from the test tools and their scripting languages.


10/30

Figure.1 Pure separation of the business from the technical

More and more tests cases can be run without the need to increase the number of scripts. Thelatter will be more proportional to the size of the AUT and not to the number of tests. Tests can bedeveloped and maintained independent from the test tool or platform. Thousands of tests can berun with out the necessity of adding more scripts. Thus reducing the maintenance costssubstantially. In addition to this, non-technical personnel are able to easily implement it as theywon't need to deal with the scripts but concentrate mostly on the test scenario's.

How:

To implement Key-phrase approach it is important that the business requirements and functionalspecification be defined, ready and documented. One should plan enough time to analyse andunderstand what and how the system is supposed to work in order to develop the Key-phrases.They will be the basis for developing the test designs or scenario's in the next phase especiallythe prologue, epilogue, navigation, validation key-phrases and test data that are necessary forformulating test designs. Key-phrases should purely reflect the business processes of the AUTand should be readable, understandable, meaningful and consistent as they are meant to

describe what and how the AUT is supposed to function.


11/30

Figure. 2 The process of Key-phrase Analysis and Development

During execution process all the necessary interpretation from the Key-phrase (business) intotechnical is handled by a single control script which is generic in nature. Furthermore, all theautomation testware needed can be developed outside this control script. This level of separationof business from the automation script will of course require a layer of technical implementationbut is advantageous in the long run. In addition, ones implemented it doesn't require highmaintenance effort.


12/30

Figure.3 An overview of the core low level Key-phrase engine.

Key-phrase approach to automation is not about automation scripts or testing tools, it is a conceptthat tries to bring both structured testing process and structured test automation into one coherentprocess. It is a methodology which will force us to structure not only our automation but also ourtesting processes, and independent of any test tool we use. It is a common known fact that testautomation is nothing but automating the testing process, and it is the quality of our testingprocess which will at the end determine the quality of our automation.

Finally, we see more and more researches and experiments being carried out on integratedapproach to test automation world wide, and several exciting concepts are popping-up. Hopefullytest automation will continue to evolve until the "silver bullet" myth becomes a reality.

The Basics of Implementing Performance Testing

In this article I will try to highlight major points that are important to remember when implementingperformance testing. Please note that each of these points are just guidelines and should beworked out and tackled in detail depending on your specific situation.

1. Define your goals

Is your goal to verify system capacity ?


13/30

Is your goal to verify performance requirements ? Is your goal to verify the scalability of your system (preparing for growth and change)? Is your goal to determine optimal hardware/application configurations ? Is your goal finding bottle necks ? Is your goal to gauge how well your products stack up against the competition

etc.,

You can address all these objective at once if you want and that is if you have all the time andmoney. Prepare your strategy and planning on how you intend to tackle your performanceobjectives and how you intend to measure and test them. Define clearly what you want tomeasure and the pass/fail criteria (determining the measurement criteria for the performancetests is easier once you determine what your performance test objectives are) .

2. Formulate your performance requirements and identify the performance critical businessprocesses

What are the business SLA's ?

Which business processes are performance critical? How many transactions defined ? How many transactions per second need to be processed ? How many concurrent and total number of users is the application supposed to support ? What are the profiles of your real users ? What is the ratio between the different user profiles ? What are the acceptable response times for the transactions ? How many operations at what time of the day ? What are the risks if the business process fails ? Which protocols are supported ? etc.,

3. Develop performance test scenario's and define metrics and measurements

After you have defined WHAT your goals are, you will need to work out the details on HOW youintend to achieve your goals. You will need to formulate detailed test procedures and define themeasurements that you will need for each individual test cases. For example you would like tomeasure:

Response time of the end user System behaviour under load Number of concurrent users Application server performance Webserver performance

Database server performance Middleware performance Performance of network components etc.,

In addition you will need to describe for example how many users, which user profiles, at whattime of the day, how many iterations, what think-time, how in ramping-up etc., to be able toachieve the objective of your test cases. You will not only simulate the real user and systemapplication working situation but you will also need to predict different 'what if' scenario's.


14/30

Finally, you might also want to select the appropriate server resource and other measurementslike for example CPU Usage, memory, disk traffic, swap in/out rate, incoming/out going packetrate, throughput, hits/seconds etc., and know how to use them in your test cases and interpretyour results correctly.

4. Script development, test execution and result Analysis

Forget about doing your performance tests manually with stop watch ! There are lot's ofperformance testing tools out there.

Ensure that you have done the proper testing tool evaluation and selection processbefore committing to any tool(see our previous articles on testing tool evaluation andselection processes). For example, your tool needs to correctly simulate your real users,support your required amount of virtual user licenses and also should be suitable to yourorganisational system engineering environment needs. Unfortunately, cost plays a majorrole in most organisations when making decision regarding testing tools but a carefulcost/benefit analysis should be done before coming to any conclusion.

Script development depends much on the tool selected for testing. Even though scriptinglanguages differ from tool to tool, the common thing about most performance testing tools

is that they all will try to simulate the behaviour of your real users. Do not forget that a tool is nothing but a means to help you achieve the objectives of yourtests, therefore focus on your test scenario. It is not executing a performance tests that'sdifficult but interpreting your collected system performance metrics to pin point yourperformance bottlenecks. Therefore ensure that all your system and test tool parametersare correctly set-up before executing your tests. This will guarantee that your results areinterpreted correctly and your advice for fine-tuning the system will be much appreciated.Do not forget that sometimes overseeing a small configuration setting somewhere onyour system or on your tool can make a huge difference in performance, causingunnecessary panic.

5. Type of performance tests

Even though different definitions are given by different people regarding performance testing. Iusually like to put Load, Volume and Stress testing under performance testing.

The difference between those three types of performance tests is in their objectives:

Load Testing: To verify if your system can sustain a requested number of users withacceptable response time.

Volume Testing: To determine the maximum number of concurrent users your systemcan manage without failure (to benchmark the maximum loads).

Stress Testing: To load test your system over extended periods of time to validate thestability and reliability (i.e., memory leaks).

As each of the test types above have it's own objectives, plan carefully which type of tests youneed, to achieve your objective and which ones to execute first. Remember that some tests willtake longer to execute and some not. Some will load the system until it breaks while the otherswill just load the system modestly depending on what you intend to measure.

6. Test vs. production environment

The key requirement for a test environment is the accurate representation and testing ofthe target production environment. Please keep in mind that accurate does not


15/30

necessarily mean identical. Sometimes in might be costly and unrealistic to set-up testenvironment identical to that of production especially with multi-tiered systems.

Perform upgrades to the hardware, operating systems, database, and application serversoftware in the test environment before upgrading the production environment. After thetest environment has been checked successfully, the upgrades to the productionenvironment will be fast, and with less downtime.

Most performance problems are not due to software code but can be attributed to setting-up and configuration of the system architecture. Therefore try setting up and configuringyour system architecture of your test environment to represent your final productionenvironment if surprises are to be avoided.

7. Define clearly the role and responsibilities of all involved parties

Unlike other type of testing projects, in performance testing you will be working VERY closely withdifferent parties among which are (depending on your organisation):

System architecture experts Software & Middleware Database administrators Network administrators Mainframe specialists End users Management etc.,

Planning and executing your performance tests in co-ordination with all the different parties is oneof the toughest challenges you will face when implementing performance testing, especially whendealing with multi-tiered systems. Performance testing is a team work and not only do you needto verify the versions, settings and configurations of your individual system components with theresponsible parties, but your test results will be the bases for the other teams to pin point thebottlenecks and tune-up the system. You will need to sit down together with the other teams toanalyse and interpret results and discuss the next strategies.

Therefore, appropriate communication and decision making channels should be established tofacilitate the co-operation between the parties, because most of the time performance issuesneed immediate intervention and above all the active participation of all the parties.

Give Me a Test Hook or Else

In case you don't know it, without automation, software testing is hopeless. Thelatest development tools and component libraries allow developers to develop somuch functionality so fast that it is literally impossible, actually inconceivable, that

you can test the finished applications by hand. Most test budgets are measuredin fractions of development, so it's not like you can spend more time and peopletesting than you do developing.

Of course it's easy for developers to automate unit testing; after all, they havecontrol of the source. They can use debuggers, instrument their code, insertbreakpoints, whatever. But if you have ever tried to automate black boxautomated testing, you quickly discover that test tools can't drive applications


16/30

whose components aren't strictly vanilla. Custom controls, third partycomponents, complex objects within containers and just about any user-definedor modified object classes give test tools fits. They can't get the object names, letalone the methods and properties needed to interact with them.

In my experience, more than half of all test automation time is spentwastedtrying to deal with these complications, and in too many cases automation failscompletely. But what is really inexcusable is that it doesn't have to be that way.Most test tools provide source code implants or DLL files that, when compiledinto the code, give the tools access to the object names, methods and propertiesthat are needed to do test automation. Adding this capability takes minutesusually only a single line of codebut it can make the difference betweenautomated and manual testing.

So what's the problem? In my opinion, it's either ignorance, paranoia or purelaziness.

Ignorance because a lot of companies think that if they compile in a test hook,then do the testing, then remove it for shipment that they have not really testedthe production code. This is nonsense. True, the production code does not havethe same hook as the tested version, but so long as the source is otherwiseidentical between the two compiles the only difference is the test hook.

The key is that these hooks don't do anything unless called. They are usually justa DLL that lets the tool inside the application's process space so it can see theobjects. They only provide information, they don't alter or create it. So, compilingthe code without the hook should have zero effect on the application functionality.

But if you just can't shake this superstition, then leave the hook in when you ship.This is where paranoia comes in. Haven't you just created a security problem?After all, now someone could use that hook to spy on your software. To that Isay, so what?

Let's face it. The newer runtime-based languages (Java, .Net) are basicallyinterpreted anyway. You can reverse the original source code out to the letter. Sodon't kid yourself that someone might be able to somehow peek into yoursoftware just because of a hook. Good grief, there are OS security holes rightnow that let strangers across the globe take complete control of your whole

computer and everything it is attached to.

If you are still really freaked out about it, then identify those objects that are highrisk and make their methods and properties private. Even the hooks can't get intothose. But for goodness' sake don't do it to any that are needed for interactionwith user interface objects; since they are exposed to the user there can't bemuch to hide in the first place, and in the second place you'll cripple automation.


17/30

Which leads me to the last problem: laziness. I go nonlinear when I heardevelopers complain about the "effort" it takes to address automated testability.Last time I checked it takes only a few minutes to either add or remove lines ofcode, and since compilation is usually automated anyway it may take zero timeafter the initial setup. If you can't be bothered to invest a few precious minutes to

save your company weeks or months of work, or to enable automation that couldmake a difference in orders of magnitude in quality or time to market, maybe youshould just retire since you don't really want to work in the first place.

But in the final analysis, the reason doesn't really matter. The real question is notwhether you should provide a test hook for automation, it is why would thereeven be a controversy in the first place. Why would management evencontemplate, let alone tolerate, applications requiring manual testing thatincreases costs while reducing quality? Why aren't developers required to delivertest hooks as a matter of course? If you think about it, it's completely crazy toeven argue about it. The benefits are so undeniable and the risks are so

debatable that there should not even be a question, let alone a war.

Of course I could be wrong, but in 20 years of test automation I have never hador even heard of a test hook backfiring. Has anyone else?

Principles of Performance Testing

Picture this: you convinced management of the need for performance testing your web site ormulti client client/server application. You even managed to get a budget. Now, how do you start?

First of all: gather the right people to start an interdisciplinary task force. Since people of differentdepartments will be working together, its important to straighten out the roles and responsibilities.

Defining your strategy starts by setting the goal. Performance testing is a wide term and canmean different things. You will have to choose between different types of performance testing inorder to predict the real-world impact on your system (load testing, endurance testing, stresstesting, ).

Once this is clear, you need to gather information on system usage. What different types oftransactions (login, search, buy something, ) exist and which are critical, from a performancepoint of view? In a lot of cases, 10% of the transactions constitute 90% or more of the load on thesystem. How many users do you expect during peak time and what kind of user profiles do theyhave?

Based on the information you have already gathered, calculate the necessary workload (numberof users) and determine the necessary duration and workload patterns. You also decide on whatkind of measurements are required. You will at least want to measure the typical performancesystem characteristics: throughput, (perceived) response time and availability.

By now, you have obtained the information to come to a well-founded decision on tool selection.The classic performance testing tool lets you generate a large number of virtual users, imitating


18/30

real business processes. However, if you want to check the complete system architecture, youmight need extra tools for database monitoring, log analysis,

Once your tool is in place, the rest of the test environment needs to be set up before virtual userscan start attacking your application. Make sure that the features of the entire infrastructure can beexercised exactly the same way as in production. One solution is to use a scaled down version of

the production system (e.g. 2 instead of 6 Web servers) and extrapolate the results.

The execution phase is iterative: test a little, tune a little. Save the crucial measurements as abenchmark in order to compare and check if the system adjustments have had an impact. Aftereach test run, the log files, reports and graphs should provide enough input to detect the systemsbottleneck. Isolation testing can be done on the specific module or system component where youthink the bottleneck is residing. Try to define the minimum test that is needed to reproduce thedefect.

Your job is not over once the web site is up and running. Especially for web sites, monitoring inproduction is still required. Monitoring allows you to check if the tolerance levels for the responsetimes for business process transactions are not violated.

Performance Fine-Tuning - A Case Study

In previous articles, we explained the need for network monitoring in order to find bottlenecks inan application. Now we take it a step further and look into a case of application fine-tuning andelimination of bottlenecks. Performance fine-tuning is the activity of increasing the performance ofsoftware, while capacity planning is making a decision on what hardware to purchase to fulfill agiven role.We were testing an application that would enable brokers to manage insurance applications on-line.

In brief, the main functionality had already been approved by the acceptance test team, but -when we were called in the application was still facing a huge performance problem.

From the start we emphasized that, before starting tests with multiple users, all errors for a singleuser needed to disappear from the logs. When for weeks in a row no satisfactory improvementswere made, we finally got their attention. We introduced a benchmark as the cumulative responsetime of the different screens accessed, when running a standard business process with a singleuser or multiple users. After a year, this benchmark was still obtained generating multiple errors.The performance test team entered the project at a time when the application was in such anadvanced stage of development that everyone had by then realized that performance wasbecoming the issue.

Running a single user consumed up to 50 percent of the application servers CPU resources. Aslong as this situation was holding up, the application evidently had problems to run more thanthree users. As a result, several architectural changes were implemented and the development

team fixed the (java) code on the application server. After having installed several patches andnew releases, CPU usage decreased below 25 percent for a single user with a single CPU on theapplication server.

Error-removal further decreased CPU usage, however, the application still had problems toconcurrently run more than 25 users, while it was designed for about 250 concurrent users.

The code and architectural changes for performance took such a big part out of the budget thatthe final delivery could no longer be postponed no matter how good or bad the application


19/30

performed. The deadline had been shifted for almost a year, so no more major delays could beallowed anymore.

What I'm trying to prove here is that part of this debacle could have been avoided by setting up aperformance strategy from day 1. The deadline would not have been crossed - and definitely notby a whole year if we had been there from the start of the project to deploy a performance

strategy at the highest level: analysing business processes and tuning the rules of the business,tuning the application design and the structure of and access to the database, analysingcomponents and testing on integration level.

During all these steps, effective performance analysis should be done, based on a provenmethodology, but first of all, you need to understand your environment.

Typical Web Performance Problems and their Causes:

A. Long resonse time from the end-users' point of view

B. Long response time as measured by the servers

C. Memory leaks

D. High CPU usage

E. Too many open connections between the application and end-users

F. Lengthy queues for end-user requests

G. Too many table scans of the database

H. Database deadlocks

I. Erroneous data returned

J. HTTP errors

Application of Statistics in Load Testing

Performance testers mostly use common sense to determine how many tests to run. In mostcases thats quite alright. But when response time figures have to be presented to themanagement, even good testers have the urge to run far too many tests just to be sure. So,

instead of wasting valuable time, it is more appropriate to base the number of tests on somestatistical theories. By doing this, you can clearly motivate the amount of iterations you run andmaybe give your conscience a rest.

Imagine the following. You are willing to assume your tests are distributed normally around amean. You are at ease with using the standard deviation of a benchmark test (50 iterationsminimum) as the real standard deviation and you feel comfortable letting the error be a function ofthe standard deviation. In this situation, the answer is nothing more than straightforward: Usebetween 10 and 25 samples. In case of 10, youve got a 90 percent certainty that (confidenceinterval = 90) your real transaction average is located between the sample average minus half of


20/30

the standard deviation and the sample average plus half of the standard deviation. If you want 95percent certainty, you need to run 15 tests and for 99 percent, 27 iterations.

When you are looking for an absolute error which is not depending on the standard deviation -things get a bit more complicated. A presentation could seem clearer when the error is expressedin function of the mean. If this is the case, you should run a different number of tests for each

transaction, because their standard deviations will differ. This could make sense for some of us,but in the end the result can be too high for some transactions when the standard deviationincreases.

E = Z /2 * /n

E is the allowed error you are willing to accept and corresponds with the confidence intervalwidth.

Z is the critical value and can be found in the Gauss table for normal distribution.

Solve the equation for n and when you have found n, it is better to round up to the higher whole

number.

So, practically we use both approaches; the error in function of the standard deviation and infunction of the mean. Of course the results could always be manipulated based on the error wewere willing to accept. We wanted the real mean between the sample mean plus and minus halfof the standard deviation with a confidence level of 99. After using that in the formula above (inour situation), it meant that 25 tests had to be run. Between the mean plus and minus a tenth ofthe mean for the main business process, resulted in 28 tests to be run. I suggested to use onlythis amount of iterations in a final stage, not during fine-tuning. Here we used a confidence levelof 90 which resulted in respectively 11 and 12 runs. We choose 10, corresponding to an 88%

certainty with a relatively small error (0.5 and 0.1 ).

To determine whether or not your population is normal, you could do a 2-test (Goodness of fit)

for normal distribution, but that is a different topic. What you should do to increase the chances ofhaving good samples and benchmark tests, is to remove extreme values and start from there.

Transaction E=5 sec E=2 sec E=0,5*E=0,1

E=0,1

(Z /2)

Transaction 1 5 31 27 664 100 2,575 00:04,3 00:11,1

Transaction 2 4 25 27 664 95 2,575 00:03,8 00:10,1

Transaction 3 1 1 27 664 46 2,575 00:00,5 00:01,9

Transaction 4 1 5 27 664 22 2,575 00:01,6 00:08,8

Whole Contract 12 71 27 664 28 2,575 00:06,5 00:31,9

Table 1: Illustration of the above analysis for confidence level of 99 (Z /2)


21/30

Figure 2. Example of a test results used for the above analysis.

Measure Twice, Cut Once

It is an old carpenter's rule: measure twice, cut once. But it applies just as much when you cutyour first code for test scripts for automated load and stress testing. In place of the carpenter'strusty set square and tape measure, we now have a host of open-source test ware that can helpconcentrate your testing effort where it is most needed. In this workshop, we shall analyse staticlog data prior to the creation of LoadRunner scripts.

Our scenario is that of a major upgrade of a pre-existing application. The application uses a web-based architecture and is a good fit for load testing using the HTTP recording mode ofLoadRunner. On the server side, we have Apache, which by default provides a standard log file(see figure 1).


22/30

Figure 1. Apache log file.

A pre-requisite of this analysis technique is to capture log files that provide a good representationof typical usage data, and this is entirely application-specific. The capture period may be hours,days, weeks or a longer period. It may be that the usage reflects different types of users readers, people who update, and administrators. In all cases, the scope should match testingrequirements.

Stage 1 - Import

As log files are generally unwieldy, the first hop of our journey is to import them into a relational

format. We used a Perl script to parse the Apache logs and to import the data in a mySqldatabase (see figure 2). Perl and mySql are both open-source tools, but there is nothing toprevent the same technique being adopted in Visual Basic with Oracle, Python with Access, orPL/I or COBOL and DB2.


23/30

Figure 2. Populate a SQL database with logfile data.

On Windows NT and Windows 2000, there are APIs that can access the system logs, which arenot held in plain text format as they are on other platforms: these can still be used to populate arelational database.

Stage 2 - Extraction

Now our usage data is stored in a uniform form, we can begin extraction and analysis. There area number of tools to use for this.

1) Perl or other scripting languages. Write custom programs to extract the data and combine it.

2) Excel. Use the Data menu to create queries on the data.

3) GUI database tools like TOAD (for Oracle).


24/30

Figure 3. Querying the data from a Perl script.

What we are looking for are items such as the proportion of updates to queries. If, for instance,

the log captures a url such as 'http://www.myapp.com/query.pl?since= 56' for queries since 56days ago and 'http://www.myapp.com/update.pl?cust_id=109?item_id=2077?quantity=6', thenqueries need to be formulated to search out the 'query' and 'update' strings and count them.

Stage 3 - Additional analysis

There is a good opportunity here to do additional analysis. Suppose our application has logged1200 updates and 8000 queries, but the log contains 10000 rows? There are 800 rowsunaccounted for. Always check these. They may represent attempted hacks or application errors:who knows? In any case, they are worth knowing about.

These results are used to provide the profile of the instruction mix for the load generation scripts.Note that the "Profile" page is displayed by default whenever the user first logs in and thereforeaccounts for a high percentage of the total even though the users do not specifically request it.

Stage 4 - Reporting and using the data

Having now our usage data, when we create a pair of scenarios in LoadRunner (scenarios toexercise the application), we can easily write a driver routine as follows. This driver will ensurethat the t


25/30

estingof the application respects known usage.

Figure 4. Consolidated usage profile presented in a report.

Conclusion

Static analysis of log data from a hosting environment provides a useful way to determine howbest to test an application both for performance and, potentially, for correctness or security.

Open-source tools can cater for log analysis for little outlay, provided that your team has therequisite skills and tools. These are often the ones they use day in day out or are "weekend"skills.

The amount of time spent is easily recouped by that saved in waiting for seldom-needed tests tocomplete their execution.

Subsequent testing can reuse the methodology, thereby amortising the additional cost oversuccessive deliveries, and maintaining quality at a nominal cost.

Encourage your developers to make full use of the logging and tracing infrastructure. Fewerproprietary developments to support.


26/30

Overview on Testing Tools Evaluation and Selection Process

Youre planning to develop software and you want to ensure good quality. Testing it thoroughly isthe thing to do. If management approves the proposal to automate part of the manual testing,which tool would you select?

Before deciding on which tool to buy, broaden your perspective: dont just think about the project

you are responsible for, but inform yourself on future projects in the organisation. Ideally, the testtool should both fit the criteria of the organisations system engineering project as well as theneeds of a pilot project that is still in a very preliminary stage. In practice, most projects arealready in the system design phase, when the matter of test tool selection is raised.

Once management has given the green light for automated testing, a detailed list of test toolrequirements should be developed, based on the general needs combined with the specificneeds of the project at hand.

If you want to gain broad support, include management, project staff and end-users expectationsin the evaluation criteria.

Questions to ask are:

How will the tool be used within the organisation? Will other groups and departments use thetool? What is the most important function of the tool, what is the least important function? Howwill the tool mainly be used? How portable must the tool be?

Do not forget to include the expectations from the (eventually existing) test team itself. As in mostcases a single tool will not live up to all organisational test tool interests and requirements, itshould certainly meet the more immediate requirements.

Next step is that you define the criteria for a tool evaluation environment, based upon the analysisof the system and software architectures available in your organisation.

You then identify the test tool types (e.g. for regression tests, volume or stress testing, usabilitytesting) that might apply to the particular project. For each phase in the testing life cycle, there isa tool that can support you. Once identification is finished, determine a list of possible test tools.These tools should then be screened using the pilot project environment, resulting in a detailedevaluation report. The report contains the results of the different tools related to the evaluationcriteria. If you want to evaluate one or more test tools, it is better to first test the tool in an isolatedtest environment and then apply the test tool on a pilot project. Before doing this however, let thetool vendor demonstrate how the tool operates.


27/30

Together with the evaluation of the tool, one has to evaluate the structure of the testing team: doyou foresee a centralised test team or do you opt for a distributed test team? A centralised testteam often prefers a more powerful automated test tool with great flexibility, programminglanguage capabilities and growth potential. A decentralised organisation will be better served by auser-friendly tool, minimising cost and the time associated with learning how to use it.

Choosing the tool is one thing, identifying the people who will perform the evaluation of the testtool is also critical.

Evaluating and choosing an automated test tool is not a simple thing to do. In a big organisation,take sufficient time to scrutinise the tool in order to be sure that it can be used on other projects. Ifthere isnt any tool that really meets the evaluation criteria, do not decide on buying one anyway,because by doing so you can lose a lot of credibility and endanger future test automationprojects.

"For the most part, testers have been testers, not programmers. Consequently, the simplecommercial solutions have been far too complex to implement and maintain; and they becomeshelfware."

Highly repeatable testing can actually minimize the chance of discovering all the importantproblems, for the same reason that stepping in someone else's footprints minimizes the chanceof being blown up by a land mine."

Software Testing and Continuous Quality Improvement

Software Testing: A Craftsman's Approach, Second Edition

Test Driven Development: By Example

How to Break Software: A Practical Guide to Testing

Effective Software Testing: 50 Specific Ways to Improve YourTesting

Automated Software Testing: Introduction, Management, andPerformance
http://www.amazon.com/exec/obidos/ASIN/0849398339/cyberarchitec-20?dev-t=D68HUNXKLHS4JY%26camp=2025%26link_code=xm2http://www.amazon.com/exec/obidos/ASIN/0849308097/cyberarchitec-20?dev-t=D68HUNXKLHS4JY%26camp=2025%26link_code=xm2http://www.amazon.com/exec/obidos/ASIN/0321146530/cyberarchitec-20?dev-t=D68HUNXKLHS4JY%26camp=2025%26link_code=xm2http://www.amazon.com/exec/obidos/ASIN/0201796198/cyberarchitec-20?dev-t=D68HUNXKLHS4JY%26camp=2025%26link_code=xm2http://www.amazon.com/exec/obidos/ASIN/0201794292/cyberarchitec-20?dev-t=D68HUNXKLHS4JY%26camp=2025%26link_code=xm2http://www.amazon.com/exec/obidos/ASIN/0201794292/cyberarchitec-20?dev-t=D68HUNXKLHS4JY%26camp=2025%26link_code=xm2http://www.amazon.com/exec/obidos/ASIN/0201432870/cyberarchitec-20?dev-t=D68HUNXKLHS4JY%26camp=2025%26link_code=xm2http://www.amazon.com/exec/obidos/ASIN/0201432870/cyberarchitec-20?dev-t=D68HUNXKLHS4JY%26camp=2025%26link_code=xm2http://www.amazon.com/exec/obidos/ASIN/0201432870/cyberarchitec-20?dev-t=D68HUNXKLHS4JY%26camp=2025%26link_code=xm2http://www.amazon.com/exec/obidos/ASIN/0201794292/cyberarchitec-20?dev-t=D68HUNXKLHS4JY%26camp=2025%26link_code=xm2http://www.amazon.com/exec/obidos/ASIN/0201796198/cyberarchitec-20?dev-t=D68HUNXKLHS4JY%26camp=2025%26link_code=xm2http://www.amazon.com/exec/obidos/ASIN/0321146530/cyberarchitec-20?dev-t=D68HUNXKLHS4JY%26camp=2025%26link_code=xm2http://www.amazon.com/exec/obidos/ASIN/0849398339/cyberarchitec-20?dev-t=D68HUNXKLHS4JY%26camp=2025%26link_code=xm2http://www.amazon.com/exec/obidos/ASIN/0201331403/cyberarchitec-20?dev-t=D68HUNXKLHS4JY%26camp=2025%26link_code=xm2http://www.amazon.com/exec/obidos/ASIN/0849308097/cyberarchitec-20?dev-t=D68HUNXKLHS4JY%26camp=2025%26link_code=xm2http://www.amazon.com/exec/obidos/ASIN/0321146530/cyberarchitec-20?dev-t=D68HUNXKLHS4JY%26camp=2025%26link_code=xm2http://www.amazon.com/exec/obidos/ASIN/0201796198/cyberarchitec-20?dev-t=D68HUNXKLHS4JY%26camp=2025%26link_code=xm2http://www.amazon.com/exec/obidos/ASIN/0201794292/cyberarchitec-20?dev-t=D68HUNXKLHS4JY%26camp=2025%26link_code=xm2http://www.amazon.com/exec/obidos/ASIN/0201794292/cyberarchitec-20?dev-t=D68HUNXKLHS4JY%26camp=2025%26link_code=xm2http://www.amazon.com/exec/obidos/ASIN/0201432870/cyberarchitec-20?dev-t=D68HUNXKLHS4JY%26camp=2025%26link_code=xm2http://www.amazon.com/exec/obidos/ASIN/0201432870/cyberarchitec-20?dev-t=D68HUNXKLHS4JY%26camp=2025%26link_code=xm2http://www.amazon.com/exec/obidos/ASIN/0849398339/cyberarchitec-20?dev-t=D68HUNXKLHS4JY%26camp=2025%26link_code=xm2


28/30

The Complete Guide to Software Testing

Unit Testing in Java: How Tests Drive the Code

Testing Applications on the Web: Test Planning for Internet-BasedSystems

Managing the Testing Process: Practical Tools and Techniques forManaging Hardware and Software Testing, 2nd Edition

Quality Web Systems: Performance, Security, and Usability

Black-Box Testing : Techniques for Functional Testing of Softwareand Systems

Rapid Testing

Practical Guide to Testing Object-Oriented Software

Introducing Software Testing

Test Process Improvement: A Practical Step-by-Step Guide toStructured Testing

Testing Embedded Software

Automated Web Testing Toolkit: Expert Methods for Testing and

Managing Web Applications
http://www.amazon.com/exec/obidos/ASIN/0471565679/cyberarchitec-20?dev-t=D68HUNXKLHS4J%26camp=2025%26link_code=xm2http://www.amazon.com/exec/obidos/ASIN/1558608680/cyberarchitec-20?dev-t=D68HUNXKLHS4J%26camp=2025%26link_code=xm2http://www.amazon.com/exec/obidos/ASIN/047139470X/cyberarchitec-20?dev-t=D68HUNXKLHS4J%26camp=2025%26link_code=xm2http://www.amazon.com/exec/obidos/ASIN/047139470X/cyberarchitec-20?dev-t=D68HUNXKLHS4J%26camp=2025%26link_code=xm2http://www.amazon.com/exec/obidos/ASIN/0471223980/cyberarchitec-20?dev-t=D68HUNXKLHS4J%26camp=2025%26link_code=xm2http://www.amazon.com/exec/obidos/ASIN/0471223980/cyberarchitec-20?dev-t=D68HUNXKLHS4J%26camp=2025%26link_code=xm2http://www.amazon.com/exec/obidos/ASIN/0201719363/cyberarchitec-20?dev-t=D68HUNXKLHS4J%26camp=2025%26link_code=xm2http://www.amazon.com/exec/obidos/ASIN/0471120944/cyberarchitec-20?dev-t=D68HUNXKLHS4J%26camp=2025%26link_code=xm2http://www.amazon.com/exec/obidos/ASIN/0471120944/cyberarchitec-20?dev-t=D68HUNXKLHS4J%26camp=2025%26link_code=xm2http://www.amazon.com/exec/obidos/ASIN/0130912948/cyberarchitec-20?dev-t=D68HUNXKLHS4J%26camp=2025%26link_code=xm2http://www.amazon.com/exec/obidos/ASIN/0201325640/cyberarchitec-20?dev-t=D68HUNXKLHS4J%26camp=2025%26link_code=xm2http://www.amazon.com/exec/obidos/ASIN/0201719746/cyberarchitec-20?dev-t=D68HUNXKLHS4J%26camp=2025%26link_code=xm2http://www.amazon.com/exec/obidos/ASIN/0201596245/cyberarchitec-20?dev-t=D68HUNXKLHS4J%26camp=2025%26link_code=xm2http://www.amazon.com/exec/obidos/ASIN/0201596245/cyberarchitec-20?dev-t=D68HUNXKLHS4J%26camp=2025%26link_code=xm2http://www.amazon.com/exec/obidos/ASIN/0321159861/cyberarchitec-20?dev-t=D68HUNXKLHS4J%26camp=2025%26link_code=xm2http://www.amazon.com/exec/obidos/ASIN/0471414352/cyberarchitec-20?dev-t=D68HUNXKLHS4J%26camp=2025%26link_code=xm2http://www.amazon.com/exec/obidos/ASIN/0471414352/cyberarchitec-20?dev-t=D68HUNXKLHS4J%26camp=2025%26link_code=xm2http://www.amazon.com/exec/obidos/ASIN/0471414352/cyberarchitec-20?dev-t=D68HUNXKLHS4J%26camp=2025%26link_code=xm2http://www.amazon.com/exec/obidos/ASIN/0321159861/cyberarchitec-20?dev-t=D68HUNXKLHS4J%26camp=2025%26link_code=xm2http://www.amazon.com/exec/obidos/ASIN/0201596245/cyberarchitec-20?dev-t=D68HUNXKLHS4J%26camp=2025%26link_code=xm2http://www.amazon.com/exec/obidos/ASIN/0201719746/cyberarchitec-20?dev-t=D68HUNXKLHS4J%26camp=2025%26link_code=xm2http://www.amazon.com/exec/obidos/ASIN/0201325640/cyberarchitec-20?dev-t=D68HUNXKLHS4J%26camp=2025%26link_code=xm2http://www.amazon.com/exec/obidos/ASIN/0130912948/cyberarchitec-20?dev-t=D68HUNXKLHS4J%26camp=2025%26link_code=xm2http://www.amazon.com/exec/obidos/ASIN/0471120944/cyberarchitec-20?dev-t=D68HUNXKLHS4J%26camp=2025%26link_code=xm2http://www.amazon.com/exec/obidos/ASIN/0201719363/cyberarchitec-20?dev-t=D68HUNXKLHS4J%26camp=2025%26link_code=xm2http://www.amazon.com/exec/obidos/ASIN/0471223980/cyberarchitec-20?dev-t=D68HUNXKLHS4J%26camp=2025%26link_code=xm2http://www.amazon.com/exec/obidos/ASIN/047139470X/cyberarchitec-20?dev-t=D68HUNXKLHS4J%26camp=2025%26link_code=xm2http://www.amazon.com/exec/obidos/ASIN/1558608680/cyberarchitec-20?dev-t=D68HUNXKLHS4J%26camp=2025%26link_code=xm2http://www.amazon.com/exec/obidos/ASIN/0471565679/cyberarchitec-20?dev-t=D68HUNXKLHS4J%26camp=2025%26link_code=xm2http://www.amazon.com/exec/obidos/ASIN/0471565679/cyberarchitec-20?dev-t=D68HUNXKLHS4J%26camp=2025%26link_code=xm2http://www.amazon.com/exec/obidos/ASIN/1558608680/cyberarchitec-20?dev-t=D68HUNXKLHS4J%26camp=2025%26link_code=xm2http://www.amazon.com/exec/obidos/ASIN/047139470X/cyberarchitec-20?dev-t=D68HUNXKLHS4J%26camp=2025%26link_code=xm2http://www.amazon.com/exec/obidos/ASIN/047139470X/cyberarchitec-20?dev-t=D68HUNXKLHS4J%26camp=2025%26link_code=xm2http://www.amazon.com/exec/obidos/ASIN/0471223980/cyberarchitec-20?dev-t=D68HUNXKLHS4J%26camp=2025%26link_code=xm2http://www.amazon.com/exec/obidos/ASIN/0471223980/cyberarchitec-20?dev-t=D68HUNXKLHS4J%26camp=2025%26link_code=xm2http://www.amazon.com/exec/obidos/ASIN/0201719363/cyberarchitec-20?dev-t=D68HUNXKLHS4J%26camp=2025%26link_code=xm2http://www.amazon.com/exec/obidos/ASIN/0471120944/cyberarchitec-20?dev-t=D68HUNXKLHS4J%26camp=2025%26link_code=xm2http://www.amazon.com/exec/obidos/ASIN/0471120944/cyberarchitec-20?dev-t=D68HUNXKLHS4J%26camp=2025%26link_code=xm2http://www.amazon.com/exec/obidos/ASIN/0130912948/cyberarchitec-20?dev-t=D68HUNXKLHS4J%26camp=2025%26link_code=xm2http://www.amazon.com/exec/obidos/ASIN/0201325640/cyberarchitec-20?dev-t=D68HUNXKLHS4J%26camp=2025%26link_code=xm2http://www.amazon.com/exec/obidos/ASIN/0201719746/cyberarchitec-20?dev-t=D68HUNXKLHS4J%26camp=2025%26link_code=xm2http://www.amazon.com/exec/obidos/ASIN/0201596245/cyberarchitec-20?dev-t=D68HUNXKLHS4J%26camp=2025%26link_code=xm2http://www.amazon.com/exec/obidos/ASIN/0201596245/cyberarchitec-20?dev-t=D68HUNXKLHS4J%26camp=2025%26link_code=xm2http://www.amazon.com/exec/obidos/ASIN/0321159861/cyberarchitec-20?dev-t=D68HUNXKLHS4J%26camp=2025%26link_code=xm2http://www.amazon.com/exec/obidos/ASIN/0471414352/cyberarchitec-20?dev-t=D68HUNXKLHS4J%26camp=2025%26link_code=xm2http://www.amazon.com/exec/obidos/ASIN/0471414352/cyberarchitec-20?dev-t=D68HUNXKLHS4J%26camp=2025%26link_code=xm2


29/30

"A tool is only as good as the process being used to implement the tool. How a tool isimplemented and used is what really matters"


30/30

"If software defects were gold, then software testing would be gold mining. Test planning wouldbe the geologist's surveys and preparation activities done before mining takes place: Thegeologist establishes a plan and strategy, increasing the probablity that digging at particular spotsusing a particular strategy would prove to be successful."

"Why is there so much involved with automation? The answer is straightforward: because itsdifficult. Decisions about which tool(s) to use, how to architect and implement a test suite, andwho will do the work are complicated. Then there are software design considerations, supportingscripts necessary to run the automation, source control, and library construction. Add to that thecomplexity of managing a large automation suite, continued maintenance of scripts, and addingto the test suite with new functionalityone can easily be left with the looming question, "Is thisreally going to be worth it?"

Software testing is like fishing,but you get paid.

a reflection on test automation

Documents