automation 2.0 taking test automation beyond regression testing · taking test automation beyond...
TRANSCRIPT
d
Automation 2.0
Taking test automation beyond
regression testing
401-408, A-Wing, Pride Silicon Plaza, S.B. Road, Shivaji Nagar, Pune -411006,
Maharashtra, INDIA
Email: [email protected] Tel: +91-20-41020202
Global Delivery Centre:
Introduction
Whenever Test Automation is a point of discussion, it has always been linked with regression
testing. Traditionally, it has been believed that test automation can only be used to support
regression testing by taking the existing manual tests & converting them into automated test
suites which are then rerun as part of regression testing to assess the quality of the software.
Although there is no denying in the fact that test automation is best suited for regression testing,
however there are few more facts attached to this. Regression tests of this sort miserably fail to
detect new defects. Such automated regression suites are the simplest form of test automation
which is highly expensive to maintain. It is fragile and breaks easily and hardly utilizes the full
potential of test automation. These regression suites merely automate what the manual testers
have been doing, in fact many a times, losing many of the advantages which manual testing has
for finding defects.
Through this document, Nitor intends to go much beyond the regular regression testing and try
to throw light on what all can be achieved by leveraging the strengths and possibilities of test
automation. Although there are limitless possibilities which one can achieve through test
automation, this documents restricts to only few of those key possibilities, which would help in
effectively extending the reach of test automation much beyond traditional regression testing
Automation beyond regression
Below mentioned are some of the key possible areas, wherein test automation can be utilized to
leverage and fully utilize the immense potential of test automation
Building a BVT (Build Validation Test) suite
Creating test data for heavy data dependent scenarios
Continuous Integration, thereby facilitating end to end (Build-Deploy-Test) testing
Automated Unit Testing
Technical testing such as Web Service, SOA and API testing
Test Automation leveraging
Selenium Web Driver
BVT (Build Validation Test) Suite
With extensive adaption and implementation of agile methodology, there has been a drastic shift
when it comes to test execution cycles. Unlike the traditional Waterfall methodology, wherein
the QA team gets a prolonged test execution cycle, in Agile, the test execution cycles have
become much shorter & a repetitive effort. Within a sprint cycle (of 2 or 3 weeks), there are
multiple test execution cycles which are carried out to test the intended functionality being
developed and released.
The test execution cycles usually begin with a Smoke test cycle, wherein the basic
functionalities of the application are checked, just to ensure whether the build is stable enough
& testable enough so as to proceed further with a comprehensive functional testing cycle. This
smoke testing suite usually consists of a fixed set of test scenarios/test cases designed using a
‘shallow & wide’ approach covering the basic aspects & key functionalities of the entire
application. These tests are subset of tests cases that verify main functionalities. As this smoke
test suite is executed for every build cycle (almost on a daily basis), it does make sense to have
it automated. This automated smoke test suite is nothing but the ‘Build Verification Test’ suite
which is a set of test cases run on each new build to verify that the build is testable.
Benefits of automating the BVT suite –
Saves the manual testing effort in executing the same set of test cases manually for
every build cycle
Helps to surface out critical defects early
Helps to save the efforts of QA team in setup and testing of the build, when major
functionality is broken
Test Data Generation
One of the most important aspects in any testing activity is the usage of effective test data. Test
data plays a very important role in success of testing cycles. The test data drives the test cases.
However, generating the test data becomes a humungous task, especially for data driven/menu
driven applications which are heavily data dependent. In such cases, generating the test data
takes away valuable time of the QA resources. At times, when the customer is ready to share
his live/production data the life becomes less miserable, however using customer’s data is
never a wise approach and at times, if such live data is exposed to unauthorized people, it not
only puts the credibility at stake, but eventually leads to loss of business.
To overcome such situations and to generate effective and quality test data without spending
too much of manual effort, generating the test data through automation scripts is the most viable
and feasible solution. There are several test automation and automated tools available which
can be utilized for creating a comprehensive and effective set of test data.
Continuous Integration
Continuous Integration (CI) is a practice/process in which the developer integrates the code into
a shared repository frequently, which can even be several times a day. This arises the need to
test the quality of the build for each commit or each check-ins. Carrying out such repetitive tests
manually consumes a lot of manual efforts. Moreover, usually as it is the same test of scripts
which needs to be tested every time, it makes more sense to have these tests automated rather
than testing them manually. This is where executing automated test scripts in a CI environment
is beneficial.
In CI, each of the integrations or check-in builds are verified by an automated build, allowing
teams to detect integration and other errors/problems early. CI allows the team to run
automates test scripts after each commit & thus convey the results to the development team
quickly and help to locate the issues more easily.
The below diagram depicts the CI process:
All the above mentioned phases/activities of CI can be automated, right from compiling the build
to publishing/deploying the build to triggering the unit testing suite, automated BVT suite,
functional testing suite to reporting the results to the relevant stakeholder, everything can be
made possible through a single test automation framework
Using test automation in CI process effectively can yield multiple benefits. Some of these have
been mentioned below:
Helps in unearthing the issues/defects early and also helps in identifying the location of
those defects
Debugging time is reduced, hence developer can spend more time on their core
activities
Better confidence on the quality of the build
Risks are reduced
Automated Unit Testing
There is no denying in the fact that, it’s virtually impossible to write a bug free code. We humans
are bound to make mistakes. Hence, testing of the code been written by the developer is so
very imperative in the overall development cycle to ensure a thorough quality check. This is
where Unit testing comes into picture.
A unit test is a piece of code that tests another piece of code. This unit testing can be
automated by using several unit testing tools. There are several commercial and open source
tools available such as N-Unit, J-Unit, Struts, TestNG etc: Based on the platform and the
technology for which the code needs to be tested, the appropriate automation tool and
framework can be used.
An automated unit test suite has a number of important and tangible advantages:
Unit tests find problems early in the development cycle.
An automated unit test suite test your code in two dimensions:
o Time dimension because once you’ve written a unit test, it ensured that the code
works well for now and in the future
o Space dimension because unit tests written for other features ensures that the
new code did not break them
The development process becomes more flexible
Reduces the need and effort for manual testing
All of the above leads to making software more predictable, robust, repeatable and
better quality.
Technical testing – API testing, SOA testing and Web service testing
Although technical testing can be carried out manually, however there are several limitations
and challenges in doing so.
Complexities and dependencies needs to be addressed
Lack of required technical skills
Human errors leading to erroneous results
These limitations can be easily overcome by implementing test automation for carrying out
these technical testing activities. There are several tools (both commercial & open source)
currently available in the market which can be used & by leveraging the inherent benefits of
automated testing tools; the technical testing assignments can be made much more effective,
reliable, faster & error-free.
Conclusion
Test Automation can do much more than traditional regression testing. Hence, limiting test
automation for performing functional testing or regression testing would not only do injustice to
test automation, but would also be like utilizing just 10% of its overall capabilities. Leveraging
the benefits of test automation and utilizing it effectively in tasks which are complex, tedious,
time & energy consuming and difficult to carry out manually is the what the current time &
situation demands.
About Nitor Test Services
Nitor Test Center of Excellence (TCoE) is an independent test practice with well-defined testing
processes and profound knowledge of tools and techniques to cater to the testing needs of
varied products/applications. Our professional team of test analysts has worked on products of
varying complexity in terms of technology and functionality, projects and solutions across focus
areas such as Business Intelligence, Portals & Collaboration and Enterprise Mobility.
Nitor offers end to end testing services at aid our partners in continuous business growth through our
niche technology focus and diverse domain expertise. Our TCoE offering include
Highlights of Test Center of Excellence (TCoE)
ISTQB Gold Certified test partner
SSAE Soc- II Type-1 accredited organization
Technology and platform agnostic. NitorTCoE is technology /software / tool independent
70+ strong team of certified test engineers
Seven years of profound QA experience
Strong knowledge base with expertise in diverse methodologies and tools
Contact Us
NitorInfotech
401-408, A-Wing, Pride Silicon Plaza
S.B. Road
Pune -411006; Maharashtra, INDIA
Email: [email protected]