3testing
Post on 14-Apr-2018
212 Views
Preview:
TRANSCRIPT
-
7/27/2019 3testing
1/43
Usability Testing
In Usability Testing, a small-set of target end-users, of a software system, use it to expose usability defects. This
testing mainly focuses on the users-ease to use the application, flexibility in handling controls and ability of the
system to meet its objectives.This testing is recommended during the initial design phase of SDLC, which gives
more visibility on the expectations of the users
Need for Usability Testing
Aesthetics and design are important. How well a product looks usually determines how well it works.There are
many software applications / websites, which miserably fail, once launched, due to following reasons
-
7/27/2019 3testing
2/43
Where do I click next?
Which page needs to be navigated?
Which Icon or Jargon represents what?
Error messages are not consistent or effectively displayed
Session time not sufficient.
Usability Testing identifies usability errors in the system early in development cycle and can save a
product from failure.
Goals of Usability Testing: Goal of this testing is to satisfy users and it mainly concentrates on the following
parameters of a system:
Effectiveness of the system
Is the system is easy to learn?
Is the system useful and adds value to the target audience?
Is Content, Color, Icons, Images used are aesthetically pleasing ?
Efficiency
Navigation required to reach desired screen/webpage should be very less. Scroll bars shouldnt be used
frequently.
Uniformity in the format of screen/pages in your application/website.
Provision to search within your software application or website
Accuracy
No outdated or incorrect data like contact information/address should be present.
No broken links should be present.
User Friendliness
Controls used should be self-explanatory and must not require training to operate
Help should be provided for the users to understand the application / website
Alignment with above goals helps in effective usability testing
Usability Testing Process
Usability testing process consists of the following phases
Planning:- During this phase the goals of usability test are determined. Having volunteers sit in front of
your application and recording their actions is not a goal. You need to determine critical functionalities and
objectives of system. You need to assign tasks to your testers, which exercise these critical functionalities. During
this phase , usability testing method, number & demographics of usability testers , test report formats are also
determined
Recruiting: During this phase, you recruit the desired number of testers as per your usability test plan.
Finding testers who match your demographic (age , sex etc ) and professional ( education , job etc .) profile can
take time.
UsabilityTesting: During this phase, usability tests are actually executed.
-
7/27/2019 3testing
3/43
Data Analysis: Data from usability tests is thoroughly analyzed to derive meaningful inferences and give
actionable recommendations to improve overall usability of your product.
Reporting: Findings of the usability test is shared with all concerned stakeholders which can include
designer, developer, client, and CEO
Methods of Usability Testing
There are two methods available to do usability testing
1. Laboratory Usability Testing
2. Remote Usability Testing
Laboratory Usability Testing:. This testing is conducted in a separate lab room in presence of the
observers. The testers are assigned tasks to execute. The role of the observer is to monitor behavior of the testers
and report outcome of testing. The observer remains silent during the course of testing. In this testing both
observers and testers are present in same physical location.
Remote Usability Testing : Under this testing observers and testers are remotely located. Testers
access the System Under Test, remotely and perform assigned tasks. Testers voice , screen activity , testers
facial expressions are recorded by an automated software. Observers analyze this data and report findings of the
test. Example of such a software - http://silverbackapp.com/
How many users do you need ?
Research ( Virzi, 1992 and Neilsen & Landauer , 1993) indicates that 5 users are enough to uncover
80% of usability problems. Some researchers suggest other numbers.
The truth is , the actual number of user required depends on the complexity of the given application and your
usability goals. Increase in usability participants results into increased cost , planning , participant managementand data analysis.
But as a general guideline , if you on a small budget and interested in DIY usability testing 5 is a good number to
start with. If budget is not a constraint its best consult experienced professionals to determine number of users.
http://www.google.com/url?q=http%3A%2F%2Fsilverbackapp.com%2F&sa=D&sntz=1&usg=AFQjCNFoWbhl-XjU1mwU5q0zXlc-kX4Ywghttp://www.google.com/url?q=http%3A%2F%2Fsilverbackapp.com%2F&sa=D&sntz=1&usg=AFQjCNFoWbhl-XjU1mwU5q0zXlc-kX4Ywg -
7/27/2019 3testing
4/43
Bestpractices- Usability testing
The primary goal of this testing is to find crucial usability problems before the product is launched . Following
things have to be considered to make testing success:
Start the testing during the early stage of design and development
Its a good practice to conduct usability testing on your competitors product before you begin
development. This will help you determine usability standards for your target audience
Select the appropriate users to test the system(Can be experts/non-experts users/50-50 of Experts and
Non-Experts users)
Use a bandwidth shaper . For instance , your target audience has poor network connectivity , limit network
bandwidth to say 56 Kbps for your usability testers.
Testers need to concentrate on critical & frequently used functionalities of the system.
Assign a single observer to each tester. This helps observer to accurately note testers behavior. If anobserver is assigned to multiple testers, results may be compromised
Educate Designers and Developers that this testing outcomes is not a sign of failure but its a sign of
Improvement
Pros and Cons of Usability testing:
As with anything in life, usability testing has its merits and de-merits. Lets look at them
Pros:
It helps uncover usability issues before the product is marketed.
It helps improve end user satisfaction
It makes your system highly effective and efficient
It helps gather true feedback from your target audience who actually use your system during usability test.
You do not need to rely on opinions from random people.
Cons:
Cost is a major consideration in usability testing. It takes lots of resources to set up a Usability Test Lab.
Recruiting and management of usability testers can also be expensive .However, these costs pay
themselves up in form of higher customer satisfaction, retention and repeat business. Usability testing is
therefore highly recommended.
Functional Testing - Stuff You Must Know!
unctional testing verifies that each function of the software application operates in conformance with the
requirement specification. This testing mainly involves black box testing and it is not concerned about the source
code of the application.Each and every functionality of the system is tested by providing appropriate input, verifying
the output and comparing the actual results with the expected results. This testing involves checking of User
Interface, APIs, Database, security, client/ server applications and functionality of the Application Under Test. The
testing can be done either manually or using automation
-
7/27/2019 3testing
5/43
What do you test in Functional Testing?
The prime objective of Functional testing is checking the functionalities of the software system. It mainly
concentrates on -
Mainline functions: Testing the main functions of an application
Basic Usability: It involves basic usability testing of the system. It checks whether an user can freely
navigate through the screens without any difficulties.
Accessibility: Checks the accessibility of the system for the user
Error Conditions: Usage of testing techniques to check for error conditions. It checks whether suitable
error messages are displayed.
Functional Testing Process:
In order to functionally test an application, following steps must be observed.
Understand the Requirements
Identify test input (test data)
Compute the expected outcomes with the selected
test input values
Execute test cases
Comparison of actual and computed expected result
-
7/27/2019 3testing
6/43
Functional Vs Non-Functional Testing:
Functional Testing
Non-Functional TestinG
Functional testing is performed using the
functional specification provided by the client and
verifies the system against the functional
requirements.
Non-Functional testing checks the Performance,
reliability, scalability and other non-functional aspects of
the software system.
Functional testing is executed first Non functional testing should be performed after
functional testing
Manual testing or automation tools can be used for
functional testing
Using tools will be effective for this testing
Business requirements are the inputs to functional
testing
Performance parameters like speed , scalability are
inputs to non-functional testing.
Functional testing describes what the product does Nonfunctional testing describes how good the product
works
Easy to do manual testing Tough to do manual testing
Types of Functional testing are
Unit Testing
Smoke Testing Sanity Testing
Integration Testing
White box testing
Black Box testing
User Acceptance testing
Regression Testing
Types of Non functional testing are
Performance Testing
Load Testing Volume Testing
Stress Testing
Security Testing
Installation Testing
Penetration Testing
Compatibility Testing
Migration Testing
Functional testing tools:
There are several tools available in the marker to perform functional testing. They are explained as follows:
-
7/27/2019 3testing
7/43
JUnit Used mainly for Java applications and this can be used in Unit and system testing
soapUI This is an open source functional testing tool, mainly used for Web service testing. It supports
multiple protocols such HTTP, SOAP and JDBC.
Watir This is functional testing tool for web applications. It supports tests executed at the web browser
and uses ruby scripting language
Conclusion:
Functional testing is process of testing functionalities of the system and ensures that the system is
working as per the functionalities specified in the business document. The goal of this testing is to check whether
the system is functionally perfect!!!
Little know Details on "End to End Testing"
Unlike System Testing, End-to-End Testing not only validates the software system under test but also checks its
integration with external interfaces. Hence, the name End-to-End. The purpose of End-to-End Testing is to
exercise a complete production-like scenario. Along with the software system, it also validates batch/data
processing from other upstream/downstream systems.
End to End Testing is usually executed after functional and system testing. It uses actual production like data and
test environment to simulate real-time settings. End-to-End testing is also called Chain Testing
Why End to End Testing ?
Modern software systems are complex and are interconnected with multiple sub-systems
A sub-system may be different from the current system or may be owned by another organization. If any one of
the sub-system fails, the whole software system could collapse. This is major risk and can be avoided by
End-to-End testing. End-to-End testing verifies the complete system flow. It increase test coverage of various
-
7/27/2019 3testing
8/43
sub-systems. It helps detect issues with sub-systems and increases confidence in the overall software product.
End to End testing Process:
The following diagram gives an overview of the End to End testing process.
The chief activities involved in End to End Testing are -
Study of end to end testing requirements
Test Environment setup and hardware/software requirements
Describe all the systems and its subsystems processes.
Description of roles and responsibilities for all the systems
Testing methodology and standards
End to end requirements tracking and designing of test cases
Input and output data for each system
How to create End-to-End Test Cases?
End to End Testing Design framework consists of three parts
1. Build user functions
2. Build Conditions
3. Build Test Cases
Lets look at them in detail: -
-
7/27/2019 3testing
9/43
Build User Functions
Following activities should be done as a part of build user functions:
List down the features of the system and their interconnected components
List the input data, action and the output data for each feature or function
Identify the relationships between the functions Determine whether the function can be reusable or independent
For example Consider a scenario where you login into your bank account and transfer some money to
another account from some other bank (3rdparty sub-system)
1. Login into the banking system
2. Check for the balance amount in the account
3. Transfer some amount from your account to some other bank account (3rdparty sub-system)
4. Check the your latest account balance
5. Logout of the application
Build Conditions based on User Function
Following activities are performed as a part of build conditions:
Building a set of conditions for each user function defined
Conditions include sequence, timing and data conditions
For example Checking of more conditions like
Login Page
Invalid User Name and Password
Checking with valid user name and password
Password strength checking
Checking of error messages
Balance Amount
Check the current balance after 24 hours.(If the transfer is sent to different bank) Check for the error message if the transfer amount is greater than the current balance amount
Build Test Scenario
Building the test scenario for the user function defined
In this case,
Login into the system
Check of bank balance amount
Transfer the bank balance amount
Build Multiple Test cases
Build one or more test cases for each scenario defined .Test cases may include each condition as single
test case.
Metrics for end to end testing:
-
7/27/2019 3testing
10/43
Following are few of many metrics used for End to End Testing.
Test Case preparation status: It givesTest Case preparation progress against planned
Weekly Test Progress- Provides week-wise details of percentage test completion- Failed, not executed &
executed against planned for execution tests.
Defects Status & Details- It gives Percentage of open & closed defects by week. Also, week-wise defects
distribution based on severity and priority
Environment Availability -Total number of hours up / Total number of hours scheduled per day for
testing
Difference between End to End Testing and System Testing:
End to End Testing System Testing
Validates the software system as well as
interconnected sub-systems
Validates just the software system as per the
requirements specifications.
It checks the complete end-to-end process flow. It checks system functionalities and features.
All interfaces, backend systems will be considered
for testing
Functional and Non-Functional Testing will be
considered for testing
Its executed once system testing is completed. Its executed after integration testing.
End to End testing involves checking external
interfaces which can be complex to automate. Hence
Manual Testing is preferred.
Both Manual and Automation can be performed for
system testing
Conclusion
End to end testing is the process verifying a software system along with its sub-systems. The biggest
challenge in this testing is to have enough knowledge of the whole system as well as interconnected sub-system.
Positive Vs Negative testing
Software testing is process of verifying and validating the software or application and checks whether it is working
as expected. The intent is to find defects and improve the product quality. There are two ways to test the software ,
viz, Positive Testing and Negative Testing.
Positive testing can be performed on the system by providing the valid dataas input. It checks whether
an application behaves as expected with the positive input. . This is to test to check the application that does what it
is supposed to do so
For example
-
7/27/2019 3testing
11/43
There is a text box in an application which can accept only numbers. Entering values upto 99999 will be
acceptable by the system and any other values apart from this should not be acceptable. To do positive testing, set
the valid input values from 0 to 99999 and check whether the system is accepting the values.
Negative Testing can be performed on the system by providing invalid data as input. It checks whether an
application behaves as expected with the negative input. This is to test the application that does not do anything
that it is not supposed to do so. For example
For the example above Negative testing can be performed by testing by entering alphabets characters
from A to Z or from a to z. Either system text box should not accept the values or else it should throw an error
message for these invalid data inputs.In both the testing, following needs to be considered:
Input data
Action which needs to be performed
Output Result
Testing Technique used for Positive and Negative Testing:
Following techniques are used for Positive and negative validation of testing is:
Boundary Value Analysis
Equivalence Partitioning
Boundary Value Analysis:
This is one of the software testing technique in which the test cases are designed to include values at the
boundary. If the input data is used within the boundary value limits, then it is said to be Positive Testing. If the inputdata is picked outside the boundary value limits, then it is said to be Negative Testing.
-
7/27/2019 3testing
12/43
For example
A system can accept the numbers from 0 to 10 numeric values. All other numbers are invalid values.
Under this technique , boundary values 0 , 10 and -10 will be tested.
Equivalence Partitioning:
This is a software testing technique which divides the input date into many partitions .Values from each
partition must be tested atleast once. Partitions with valid values are used for Positive Testing. While ,partitions
with invalid values are used for negative testing.
-
7/27/2019 3testing
13/43
For example-
Numeric values Zero to ten can be divided to two( or three )partition. In our case we have two partitions -10
to -1 and 0 to 10. Sample values (5 and -5) can be taken from each part to test the scenarios.
Conclusion:
Testing helps deliver quality software application and ensures the software is bug free before the
software is launched. For effective testing, use both - Positive and Negative testing which give enough confidence
in the quality of the software. Real time users are can input any values and those needs to be tested before
release.
Static Vs Dynamic Testing
Under Static Testing code is not executed. Rather it manually checks the code, requirement documents, anddesign documents to find errors. Hence, the name static.
Main objective of this testing is to improve the quality of software products by finding errors in early stages of the
development cycle. This testing is also called as Non-execution technique or verification testing.
Static testing involves manual or automated reviews of the documents. This review, is done during initial phase of
testing to catch defect early in STLC. It examines work documents and provides review comments
Work document can be of following:
Requirement specifications
Design document
Source Code Test Plans
Test Cases
Test Scripts
Help or User document
Web Page content
UnderDynamic Testing code is executed. It checks for functional behavior of software system , memory/cpu usage
and overall performance of the system. Hence the name Dynamic
-
7/27/2019 3testing
14/43
Main objective of this testing is to confirm that the software product works in conformance with the business
requirements. This testing is also called as Execution technique or validation testing.
Dynamic testing executes the software and validates the output with the expected outcome. Dynamic testing is
performed at all levels of testing and it can be either black or white box testing.
Testing Techniques used for Static Testing:
Informal Reviews: This is one of the type of review which doesnt follow any process to find errors in the
document. Under this technique , you just review the document and give informal comments on it.
Technical Reviews:A team consisting of your peers, review the technical specification of the software
product and checks whether it is suitable for the project. They try to find any discrepancies in the
specifications and standards followed. This review concentrates mainly on the technical document
related to the software such as Test Strategy, Test Plan and requirement specification documents.
Walkthrough: The author of the work product explains the product to his team. Participants can ask
questions if any. Meeting is led by the author. Scribe makes note of review comments
Inspection: The main purpose is to find defects and meeting is led by trained moderator. This review is a
formal type of review where it follows strict process to find the defects. Reviewers have checklist to review
the work products .They record the defect and inform the participants to rectify those errors.
Static code Review: This is systematic review of the software source code without executing the code. It
checks the syntax of the code, coding standards, code optimization, etc. This is also termed as white box
testing .This review can be done at any point during development.
-
7/27/2019 3testing
15/43
Testing Techniques used for Dynamic Testing:
Unit Testing:Under unit testing , individual units or modules is tested by the developers. It involves
testing of source code by developers.
Integration Testing: Individual modules are grouped together and tested by the developers. The purpose
is to determine that modules are working as expected once they are integrated.
System Testing: System testing is performed on the whole system by checking whether the system or
application meets the requirement specification document.
Also , Non-functional testing like performance, security testing fall under the category of dynamic testing.
Difference between Static and Dynamic Testing:
Static Testing Dynamic Testing
Testing done without executing the program Testing done by executing the program
This testing does verification process Dynamic testing does validation process
Static testing is about prevention of defects Dynamic testing is about finding and fixing the defects
Static testing gives assessment of code and
documentation
Dynamic testing gives bugs/bottlenecks in the software
system.
Static testing involves checklist and process to be
followed
Dynamic testing involves test cases for execution
This testing can be performed before compilation Dynamic testing is performed after compilation
Static testing covers the structural and statement Dynamic testing covers the executable file of the code
-
7/27/2019 3testing
16/43
coverage testing
Cost of finding defects and fixing is less Cost of finding and fixing defects is high
Return on investment will be high as this process
involved at early stage
Return on investment will be low as this process involves
after the development phase
More reviews comments are highly recommended
for good quality
More defects are highly recommended for good quality.
Requires loads of meetings Comparatively requires lesser meetings
Conclusion:
Verification and Validation are two measures used to check that the software product meets the requirements
specifications. Static testing involves verification whereas dynamic testing involves validation. Together they help
improve software quality.
Adhoc Testing Stuff You Must Know !
Adhoc testing is an informal testing type with an aim to break the system. This testing is usually an unplanned
activity . It does not follow any test design techniques to create test cases. In fact is does not create test cases
altogether! This testing is primarily performed if the knowledge of testers in the system under test is very high.
Testers randomly test the application without any test cases or any business requirement document.
Adhoc Testing does not follow any structured way of testing and it is randomly done on any part of application. Main
aim of this testing is to find defects by random checking. Adhoc testing can be achieved with the testing technique
called Error Guessing. Error guessing can be done by the people having enough experience on the system to
geuss the most likely source of errors.
This testing requires no documentation/ planning /process to be followed. Since this testing aims at finding
defects through random approach, without any documentation, defects will not be mapped to test cases. Hence,
sometimes, it is very difficult to reproduce the defects as there are no test-steps or requirements mapped to it.
When execute Adhoc Testing?
Adhoc testing can be performed when there is limited time to do elaborative testing. Usually adhoc testing
is performed after the formal test execution. And if time permits, adhoc testing can be done on the system).Adhoc
-
7/27/2019 3testing
17/43
testing will be effective only if the tester is knowledgeable of the System Under Test.
Types of adhoc testing
There are different types of Adhoc testing and they are listed as below:
Buddy Testing Two buddies mutually work on identifying defects in the
same module. Mostly one buddy will be from development
team and another person will be from testing team. Buddy
testing helps the testers develop better test cases and
development team can also make design changes early.
This testing usually happens after unit testing completion.
Pair testing Two testers are assigned modules, share ideas and work
on the same machines to find defects. One person can
execute the tests and another person can take notes on
the findings. Roles of the persons can be a tester and
scriber during testing.
Comparison Buddy and Pair Testing
Buddy testing is combination of unit and system testing
together with developers and testers but Pair testing is
done only with the testers with different knowledge
levels.(Experienced and non-experienced to share their
ideas and views)
Monkey Testing Randomly test the product or application without testcases with a goal to break the system.
Best practices of Adhoc testing
Following bestpratices can ensure effective Adhoc Testing.
Good business knowledge
Testers should have good knowledge of the business and clear understanding of the requirements-
-
7/27/2019 3testing
18/43
Detailed knowledge of the end to end business process will help find defects easily. Experienced testers find
more defects as they are better at error guessing.
Test Key Modules
Key business modules should be identified and targeted for adhoc testing.. Business critical modules
should be tested first to gain confidence on the quality of the system.
Record Defects
All defects need to be recorded or written in a notepad . Defects must be assigned to developers for fixing.
For each valid defect ,corresponding test cases must be written & must be added to planned test cases.
These defect findings should be made as lesson learnt and these should be reflected in our next system
while we are planning for test cases.
Conclusion:
The advantage of Adhoc testing is to check for the completeness of testing and find more defects then
planned testing. The defect catching test cases are added as additional test cases to the planned test cases.Adhoc Testing saves lot of time as it doesnt require elaborate test planning , documentation and test
case design.
Lets explore - Exploratory Testing!
Under scripted testing, you design test cases first and later proceed with test execution. On the contrary,
exploratory testing is simultaneous process of test design and test execution all done at the same time.
Scripted Test Execution is usually a non-thinking activity where testers executes the test steps and compare the
actual results with expected results. Such test execution activity can be automated does not require much cognitive
skills.
Exploratory testing, on the other hand, is all about discovery, investigation and learning. This emphasizes is on
personal freedom and responsibility of the individual tester. Test cases are not created in advance but testers
check system on the fly. They may note down ideas about what to test before test execution. The focus of
exploratory testing is more on testing as a thinking activity.
-
7/27/2019 3testing
19/43
Though the current trend in testing is to push forautomation, exploratory testing is a new way of thinking.
Automation has its limits
Here are some major differences between Scripted and exploratory testing:
Scripted Testing Exploratory Testing
Directed from requirements Directed from requirements and exploring during
testing
Determination of test cases well in advance Determination of test cases during testing
Confirmation of testing with the requirements Investigation of system or application
Emphasizes on prediction and decision making Emphasizes on adaptability and learning
Involves confirmed testing Involves Investigation
Is about Controlling tests Is about Improvement of test design
Like making a speech you read from a draft Like making a conversion its spontaneous
The script is in control The testers mind is in control
-
7/27/2019 3testing
20/43
Exploratory testing -
Is not random testing but it is adhoc testing with purpose of find bugs
Is structured and rigorous
Is cognitively (thinking) structured as compared to procedural structure of scripted testing. This structure
comes from Charter, time boxing etc.
Is highly teachable and manageable
Is not a technique but it is an approach. What actions you perform next is governed by what you are doing
currently
Exploratory Test Preparation:
Exploratory test preparation goes through following 5 stages detailed below and it is also called session
based test management (SBTM Cycle):
1. Create a Bug Taxonomy (classification )
Categorize common types of faults found in the past projects
Analyze the root cause analysis of the problems or faults
Find the risks and develop ideas to test the application.
2. Test Charter
Test Charter should suggest
i. what to test
ii. how it can be tested
iii. What needs to be looked
Test ideas are the starting point of exploration testing
Test charter helps determine how the end user could use the system
3. Time Box
This method includes pair of testers working together not less than 90 minutes
There should not be any interrupted time in those 90 minutes session
Time box can be extended or reduced by 45 minutes
This session encourages testers to react on the response from the system and prepare for the
correct outcome
4. Review Results:
Evaluation of the defects
Learning from the testing
Analysis of coverage areas
5. Debriefing:
Compilation of the output results
Compare the results with the charter
Check whether any additional testing is needed
During exploratory execution, following needs to be done:
Mission of testing should be very clear
Keep notes on what needs to be tested, why it needs to be tested and the assessment of the product
quality
Tracking of questions and issues raised during exploratory testing
Better to pair up the testers for effective testing
The more we test, more likely to execute right test cases for the required scenarios
It is very important to take document and monitor the following
Test Coverage Whether we have taken notes on the coverage of test cases and improve the quality of
the software
-
7/27/2019 3testing
21/43
Risks Which risks needs to be covered and which are all important ones?
Test Execution Log Recordings on the test execution
Issues / Queries Take notes on the question and issues on the system
Smarter exploratory testing finds more errors in less time.
Pros and Cons of Exploratory Testing:
Pro
Con
This testing is useful when requirement
documents are not available or partially
available
It involves Investigation process which helps
find more bugs than normal testing-
Uncover bugs which are normally ignored by
other testing techniques
Helps to expand the imagination of testers by
executing more and more test cases which
finally improves productivity as well
This testing drill down to smallest part of
application and covers all the requirements
This testing covers all the types of testing
and it covers various scenarios and cases
Encourages creativity and intuition
Generation of new ideas during test
execution
this testing purely depends on the tester skills
limited by domain knowledge of the tester
Not suitable for Long execution time
Challenges of Exploratory Testing:
There are many challenges of exploratory testing and those are explained below:
-
7/27/2019 3testing
22/43
Learning to use the application or software system is a challenge
Replication of failure is difficult
Determining whether tools need to be used can be challenging
Determine the best test cases to execute can be difficult
Reporting of the test results is challenge as report doesnt have planned scripts or cases to compare with
the actual result or outcome
Documentation of all events during execution is difficult to record
Dont know when to stop the testing as exploratory testing has definite test cases to execute.
When use exploratory testing?
Exploratory testing can be used extensively when
The testing team has experienced testers
Early iteration is required
There is a critical application
New testers entered into the team
Conclusion:
Exploratory testing is performed to overcome the limitations of scripted testing. It helps in improving test
case suite. It empathizes on learning and adaptability.
All About Compatibility Testing
What is compatibility?
Compatibility is nothing but capability of existing or living together. In normal life, Oil is not compatible with water,
but milk can be easily combined with water.
What is compatibility testing?
-
7/27/2019 3testing
23/43
In computer world, compatibility is to check whether your software is capable of running on different hardware,
operating systems, applications , network environments or mobile devices.
Compatibility Testing is a type of the Non-functional testing
Types of compatibility tests
Lets look into compatibility testing types
Hardware : It checks software to be compatible with different hardware configurations .
Operating Systems: It checks your software to be compatible with different Operating Systems like
Windows , Unix , Mac OS etc.
Software: It checks your developed software to be compatible with other softwares.For example: MS
Word application should be compatible with other softwares like MS Outlook,MS Excel , VBA etc.
Network: Evaluation of performance of system In network with varying parameters such as Bandwidth,
Operating speed, Capacity. It also checks application in different networks with all parameters mentioned
earlier.
Browser: It checks compatibility of your website with different browsers like Firefox , Google Chrome ,
Internet Explorer etc.
Devices : It checks compatibility of your software with different devices like USB port Devices, Printers and
Scanners, Other media devices and Blue tooth.
Mobile: Checking you software is compatible with mobile platforms like Android , iOS etc.
-
7/27/2019 3testing
24/43
Versions of the software: It is verifying you software application to be compatible with different versions of
software.For instance checking your Microsoft Word to be compatible with Windows 7, Windows 7 SP1 ,
Windows 7 SP 2 , Windows 7 SP 3.
There are two types of version checking
Types of Version Checking
Backward compatibility Testing is to verify the behavior of the developed hardware/software with the older
versions of the hardware/software.
Forward compatibility Testing is to verify the behavior of the developed hardware/software with the newer
versions of the hardware/software.
Tools for compatibility testing
Adobe Browser Lab Browser Compatibility Testing:This tool helps check your application in different
browsers.
Secure Platform Hardware Compatibility tool:This tools includes necessary drivers for a specific hardware
platform and it provides information on tool to check for CD burning process with CD burning tools.
Virtual Desktops Operating System Compatibility: This is used to run the applications in multiple operating
-
7/27/2019 3testing
25/43
systems as virtual machines. N Number of systems can be connected and compare the results.
Compatibility testing process:
1. Initial phase of compatibility testing is to define the set of environments or platforms the application is
expected to work on.
2. Tester should have enough knowledge on the platforms / software / hardware to understand the expected
application behavior under different configurations.
3. Environment needs to be set-up for testing with different platforms, devices, networks to check whether
your application runs well under different configurations.
4. Report the bugs .Fix the defects. Re-test to confirm defect fixing.
Conclusion:
The most important use of compatibility testing is to ensure whether developed software works under different
configurations (as stated in requirements documentation). This testing is necessary to check whether the
application is compatible with clients environment.
GUI Testing
What is GUI ?
There are two types of interfaces in a computer application.Command Line Interface is where you type text and
computer responds to that command.GUI stands for Graphical User Interface where you interact with the computer
using images rather than text.Following are the GUI elements which can be used for interaction between the user
and application:
What is GUI Testing?
Graphical User Interface (GUI) testing is the process of testing the systems GUI of the System Under Test. GUI
testing involves checking the screens with the controls like menus, buttons, icons, and all types of bars tool bar,
menu bar, dialog boxes and windows etc.
What do you Check in GUI Testing ?
-
7/27/2019 3testing
26/43
The following checklist will ensure detailed GUI Testing.
Check all the GUI elements for size, position, width, length and acceptance of characters or numbers. For
instance, you must be able to provide inputs to the input fields.
Check you can execute the intended functionality of the application using the GUI
Check Error Messages are displayed correctly Check for Clear demarcation of different sections on screen
Check Font used in application is readable
Check the alignment of the text is proper
Check the Color of the font and warning messages is aesthetically pleasing
Check that the images have good clarity
Check that the images are properly aligned
Check the positioning of GUI elements for different screen resolution.
Approach of GUI Testing:
GUI testing can be done through three ways:
Manual Based Testing : Under this approach, graphical screens are checked manually by testers in
conformance with the requirements stated in business requirements document.
Record and Replay : GUI testing can be done using automation tools. This is done in 2 parts. During
Record , test steps are captured into the automation tool. During playback, the recorded test steps are executed on
the Application Under Test. Example of such tools - QTP .
-
7/27/2019 3testing
27/43
Model Based Testing
A model is a graphical description of systems behavior. It helps us to understand and predict the system
behavior. Models help in a generation of efficient test cases using the system requirements. Following needs to be
considered for this model based testing:
Build the model
Determine Inputs for the model
Calculate expected output for the model
Run the tests
Compare the actual output with the expected output
Decision on further action on the model
Some of the modeling techniques from which test cases can be derived:
Charts Depicts the state of a system and checks the state after some input.
Decision Tables Tables used to determine results for each input applied
Model based testing is an evolving technique for the generating the test cases from the requirements .Its main
advantage, compared to above two methods, is that it can determine undesirable states that your GUI can
attain.
Conclusion:
-
7/27/2019 3testing
28/43
Success of a software product immensely depends on how the GUI interacts with the user and ease in using its
various features. Hence GUI testing is very important. Manual GUI testing can sometimes be repetitive and boring
and hence error prone. Automation is highly recommended for GUI testing.
Learn API testing in 10 minutes!!!
Before going to API testing, first lets understand -
What is an API?
API is an acronym forApplication Programming Interface.It enables communication between two separate
software system. A software system implementing an API contains functions/sub-routines which can be executed
by another software system.
What is API Testing?
API testing is checking Application Programming Interface of a Software System. In order to check API , it
needs to be called from the calling software . Necessary test environment, database, server, application setup
needs to be prepared before doing API Testing.API testing is focused on the functionality of the softwares
business logic and it is entirely different from GUI testing. It mainly concentrates on the business logic layer of the
software architecture. This testing wont concentrate on the look and feel of an application.
Set-up of API environment and different types of test cases:
Normally, setting up the API environment is very complex as it doesnt involve user interface. Database
and server should be configured as per the application requirements. API Function should be called to checkwhether that API is working.Output of API could be
Any type of data
Status (say Pass or Fail)
Call to another API function.
The output is compared with the expected outcome. API test cases are written based on the input data that users
are expected to key-in.
-
7/27/2019 3testing
29/43
For example There is an API function which should add two integer numbers .
1.Long add(int a, int b)
The numbers have to be given as input parameters. Output should be summation of two integer numbers . This
output needs to be verified with expected outcome.
Calling needs to be done such as
1.add (1234, 5656)
Exceptions have to be handled if number is exceeding the integer limit.
Consider the below API function -
1. Lock()
2. Unlock()
3. Delete()
They return any value such as True (in case of success) or false (In case of error) as an output .A more accurate
test case would be , can call the functions in any of the script and later check for changes either in the database or
the Application GUI.
Calling of another API / Event
In this case , we call one of the API function which in turn will call another function.For example First API function
can be used for deleting a specified record in the table and this function in turn call another function to REFRESH
the database.
-
7/27/2019 3testing
30/43
Approach of API Testing:
Following points helps the user to do API Testing approach:
1. Understanding the functionality of the API program and clearly define the scope of the program
2. Apply testing techniques such as equivalence classes, boundary value analysis and error guessing and
write test cases for the API
3. Input Parameters for the API need to be planned and defined appropriately4. Execute the test cases and compare expected and actual results.
Difference between Unit and API Testing:
Following are the differences between unit testing and API testing
Unit Testing API Testing
Performed by developers Performed by testers
Separate functionality is tested End to End functionality has been tested
Developer can access the source code Testers usually cannot access the code
UI testing is also involved Only API functions are tested
Only Basic functionalities are tested All functional issues are tested
API Test Automation:
Since API and unit testing both target source code, similar tools can be used for testing both.
If an API method is using .NET code, then the tool which is supporting should have .NET
Example for Automation tools are
NUnit for .Net
JUnit for Java
-
7/27/2019 3testing
31/43
Best Practices of API Testing:
There are some best practices for effective API testing:
Planning the test cases What and How to test the API functions
Important to organize the test properly so that those API can be organized properly
Parameters selection should be explicitly mentioned in the test case itself Call sequencing should be performed and well planned
Prioritize API function calls so that it will be easy for testers to test
Proper testing and test cases should be done while handling one time call functions like Delete,
CloseWindow, etc.
Challenges of API testing:
There are various challenges when we are doing API testing:
No GUI available to test the application which is difficult to give input values
Verifying and validating the output in different system is little difficult for testers
Parameters selection and categorization needs to be known to the testers Exception handling function needs to utilized and used properly
Testers should be knowledgeable in coding
Conclusion:
API consists of set of classes / functions / procedures which represent the business logic layer. If API is not tested
properly, it may cause problems not only the API application but also in the calling application.
What is Security Testing ?
What is Security?
Security is set of measures to protect an application against unforeseen actions that cause it to stop functioning or
being exploited. Unforeseen actions can be either intentional or unintentional.
What is Security testing?
-
7/27/2019 3testing
32/43
Security Testing ensures, that system and applications in an organization, are free from any loopholes that may
cause a big loss. Security testing of any system is about finding all possible loopholes and weaknesses of the
system which might result into loss of information at the hands of the employees or outsiders of the Organization.
The goal of security testing is to identify the threats in the system and measure its potential vulnerabilities. It also
helps in detecting all possible security risks in the system and help developers in fixing these problems through
coding.
Types of Security Testing:
There are seven main types of security testing as per Open Source Security Testing methodology manual.
They are explained as follows:
Vulnerability Scanning : This is done through automated software to scan a system against known
vulnerability signatures.
Security Scanning: It involves identifying network and system weaknesses, and later provides solutions
for reducing these risks. This scanning can be performed for both Manual and Automated scanning.
Penetration testing: This kind of testing simulates an attack from malicious hacker. This testing involves
-
7/27/2019 3testing
33/43
analysis of a particular system to check for potential vulnerabilities to an external hacking attempt.
Risk Assessment :This testing involves analysis of security risks observed in the organization. Risks are
classified as Low, Medium and High. This testing recommends controls and measures to reduce the
risk.
Security Auditing : This is internal inspection of Applications and Operating systems for security flaws.
Audit can also be done via line by line inspection of code
Ethical hacking: Its hacking an Organization Software systems. Unlike malicious hackers ,who steal for
their own gains , the intent is to expose security flaws in the system.
Posture Assessment: This combines Security scanning, Ethical Hacking and Risk Assessments to show
an overall security posture of an organization.
Integration of security processes with the SDLC:
It is always agreed, that cost will be more ,if we postpone security testing after software implementation
phase or after deployment. So, it is necessary to involve security testing in SDLC life cycle in the earlier phases.
Lets look into the corresponding Security processes to be adopted for every phase in SDLC
SDLC Phases Security Processes
Requirements
Security analysis for requirements and check
abuse/misuse cases
Design
Security risk analysis for designing. Development of testplan including security tests
Coding and Unit Testing
Static and Dynamic Testing and Security white box testing
-
7/27/2019 3testing
34/43
Integration Testing
Black Box Testing
System Testing
Black Box Testing and Vulnerability scanning
Implementation
Penetration Testing, Vulnerability Scanning
Support
Impact analysis of Patches
Test plan should include
Security related test cases or scenarios
Test Data related to security testing
Test Tools required for security testing
Analysis on various tests outputs from different security tools
Sample Test Scenarios for Security Testing:
Sample Test scenarios to give you a glimpse of security test cases -
Password should be in encrypted format
Application or System should not allow invalid users
Check cookies and session time for application
For financial sites, Browser back button should not work.
Methodologies
In security testing, different methodologies are followed, and they are as follows:
Tiger Box:This hacking is usually done on a laptop which has a collection of OSs and hacking tools. This
testing helps penetration testers and security testers to conduct vulnerabilities assessment and attacks.
Black Box:Tester is authorized to do testing on everything about the network topology and the technology.
Grey Box: Partial information is given to the tester about the system, and it is hybrid of white and black box
models.
Hackers - Access computer system or network without authorization
Crackers Break into the systems to steal or destroy data
-
7/27/2019 3testing
35/43
Ethical Hacker Performs most of the breaking activities but with permission from owner
Script Kiddies or packet monkeys Inexperienced Hackers with programming language skill
Myths and Facts of Security testing:
Lets talk on an interesting topic on Myths and facts of security testing:
Myth #1 We dont need a security policy as we have a small business
Fact : Everyone and every company need a security policy
Myth #2 There is no return on investment in security testing
Fact : Security Testing can point out areas for improvement that can improve efficiency and reduce
downtime, enabling maximum throughput.
Myth #3: Only way to secure is to unplug it.
Fact: The only and the best way to secure organization is to find Perfect Security. Perfect security can be
achieved by performing posture assessment and compare with business, legal and industry justifications.
Myth #4:Internet isnt safe. I will purchase software or hardware to safeguard the system and savebusiness.
Fact: One of the biggest problems is to purchase software and hardware for security. Instead,
organization should understand security first and then apply it.
Conclusion:
Security testing is most important testing for an application and check whether confidential data stays
confidential. In this type of testing, tester plays a role of the attacker and play around the system to find security
related bugs. This security testing is very important in IT industry to protect data by all means.
Model Based Testing Stuff You Must Know!
Before moving forward to Model Based testing, let first understand what a model is -
What is model?
A model is a description of a systems behavior. Behavior can be described in terms of input sequences, actions,
conditions, output and flow of data from input to output. It should be practically understandable and can be
reusable shareable must have precise description of the system under test.
There are numerous models available and it describes different aspects of the system behavior. Examples of the
model are:
Data Flow Control Flow
Dependency Graphs
Decision Tables
State transition machines
-
7/27/2019 3testing
36/43
Model Based Testing
Model based testing is a testing technique where run time behavior of a software under test is checked
against predictions made by a formal specification or model. In other means, it describes how system behaves in
response to an action ( determined by a model). Supply action ,and see, if the system responds as per the
expectation.It is a light-weight formal method to validate a system. This testing can be applied for both hardwareand software testing.
The above model explains about simplified approach of writing poem in notepad and possible actions related to
each step. For each and every action(like starting, Entering poem, Saving), Test Case can be generated and output
can be verified.Model based testing is a new approach to software testing. The evolution of Software Testing is
shown below -
-
7/27/2019 3testing
37/43
Types of MBT:
There are two types of Model based testing. First is
Offline / a priori Generation of Test Suites before executing it . Test suite is nothing but collection of test cases.
Online / on-the-fly
Generation of Test Suites during test execution
Different Models in Testing:
In order to understand the MBT, it is necessary to understand some of the models explained below. Lets go
through one by one:
Finite State Machines
This model helps testers to assess the result depending on the input selected. There can be various
combinations of the inputs which results in a corresponding state of the system.The system will have specific
state and current state which is governed by set of inputs given from the testers.
Consider the example-
There is a system which allows employees to log- in to the application. Now, current state of the employee is Out
and it became In once he signs- in to the system. Under the in state , employee can view, print, scan
documents in the system .
-
7/27/2019 3testing
38/43
State Charts
It is an extension of Finite state machine and can be used for complex and real time systems. State charts are
used to describe various behaviors of the system. It has definite number of states. The behavior of the system is
analyzed and represented in the form of events for each state.
For example
Defects are raised in Defect management tool with the status as New. Once it is fixed by developers, it has to be
changed to status Fixed. If defect is not fixed change status to Re-open. State chart should be designed in such a
way that it should call for an event for each state.
-
7/27/2019 3testing
39/43
Unified Modeling Language (UML)
Unified Modeling Language (UML) is a standardized general-purpose modeling language.UML includes a set of
graphic notation techniques to create visual models of that can describe very complicated behavior of the system.
UML has notations such as:
Activities
Actors
Business Process
Components
Programming language
Challenges of Model Based Testing:
Deployment of MBT in every organization obviously requires high amount of investments and effort.
Following are drawbacks of MBT:
Necessary Skills required in testers
Learning curve time will be more
Difficult to understand the model itself
Benefits of Model Based Testing:
Following are benefits of MBT:
Easy test case/suite maintenance
Reduction in Cost
Improved Test Coverage
Can run different tests on n number of machines
Early defect detection
-
7/27/2019 3testing
40/43
Increase in defect count
Time savings
Improved tester job satisfaction
Conclusion
Testers construct mental models anyway during their testing. Those mental models can be transformed to modelson paper. This helps testers to achieve readability and re-usability.
Learn Mutation Testing in 10 minutes
In Mutation Testing, we mutate (change) certain statements in the source code and check if the test cases are
able to find the errors. It is a type of white box testing which is mainly used for unit testing. The changes in
mutant program are kept extremely small, so it does not affect the overall objective of the program.
The goal of Mutation Testing is to assess the quality of the test cases which should be robust enough to fail
mutant code. This method is also called as Fault based testing strategy as it involves creating fault in the program
Mutation was originally proposed in 1971 but lost fervor due to high costs involved. Now, again it has picked steam
and is widely used for languages such as Java and XML.
-
7/27/2019 3testing
41/43
How to execute mutation testing?
Following are the steps to execute mutation testing:
Step 1: Faults are introduced into the source code of the program by creating many versions called mutants. Each
mutant should contain a single fault, and the goal is to cause the mutant version to fail which demonstrates theeffectiveness of the test cases.
Step 2: Test cases are applied to the original program and also to the mutant program. A test case should be
adequate, and it is tweaked to detect faults in a program.
Step 3: Compare the results of original and mutant program.
Step 4: If the original program and mutant programs generate the same output, then that the mutant is killed by the
test case. Hence the test case is good enough to detect the change between the original and the mutant program.
Step 5: If the original program and mutant program generate different output, Mutant is kept alive. In such cases ,
more effective test cases need to be created that kill all mutants.
How to Create Mutant Programs?
A mutation is nothing but a single syntactic change that is made to the program statement. Each mutant program
should differ from the original program by one mutation.
-
7/27/2019 3testing
42/43
Original Program Mutant Program
If (x>y)
Print Hello
Else
Print Hi
If(x)
Print Hello
Else
Print Hi
What to change in a Mutant Program?
There are several techniques that could be used to generate mutant programs. Lets look at them
Operand replacementoperators
Expression ModificationOperators
Statement modification Operators
Replace the operand with
another operand(x with y
or y with x) or with the
constant value.
Replace an operator or
insertion of new operators
in a program statement.
Programmatic statements are
modified to create mutant programs.
Example-
If(x>y) replace x and y
values
If(5>y) replace x by
constant 5
Example-
If(x==y)
We can replace == into >=
and have mutant program
as
If(x>=y) and inserting ++
in the statement
If(x==++y)
Example-
Delete the else part in an if-else
statement
Delete the entire if-else statement to
check how program behaves
Some of sample mutation operators:
GOTO label replacement
Return statement replacement
Statement deletion
Unary operator insertion(Like
and ++)
Logical connector replacement
Comparable array name
replacement
Removing of else part in the
if-else statement
Adding or replacement ofoperators
Statement replacement by
changing the data
Data Modification for the
variables
Modification of data types in the
program
-
7/27/2019 3testing
43/43
Automation of Mutation Testing:
Mutation testing is extremely time consuming and complicated to execute manually. To speed up the process, it is
advisable to go for automation tools. Automation tools reduce cost of testing as well.
List of tools available -
Ninja Turtles- .net mutation testing tool
Mutagenesis PHP mutation testing frameworkHeckle Ruby Mutation Testing Tool
Jester Mutation Testing Tool for Java
Mutation Score:
The mutation score is defined as the percentage of killed mutants with the total number of mutants.
Mutation Score = (Killed Mutants / Total number of Mutants) * 100
Test cases are mutation adequate if the score is 100%. Experimental results have shown that mutation testing is
an effective approach for the measuring the adequacy of the test cases. But, the main drawback is that the high
cost of generating the mutants and executing each test case against that mutant program.
Advantages and Disadvantages of Mutation Testing:
Following are the advantages of Mutation Testing:
It is a powerful approach to attain high coverage of the source program.
This testing is capable comprehensively testing the mutant program.
Mutation testing brings a good level of error detection to the software developer.
This method uncovers ambiguities in the source code, and has the capacity to detect all the faults in the
program.
Customers are benefited from this testing by getting most reliable and stable system.
On the other side, following are the disadvantages of Mutant testing:
Mutation testing is extremely costly and time consuming since there are many mutant programs that need to
be generated.
Since its time consuming, its fair to say that this testing cannot be done without an automation tool.
Each mutation will have the same number of test cases than that of the original program. So, a large number
of mutant programs may need to be tested against the original test suite. As this method involves source code changes, it is not at all applicable for black box testing.
Conclusion:
Do you want an exhaustive testing of your application? Answer is Mutation testing. It is the most comprehensive
technique to test a program. This is the method which checks for the effectiveness and accuracy of a testing
http://www.google.com/url?q=http%3A%2F%2Fjester.sourceforge.net%2F&sa=D&sntz=1&usg=AFQjCNEkIkiSDNwAX2ctaKXwMR9W586wRQhttp://www.google.com/url?q=http%3A%2F%2Fglu.ttono.us%2Farticles%2F2006%2F12%2F19%2Ftormenting-your-tests-with-heckle&sa=D&sntz=1&usg=AFQjCNGZ3F5n4rLuWMvUFPgZBdwk3vxZ8Ahttps://www.google.com/url?q=https%3A%2F%2Fgithub.com%2Fpadraic%2Fmutagenesis&sa=D&sntz=1&usg=AFQjCNGxlcHTCdHjO2lATT0bHBpzQsNiSwhttp://www.google.com/url?q=http%3A%2F%2Fwww.mutation-testing.net%2F&sa=D&sntz=1&usg=AFQjCNG8Bt7BdXprLbxxl723Ar8fKMGGcA
top related