Software Testing Principles

www.kanoah.com

ABOUT US

Kanoah is an innovative company providing ground-breaking solutions to software testing professionals on the

Atlassian JIRA platform

About Kanoah Tests

Kanoah Tests is a full featured test management, integrated seamless into JIRA with the same look-n-feel. No need to learn or switch between different applications

Coordinate all test management activities including test planning, authoring, execution, tracking and reporting from a central location

Kanoah Tests enables you to track testing progress and quality to foster collaboration and visibility across traditional and agile teams

Get real-time insights into your testing progress with out of the box reports

Easily integrate your automated tests and submit test results with Kanoah Tests’ powerful REST API or use the API to automate many areas of the application

After looking for several years at plugins for test management we finally found Kanoah Tests. The other solutions were either too complex, didn't integrate well with Jira, or were focused on a single project. Kanoah Tests proved to be an elegant solution that allowed linking between any project. Kanoah has been very responsive to feedback, requests, suggestions as well bugs. The customer service is awesome. I'd highly recommend Kanoah Tests to teams of any size looking to simplify test management and consolidate tools.

Don Pierce

Robert Murhamer

Liked Kanoah from the moment I discovered it. Integrates nicely with JIRA and especially with Agile. Test cases can be authored right from the story level, but has all other functionality a Test Case Management solution would need to have, ranging from creating test plans, executing test cases, importing test cases, API for automation, a.s.o. Additionally the team at Kanoah is amazing and responding to any question very quickly. It wasn't hard to sell to my management to purchase Kanoah. Will highly recommend Kanoah to anybody.

Zour Brosh

I just start working with Kanoah and I am impressed how it's simple to manage tests without unlimited non-used features like in most of the test management tools and still to get the needed functionality and results. The integration with Jira is a great working solution that enable to share testing and development in simple way on one system. I recommend to use Kanoah for testing management. It will help to do Kanoah as much as possible customizable like Jira to match each group methodology

For more reviews, visit: https://marketplace.atlassian.com/plugins/com.kanoah.test-manager/server/reviews

Reviews

Key Features

Native seamless integration with JIRA

No need to learn or switch between different applications

Perfect for agile & traditional testing approaches

Manage, organize and track all your testing efforts in a central place

Reuse test cases across your projects

Powerful REST API

Establish clear traceability between requirements, test cases, and defects

Execute test cases and track results that matter

Get real-time insights into your testing progress with out of the box reports

Live statistics accessible to your entire team

Benefits for the testers

No need to lear or switch between different applications

Reuse test cases across projects for regression

Link test cases to requirements and defects

API support for automated efforts

Benefits for the teams

Informed decisions based on real-time insights

End-to-end traceability and impact analysis

Centralized Test Management

Save time and increase productivity

Why choose Kanoah Tests

Coordinate all test management activities right inside JIRA

1 Testing right inside JIRA

Take advantage of the built-in reports to track the results and measure progress

2Make informed decisions

Kanoah Tests users receive priority support, even during trials

3Stelar support

Software Testing Principles

The Seven Testing PrinciplesThese principles are originally made by the folks at ISTQB and the main ideas have been recompiled from here [1] [2].

Testing shows presence of defects: Testing can show the defects are present, but cannot prove that there are no defects. Even after testing the application or product thoroughly we cannot say that the product is 100% defect free. Testing always reduces the number of undiscovered defects remaining in the software but even if no defects are found, it is not a proof of correctness. Therefore, it is important to design test cases which find as many defects as possible.

Exhaustive testing is impossible: Testing everything including all combinations of inputs and preconditions is not possible. So, instead of doing the exhaustive testing we can use risks and priorities to focus testing efforts. Accessing and managing risk is one of the most important activities and reason for testing in any project.

The Seven Testing PrinciplesEarly testing: In the software development life cycle testing activities should start as early as possible and should be focused on defined objectives. The sooner we start the testing activities the better we can utilize the available time. As soon as the initial products, such the requirement or design documents are available, we can start testing. It is common for the testing phase to get squeezed at the end of the development lifecycle, i.e. when development has finished, so by starting testing early, we can prepare testing for each level of the development lifecycle.

Defect clustering: A small number of modules contains most of the defects discovered during pre-release testing or shows the most operational failures. This is the application of the Pareto Principle to software testing: approximately 80% of the problems are found in 20% of the modules.

The Seven Testing PrinciplesPesticide paradox: If the same kinds of tests are repeated again and again, eventually the same set of test cases will no longer be able to find any new bugs. To overcome this “Pesticide Paradox”, it is really very important to review the test cases regularly and new and different tests need to be written to exercise different parts of the software or system to potentially find more defects. Because as the system evolves, many of the previously reported defects will have been fixed and the old test cases do not apply anymore. Anytime a fault is fixed or a new functionality added, we need to do regression testing to make sure the new changed software has not broken any other part of the software. However, those regression test cases also need to change to reflect the changes made in the software to be applicable and hopefully fine new defects.

Testing is context depending: Different kinds of sites are tested differently. Different methodologies, techniques and types of testing is related to the type and nature of the application. For example, a software application in a medical device needs more testing than a games software. More importantly a medical device software requires risk based testing, be compliant with medical industry regulators and possibly specific test design techniques. By the same token, a very popular website, needs to go through rigorous performance testing as well as functionality testing to make sure the performance is not affected by the load on the servers.

The Seven Testing PrinciplesAbsence –of– errors fallacy: If the system built is unusable and does not fulfil the user’s needs and expectations then finding and fixing defects does not help. Just because testing didn’t find any defects in the software, it doesn’t mean that the software is ready to be shipped. Were the executed tests really designed to catch the most defects? or where they designed to see if the software matched the user’s requirements? There are many other factors to be considered before making a decision to ship the software.

Software Testing Principles

These other principles are from known authors and editors of the Agile Development and Software Testing world [3] [4] [5].

A necessary part of a test case is a definition of the expected output or result. If the expected result of a test case has not been predefined, chances are that a plausible, but erroneous, result will be interpreted as a correct result because of the phenomenon of ‘‘the eye seeing what it wants to see". One way of combating this is to encourage a detailed examination of all output by precisely spelling out, in advance, the expected output of the program. Therefore, a test case must consist of two components:

1. A description of the input data to the program.

2. A precise description of the correct output of the program for that set of input data.

Software Testing PrinciplesTesting must be done by an independent party. Testing should not be performed by the person or team that developed the software since they tend to defend the correctness of the program. Most programmers cannot effectively test their own programs because they cannot bring themselves to shift mental gears to attempt to expose errors. Furthermore, a programmer may subconsciously avoid finding errors for fear of retribution from peers or a supervisor, a client, or the owner of the program or system being developed. In addition to these psychological issues, there is a second significant problem: The program may contain errors due to the programmer’s misunderstanding of the problem statement or specification. If this is the case, it is likely that the programmer will carry the same misunderstanding into tests of his or her own program. Note that this argument does not apply to debugging (correcting known errors).

A programming organization should not test its own programs. It is difficult for a programming organization to be objective in testing its own programs, because the testing process, if approached with the proper definition, may be viewed as decreasing the probability of meeting the schedule and the cost objectives. It is more economical for testing to be performed by an objective, independent party.

Software Testing PrinciplesAny testing process should include a thorough inspection of the results of each test. We’ve seen numerous experiments that show many subjects failed to detect certain errors, even when symptoms of those errors were clearly observable on the output listings. Put another way, errors that are found in later tests were often missed in the results from earlier tests.

Test cases must be written for input conditions that are invalid and unexpected, as well as for those that are valid and expected. The program should generate correct messages when an invalid test is encountered and should generate correct results when the test is valid. Many errors that are suddenly discovered in production software turn up when it is used in some new or unexpected way. It is hard, if not impossible, to define all the use cases for software testing. Therefore, test cases representing unexpected and invalid input conditions seem to have a higher error detection yield than do test cases for valid input conditions.

Software Testing PrinciplesTest cases must be written for input conditions that are invalid and unexpected, as well as for those that are valid and expected. The program should generate correct messages when an invalid test is encountered and should generate correct results when the test is valid. Many errors that are suddenly discovered in production software turn up when it is used in some new or unexpected way. It is hard, if not impossible, to define all the use cases for software testing. Therefore, test cases representing unexpected and invalid input conditions seem to have a higher error detection yield than do test cases for valid input conditions.

Examining a program to see if it does not do what it is supposed to do is only half the battle; the other half is seeing whether the program does what it is not supposed to do. Programs must be examined for unwanted side effects. For instance, a payroll program that produces the correct paychecks is still an erroneous program if it also produces extra checks for nonexistent employees, or if it overwrites the first record of the personnel file.

Software Testing Principles

Avoid throwaway test cases unless the program is truly a throwaway program. A common practice is to sit at a terminal and invent test cases on the fly, and then send these test cases through the program. The major issue is that test cases represent a valuable investment that, in this environment, disappears after the testing has been completed. Whenever the program has to be tested again (e.g., after correcting an error or making an improvement), the test cases must be reinvented. More often than not, since this reinvention requires a considerable amount of work, people tend to avoid it. Therefore, the retest of the program is rarely as rigorous as the original test, meaning that if the modification causes a previously functional part of the program to fail, this error often goes undetected. Saving test cases and running them again after changes to other components of the program is known as regression testing.

Do not plan a testing effort under the tacit assumption that no errors will be found. This is a mistake project managers often make and is a sign of the use of the incorrect definition of testing testing is the process of showing that the program functions correctly. Even after extensive testing and error correction, it is safe to assume that errors still exist; they simply have not yet been found.

Software Testing PrinciplesDo not plan a testing effort under the tacit assumption that no errors will be found. This is a mistake project managers often make and is a sign of the use of the incorrect definition of testing testing is the process of showing that the program functions correctly. Even after extensive testing and error correction, it is safe to assume that errors still exist; they simply have not yet been found.

The probability of the existence of more errors in a section of a program is proportional to the number of errors already found in that section. Errors tend to come in clusters and that, in the typical program, some sections seem to be much more prone to errors than other sections, although nobody has supplied a good explanation of why this occurs. The phenomenon is useful in that it gives us insight or feedback in the testing process. If a particular section of a program seems to be much more prone to errors than other sections, then this phenomenon tells us that, in terms of yield on our testing investment, additional testing efforts are best focused against this error-prone section.

Software Testing PrinciplesTesting is an extremely creative and intellectually challenging task. It is probably true that the creativity required in testing a large program exceeds the creativity required in designing that program. We already have seen that it is impossible to test a program sufficiently to guarantee the absence of all errors.

Keep software static during test. The program must not be modified during the implementation of the set of designed test cases.

Provide expected test results if possible. A necessary part of test documentation is the specification of expected results, even if providing such results is impractical.

References

[1] http://istqbexamcertification.com/what-are-the-principles-of-testing/

[2] http://www.testingexcellence.com/seven-principles-of-software-testing/

[3] http://www.guru99.com/software-testing-seven-principles.html

[4] http://www.istqb.org/images/Articles/kale_The%20key%20principle%20of%20testing.pdf

[5] The Art of Software Testing by Glenford J. Myers.

Next Steps

Getting Started Guide

GETTING STARTED RESOURCES FREE TRIAL

DocumentationSupport

TutorialsWhat we're reading this weekBlog

Start you 30-day free trial now