OM SAIRAM

Chapter 2

Literature Review and Problem Definition

“We strongly believe that, Literature Review is the backbone for the entire research work”

2.1 Introduction to Literature Review

As part of this project analysis, a distinct literature review was conducted at the

beginning of this research work. Apart from that, while introducing the naïve and

complex concepts also this research thoroughly reviewed some more valuable

references. During the review of research, many up-to-date techniques were revised and

arranged as a bottom-top hierarchical manner to understand the research improvement

in order.

In this section we discuss about the comprehensive view of software testing, testing

terminology, testing phases, testing types, testing methodologies and problem definition

for the current research work. This section covers the important information regarding

to each of the above concept in detail. We also included the relevant proceedings,

journals and publishers information as bibliography at the end of this research work.

2.2 Software Testing

Software Testing is an important phase of Software Development Life Cycle (SDLC),

which occupies more than half of the life cycle time and development cost[9, 10].

Glenford J. Myers [9] stated in his book as “Testing is the process of executing a

program with the intent of finding error”. Correctness is a major concern of software

application to ensure the reliability while development and deployment of software

projects. Software testing is a software development activity and part of SDLC, aimed

at evaluatingproduct quality and improving it by identifying defects and problems.

Testing process will be conducted as an investigation to reveal the bugs and generate

the report of product quality under test.

Testing compares the program results with specified test cases against expected results

(oracles), policies, principles, standards, specifications and contracts etc. The main

reason for testing is to debug the application at high level apart from development stage.

At the end an application testing a test report will be generated which is a collection of

records specifies the failed functionality under various circumstances. Test report

maintains only the failed conditions and their test case values but not successful ones.

2.3 Software Testing Life Cycle (STLC)

Many software development companies in mid 90’s were not concentrated on testing

and some of them were not even recognized testing as a SDLC phase, which caused to

fail many application at production and running stage. To overcome the problems of

“software failures” by 2000 entrepreneurs were not only accepted testing as a phase of

SDLC and also introduced the lifecycle for testing as “Software Testing Life Cycle

(STLC)”. Because of testing is very complex and collection of rigorous operations, it is

required to modularize in such a manner like STLC. STLC explains the sequence of

testing operations and changes from one form to another form as shown in below figure

x.x.

Figure x.x Software Testing Life Cycle (STLC)

Software Testing Life Cycle describes the testing process as a collection of sequential

phases to be implemented to ensure the meeting of test goals. Each phase of STLC is

designed and implemented in a specific and systematic manner towards to reach the

aim. Every phase is having a unique behavior and responsibilities, which are flexible

enough to update frequently. As per the requirement, phase migrations also supported in

STLC methodology. Different organizations have different phases in STLC; however

the basis remains the same.

Below are the phases of STLC:

1. Requirement Analysis Phase

2. Test Planning Phase

3. Test Design Phase

4. Test Case Implementation Phase

5. Test Case Execution Phase

6. Conclusion Phase

7. Closure and next literation Goal setting Phase

Requirement Analysis Phase

Requirement analysis is the first phase of STLC concentrates on analyzes and study of

the test requirements. Brain storming sessions with development and domain experts

will figure out the test requirements are testable or not. At this level only the boundaries

and scope of the testing will be determined to overcome the testing track deviation

problems.

Depends on the level of complexity this analysis may be expanded to sub categories and

extended to communicate with more other target audience. If any feature is not testable,

communicate it during this phase so that the mitigation strategy can be planned. At the

end, this phase generates a report with testable and non-testable features with detailed

scope and boundaries.

Test Planning Phase

Test Planning is the initial testing process to identify the activities and resources which

would help to meet the testing objectives. In this phase, each test scenario will be

considered separately to test with refined test scenarios to reveal the bug information.

At this level testing metrics also be identified to compare the results with them. The

main goal of this phase is to set up the sequential process of various testing

methodologies applicability over a testable scenario. This phase also decides the series

of several testing techniques in a systematic manner.During planning we also try to

identify the metrics, the method of gathering and tracking those metrics.Test planning is

mainly get effected by the below considerations

- Test Strategy of the organization

- Risk management and mitigation

Planning phase also defines “WHAT” to be tested with several test conditions by

following the testing requirements documentation. Test basis and the product risks were

also a part of this phase to implement the testing. Implementing test conditions should

be traceable back to the requirement. Some more factors to be considered while finding

the test conditions are:

Levels and depth of testing

Complexity of the product

Product and project risks

Involvement of SDLC

Test Management

Skills and knowledge of the Team

Stakeholders availability

Test conditions should be very detailed and categorized for comprehensive testing. For

example, for an online transaction, you can have a test condition as “User should be

able to make a payment”. Or you can detail it out by saying “User should be able to

make payment through NEFT, debit card and credit card”. The most important

advantage of writing the detailed test condition is that it increases the test coverage,

since the test cases will be written on the basis of the test condition, these details will

trigger to write more detailed test cases which will eventually increase the coverage.

Also identify the exit criteria of the testing, i.e determine some conditions when you

will stop the testing.

Test Design Phase

This phase explains the diagrammatical representation of “HOW” to test an application.

At this level, flow of test case design series and execution series will be described with

design diagrams. Test design phase is having the following responsibilities.

Multi Condition Coverage Technique

Predicate Coverage

Branch Coverage

Statement Coverage

Boundary value analysis

Test data identification

Metrics design for traceability

Test Environment setup

Test Case Implementation Phase

Detailed test case design is the major task of this phase. Test case prioritization is also

an important activity before going to execute them. At this level all possible input

combinations will be identified and tracked in a sequential manner for execution.

Sometimes based on the given policies we need to implement the application specific

policy based test cases. Apart from the above mentioned this phase also can do the

following things.

Input Combinations identification

Boundary value Evaluation

Test Oracle identification

Test-flowgeneration

TracingPath Coverage

Test Model implementation

Test case Execution Phase

In this phase, the designed test cases will be executed with the given possible input

values. This is the major phase of software testing life cycle (STLC). This phase

implementation is dependent on two important units are Test Environment Setup and

Test Input Generation. As the name suggests, this is the Software Testing Life Cycle

phase where the actual execution takes place. But before you start your execution, make

sure that your entry criterion is met. Execute the test cases, log defects in case of any

discrepancy. Simultaneously fill your traceability metrics to track your progress.

Conclusion Phase

Conclusion phase concerns on finalizing the test case execution and generating the test

results as Testing Report. These reports might be either daily status reports or weekly

status report.This report contains the input case, test oracle value, perceived test result

and bug information status etc. These report formats will be different from stake holder

to stake holder depends on their category. If Project managers belong to testing

background then they are more interested in the technical aspect of the project, so

include the technical things in your report (number of test cases passed, failed, defects

raised, severity 1 defects).

Closure and Next Iteration

In this phase, test cases were completed the execution or not will be notified. If all the

test cases are completed than the testing will be closured as successful completion.

After this completion, non-testable programs testing approaches will be executed. Next

Iteration related techniques also be notified in this phase only.

This cyclic process will continues until clearance of all approaches and various possible

testing methodologies. This iterative approach makes the testing more robust and

reliable also.

2.4 Static Testing vs Dynamic Testing

Testing is a process of code verification and validation to reveal bugs information from

the given application. In general testing process may be done in two ways are:

- Static Testing

- Dynamic Testing

Static testing is performed in a non-runtime environment, mainly concentrates on code

verification at design and implementation phase. Static testing will analyze the program

code for code flaws, malicious loops, risky blocks, design defects and syntax errors etc.

Static testing cannot execute a program for bug detection, but some static testing tools

will inspect the program code by following the requirement documents, design policies

etc.Static testing is useful to detect the bugs at the early stage of application

development phases like design and coding. This is the early state of STLC and can be

done either manually or automatically. Document based code review is the main theme

of static testing to provide the comments on reviews. The given below documentations

are reviewed in static testing:

- Specification of Project Requirements

- Aims and Objectives

- Project scope and boundaries

- Modeling of Project (Design)

- Project Source code

- Test Plans and cases

- Test Script

Static testing enables to achieve the 100% code coverage while testing. Static testing is

the most cost effective than dynamic testing.Static testing is considered as a preventive

approach to avoid the code flaws in future. Static testing conducts the informal reviews,

Technical Reviews, Walk through, Inspection and code reviews as specified in below

figure x.x.

Informal reviews concentrates on project documentation and gives the informal

comments on it. This review cannot work to reveal the bug information from project

code. Technical reviews would be done under the verification of a technical team with

specification experts. This team performs the cross verification of implemented code

with the given specification, to ensure the follow-up of specification. In this case team

strictly verifies the proposed specification standards implementation in main project

physical code. This review concentrates mainly on the technical document related to the

software such as Test Strategy, Test Plan and requirement specification documents.

Walkthrough technique describes the project flow in a formal meeting with the

technicians, stack holders and end-user representatives. Project designer will explains

the total project structure along with flow in a modularized manner with various

representation techniques. Participants are having the right to clarify their doubts with

project designer. Any valuable comments from participants would be considered and

bookmarked by designer for considerations.

Inspection and static code review mainly used to find the bugs, by following the project

module responsibilities check list. Client and other senior technicians conduct this

formal review where the strict verification process is implemented. Code review is a

process of finding the bugs without execution of the code. This process concerns on

code syntax, standards, policies, design patterns, optimizations etc. code reviews done

in the way of white box testing, at any point during the code development. Developers,

Specification designers and client representatives play a vital role at this stage.

Figure x.x Static Testing vs Dynamic Testing

Dynamic testing is a process of finding the bugs while running the program with

various inputs. Static testing is mentioned for verification whereas dynamic testing for

validation purpose. Test oracles are used in dynamic testing to compare with the results

of test case values to determine the bug detection at run-time. Dynamic testing begins

after completion of the project component code. Unit Testing, Component Testing,

Integration Testing, System Testing are the examples for dynamic testing techniques.

This process might be done either manually or automatically with testing tools and

software APIs.

Dynamic testing is less cost effective, when compared with static testing. Dynamic

testing cannot ensure the 100% code coverage like static testing. A dynamic test will

monitor system memory, functional behavior, response time, and overall performance

of the system. Dynamic testing is an important approach to reveal the bottleneck

problems and security issues of an application at run-time.

Dynamic testing expects the test case and test oracle support to implement the testing in

a formal way. Bugs Identification Ratio will be more and less meetings are required for

the dynamic testing approach than static testing. More automation is possible in case of

dynamic testing than static testing as dynamic testing concerns on code rather than

documentation and design.Dynamic testing is performed at all levels of testing and it

can be either black or white box testing.

Dynamic testing is implemented in the form of various testing techniques as specified in

figure x.x. Unit Testing allows the developers to test individual units and modules

separately in the form of white box manner. Code developers play a vital role in this

testing. This is the very basic level of application white box testing.

Integration testing performed on grouped individual units together to identify the bugs

at integration time. This testing is used to test the co-ordination level between the

modules. Tight coupling and loose coupling dependencies also be identified at this

level. The main aim of this testing is to determine that modules are working as expected

once they are integrated.

System testing allows us to test whole system functionality after integration of

individual units as a main application. This testing will be implemented as either black

box or white box manner using the project specification.

2.5 Various Dynamic Testing Approaches

Since last two decades many dynamic testing techniques and methodologies were

introduced in the area of testing, which is essential to test any application. In recent

dynamic testing drags the attention of the testing world dramatically by utilizing its

smartness and simplicity in the way of its approach towards testing and test case

development. White box testing is an efficient dynamic testing mechanism, which

generates the test cases based on the program source code verification to identify the

hidden bug information efficiently at run-time.

2.5.1 Symbolic Testing

Symbolic testing (Symbolic Execution) is a code based (white box) testing technique,

introduced in 1970’s, emerged as a prominent methodology since last decade to test the

complex application with symbols (arbitrary values) instead of actual input values. This

process is quite similar to normal execution except using symbols behalf of real input

values. The reason behind to use symbols instead of input values is to mitigate the

burden of testing with huge amount of input values. One symbolic test case execution

result is equivalent to large number of normal test case executions, which can be

verified against various test oracles by the test engineer either formally or informally.

This process not only reduces the need of test cases creation but also saving the time

and effort of test oracle generation. But this technique is not completely alleviating the

burden of generating test oracle.

Ru-Gang Xu [11] et al was introduced four symbolic execution algorithms in his

dissertation to generate the test case automatically and systematically. In his research

articles and dissertation, was implemented the concept of automatic test case generation

to alleviate the burden of test oracle generation from symbolic testing. These algorithms

mainly concerns on Symbolic grammars, Symmetry, Live-ness and Abstraction.

Symbolic grammar helps to reduce the less magnitude string generation as input values

to achieve the high path coverage ratio. Symmetry tries to mitigate the duplicate test

case execution by using the correlations among various interdependent parts of

application. Live-ness generates the test cases which identifies the infinite execution

controls in programming without termination points. Abstraction allows checking the

memory safety violations and security holes before they occurred.

After the success of Symbolic Execution with wide range aspects, it has become a base

technique for several white box testing tools like JCUTE [12], NASA’s SPF (java) [13],

Crest [Berkeley] [14] etc. Microsoft is using symbolic execution in their SAGE and

YOGI projects, IBM introducing Apollo testing toll, which is using SE as a backbone

methodology. Although several improvements were introduced, still there is a need to

improvise this symbolic execution to explore the domain of testing range.

Symbolic Execution is a program analysis and execution technique, which executes a

program with symbols instead of actual input values. SE first analyzes the path

conditions and maintains them in a stack data structure. Each time, when they visit the

branch statement than the path condition will be updated automatically in test reports.

Constraint solver is used to generate test constraints and also to solve the path

constraints in a sequential manner. Symbolic execution is also used for assertion

validation and testing to identify the run-time bugs by using the test cases.

Symbolic execution maintains a symbolic state, whichmaps variables to symbolic

expressions, and a symbolic pathconstraint PC, a first order quantifier free formula over

symbolic expressions. PC accumulates constraints on the inputsthat trigger the

execution to follow the associated path. Atevery conditional statement if (e) S1 else S2,

PC is updated with conditions on the inputs to choose between alternative paths. A fresh

path condition PC1is created andinitialized to P C ∧¬σ(e) (“else” branch) and PC is

updatedto PC ∧ σ(e) (“then” branch), where σ(e) denotes the symbolic predicate

obtained by evaluating e in symbolic state σ.Note that unlike in concrete execution, both

branches canbe taken, resulting in two execution paths. If any of PC orPC1 becomes

UN-satisfy, symbolic execution terminatesalong the corresponding path. Satisfaction

percentage is checked witha constraint solver.

Symbolic Execution is also expanded to handle the concurrent program testing with

multiple threads and locking mechanisms. Dynamic test generation capacity and

concurrency support made SE as an emerging testing technique for real time multi-

dimensional applications.

The challenges in symbolic execution is to handle the exponential number of paths in

the code, compositional techniques, Redundant path Pruning and heuristic search etc.

Although we are using this symbolic execution to generate test cases automatically, still

there is a need of test oracle in this area and also new test case designing based on

succeed test cases is not possible to implement.

2.5.2 Constraint Based Testing

2.6 Test Automation (To Be Written …..)

9. Myers, J. G. (1978, September). A Controlled Experiment in Program Testing and Code Walkthrough /Inspections. Communications of ACM, 21(9), 760-768.

10. Beizer, B. “Software Testing Techniques” 2nd Edition, International ThomsonComputer Press 1990.

11 J. Andrews, A. Groce, M. Weston and R. Xu. Directed Test Generation

UsingSymbolic Grammars, ASE, September 2008.

12 http://osl.cs.uiuc.edu/~ksen/cute/

13 http://babelfish.arc.nasa.gov/trac/jpf/wiki/projects/jpf-symbc

14 http://code.google.com/p/crest/