TEST COVERAGE: AN ART AND A SCIENCE

WITH JAY PHILIPS AND JEAN ANN HARRISON

YOUR PRESENTERS

Mobile & Software Test ConsultantEmail: yagysyjah@gmail.comTwitter: @JA_HarrisonLinkedIn Profile: http://www.linkedin.com/pub/jeanann-harrison/4/b55/865/

Company: Project RealmsWebsite: www.projectrealms.comEmail: jay@projectrealms.comTwitter: @JayphilipsLinkedIn Profile: www.linkedin.com/in/jayphilips

AGENDA

• Test Coverage Definition

• Business Impacts of Test Coverage

• Types of Test Coverage & Example

• Implementation

• Questions/Answers

TEST COVERAGE - DEFINITION

• A measure of the proportion of a program exercised by a test suite, usually expressed as a percentage. This will typically involve collecting information about which parts of a program are actually executed when running the test suite in order to identify which branches of conditional statements which have been taken.

BUSINESS IMPACT OF TEST COVERAGE

• Results can Assure the quality of test

• Results can help identify what portions of the code was actually touched for the release/fix

• Results can help identify the paths in your application that weren’t tested

• Results can identify all the paths & decision points used in the application, which will allow you to increase test coverage

TYPES OF TEST COVERAGE

CODE COVERAGE

STATEMENT COVERAGE

Statement Coverage testing can be one line of code or a block of code which executes one task. Example might be testing an If statement with no decision but fulfilling the one condition.

BRANCH COVERAGE

The code takes the tester through a series of decisions to be made when the value from the array called age is reached. The decision to do something needs to be tested and the decision to go to the next cast when the value from the array is not the value in the case. Two tests must be done for each case because there are two decisions: if yes – do this and break or no, go to the next case.

PATH COVERAGE

This code takes you down a road of the where the value of a is incremented by 1 after starting at 10. Once incremented by 1, the value of a is printed out with a new line. But, there is a change in the path, when a = 15, the path says to add 1 to the value and then go through the loop. Because a != 15, the value of a is printed. The value of a is increased by 1 and will be printed out until a reaches the value of 20 and the function stops.

TESTS FOR PATH COVERAGE EXAMPLE

The tests to follow would include checking the sequence of the values printed as well as what values are printed excluding 15. Is each value represented as “a = <value != 15”. Make sure each printed value starts on a new line.

The code’s path was written where the value of a equals 11, 12, 13, 14, 16, 17 or 18, 19, that value will be printed as: “Value of a: <numbers listed above>”. When the code’s path reaches 15, the code states that to add 1 to the value of a and then to start the loop over again, never reaching the print statement. The code doesn’t contain further instructions of adding 1 or more to the value of a but instead continues to the code stating to print out the value of a.

a=11

a=12

a=13

a=14a=15

a=16

a=17

a=18

a=19

CONDITION COVERAGE

Example of a function written in C++ where test cases are needed to cover the following Boolean conditions where z would = x if:

• Test x > 0 but y = 0• Test x > 0 but y < 0 • Test x < 0 but y > 0• Test x < 0 but y = 0• Test x = 0 but y > 0• Test x < 0 and y < 0

More tests than if x > 0 & y > 0

TEST COVERAGE FOR TYPES OF TESTS

What type of Tests Use in conjunction with: Why implementing

Unit Tests debugger toolCan help to manipulate data & simulate error conditions

Integration Tests test harness Use a test harness to simulate user interface checking on which components were tested.

Regression Tests log files

Checking overal application functionality for 2 reasons: 1) anything broken with newly added code2) checking what previously built code had been tested and touched.

Performance Tests log files

Security Tests log files

Functional Tests log files testing old vs new code – include for all.

System Integration Tests log files

Production Testing log files

IMPLEMENTATION CRITERIA

Element Consideration (what you said requirement)

Criteria

Language SupportNot all tools support all languages (ex, Java, Cobol) so determine what languages the tool will be used for to determine the tool selected

InstrumentationDetermine when the instrumentation will be done and if the tool can support it. Ex: Will it be done via instructions or probes? Will be it be via source code or byte code or other?

MeasurementDetermine which measurement type you want to use (branch, block, etc)

Reporting Determine what type of reporting you need and ability to distribute

Environment/Hardware Determine what environment(s) & hardware are needed to use the tool

TrainingDetermine when the tool vendor/consultant can provide training to your teams

Setting Expectations Before implementation, a discussion with the management stakeholders of the technical details on the source code: i.e. instrumentation.

RECOGNIZING AND SOLVING THE PROBLEM

GETTING STARTED …

Creating processes:

• Identify users, their purpose for using the software and their industry/context.

• Identify external team members (build team, development team, test team etc)

• Identify a repository location & setup for completed analysis and level of analysis to be stored.

• Identify permissions for repository

• Identify recipients of reported analysis & level of detail for the reports

• Develop content for reports, create reports

DOCUMENTING YOUR PROCESS

• Documentation Needs (especially for regulated environments)

• Setup needs for hardware, storage, team members & their roles

• Among team, create & document goals

• Chronological flow of tasks

• Training tasks

NEXT STEPS …

• Distribute & Reviewing test coverage data to stakeholders

• Using the analytics to update test cases for future testing

• Rinse and Repeat

REFERENCES• Slide 4 - Mouse with purple umbrella

http://cdn.stylefrizz.com/img/purple-umbrella-mouse.jpg

• Slide 5 - Snoopy Happy Dancehttp://4.bp.blogspot.com/--u0eQYXh6l8/U15TKvTwSOI/AAAAAAAAEQ0/PQo9QxsvQeE/s1600/pe650921_Snoopy+Happy+Dance.jpg

• Slide 6 - Code Coverage Text http://en.wikipedia.org/wiki/Code_coverage

• Slide 7 – Code Coverage Executionhttp://www.atollic.com/index.php/trueanalyzer/types-of-code-coverage-analysis

• Slide 8 - Statement Coveragehttp://www.cs.odu.edu/~cs252/Book/stmtcov.html

• Slide 9 - Branch code examplehttp://www.tutorialspoint.com/cplusplus/cpp_switch_statement.htm

• Slide 10 - Path coverage example:http://www.cs.odu.edu/~cs252/Book/branchcov.html

• Slide 12 – Condition Coverage Examplehttp://en.wikipedia.org/wiki/Code_coverage

• Slide 15 – Apollo 13 mobie cliphttps://www.youtube.com/watch?v=C2YZnTL596Q

• Slide 16 Getting Started http://blog.firespring.com/wp-content/uploads/2013/07/3-Steps-for-Getting-Started-with-Social-Media.jpg

• Slide 17 - Documenting Processeshttp://pmtips.net/wp-content/uploads/2011/09/project_issues-283x300.jpg

• Slide 18 – Next Stepshttp://thebusinessupdate.co.za/wp-content/uploads/2014/07/Steps-to-Starting-Forex-Trading-the-Right-Way.jpg