paper 13d

The Structured Testing Methodology for Software Quality Analyses of

Networking Systems

Vladimir Riabov, Ph.D.Associate Professor

Department of Mathematics & Computer Science

Rivier College, Nashua, NH

E-mail: [email protected]

56th Northeast Quality Council Conference

Mansfield, Massachusetts, October 17-18, 2006

56th NEQC Conference, 2006

Complexity Metrics for Networking Software Studies

2

Agenda:• Structured Software Testing Methodology & Graph Theory:

Approach and Tools;• McCabe’s Software Complexity Analysis Techniques;• Results of Code Complexity Analysis for two industrial projects in

Networking;• Study of Networking Protocols Implementation;• Predicting Code Errors;• Test and Code Coverage;• Conclusion: What have the Graph Theory and Structured Testing

Methodology done for us?

Developing Complex Computer Systems“If you don’t know where you’re going, any road will do,” - Chinese Proverb

“If you don’t know where you are, a map won’t help,” - Watts S. Humphrey

“You can’t improve what you can’t measure,” - Tim Lister



3

McCabe’s Structured Testing Methodology Approach and Tools

• McCabe’s Structured Testing Methodology is:- a unique methodology for software testing developed in 1976

[IEEE Transactions on Software Engineering, Vol. SE-2, No. 4, 1976, pp. 308-320];

- based on the Theory of Graphs;- approved as the NIST Standard (1996) in the structured testing;- a leading tool in computer, IT, and aerospace industries (HP, GTE, AT&T, Alcatel, GIG, Boeing, NASA, etc.) since

1977;- provides Code Coverage Capacity.

• Author’s Experience with McCabe IQ Tools since 1998:- leaded three projects in networking industry that required Code

Analysis, Code Coverage, and Test Coverage;- completed BCN Code Analysis with McCabe Tools;- completed BSN Code Analysis with McCabe Tools;- studied BSN-OSPF Code Coverage & Test Coverage.



4

McCabe’s Publication on the Structured Testing Methodology (1976)

NIST Standard on the Structured Testing Methodology (1996)



5

McCabe’s Structured Testing Methodology

• The key requirement of structured testing is that all decision outcomes must be exercised independently during testing.

• The number of tests required for a software module is equal to the cyclomatic complexity of that module.

• The software complexity is measured by metrics: - cyclomatic complexity, v - essential complexity, ev - module design complexity, iv - system design, S0 = iv - system integration complexity, S1 = S0 - N + 1 for N

modules - Halstead metrics, and 52 metrics more.

• The testing methodology allows to identify unreliable-and- unmaintainable code, predict number of code errors and maintenance efforts, develop strategies for unit/module testing, integration testing, and test/code coverage.



6

Basics: Analyzing a Software Module

• For each module (a function or subroutine with a single entry point and a single exit point), an annotated source listing and flowgraph is generated.

• Flowgraph is an architectural diagram of a software module’s logic.

1 main()2 {3 printf(“example”);4 if (y > 10)5 b();6 else7 c();8 printf(“end”);9 }

Statement CodeNumber

main Flowgraph

node:statement or blockof sequential statements

condition

end of condition

edge: flow of controlbetween nodes

1-3

4

5 7

8-9

Battlemap

main

b c printf



7

if (i) ; if (i) ; else ; if (i || j) ;

do ; while (i); while (i) ; switch(i) { case 0: break; ... }

Flowgraph Notation (in C)

if (i && j) ;



8

Flowgraph and Its Annotated Source Listing

Module: marketing

Annotated Source Listing

Program : corp4 09/23/99File : ..\code\corp4.iLanguage: instc_nppModule Module Start Num ofLetter Name v(G) ev(G) iv(G) Line Lines------ ----------------------------------------------------------- ----- ------ B marketing 2 1 2 16 10

16 B0 marketing()17 {18 int purchase;1920 B1* B2 purchase = query("Is this a purchase");21 B3 if ( purchase == 1 )22 B4* B5 development();23 else24 B6* B7 B8 support();25 B9 }

0

1*

2

3

4*

5

6*

7

8

9

Origin information

Node correspondence

Metric information

Decision construct



9

Would you buy a used car from this software?

• Problem: There are sizeand complexity boundariesbeyond which softwarebecomes hopeless– Too error-prone to use– Too complex to fix– Too large to redevelop

• Solution: Control complexityduring development andmaintenance– Stay away from the boundaries.



10

Important Complexity Measures

• Cyclomatic complexity: v = e - n + 2 (here: e = edges; n = nodes) – Amount of decision logic

• Essential complexity: ev– Amount of poorly-structured logic

• Module design complexity: iv– Amount of logic involved with subroutine calls

• System design complexity: S0 = iv – Amount of independent unit (module) tests for a system

• System integration complexity: S1 = S0 - N + 1 – Amount of integration tests for a system of N modules.



11

• Cyclomatic complexity, v - a measure of the decision logic of a software module.– Applies to decision logic embedded within written

code.– Is derived from predicates in decision logic.– Is calculated for each module in the Battlemap.– Grows from 1 to high, finite number based on the

amount of decision logic.– Is correlated to software quality and testing quantity;

units with higher v, v > 10, are less reliable and require high levels of testing.

Cyclomatic Complexity



12

Cyclomatic Complexity 1

4

2

6

7

8

9

11

13

14

15

3 5

10 12

region method regions = 11Beware of crossing lines

R1 R2

R3 R4

R5

R6R7

R8R9

R10

R11

19

23

1

23

45

67

89

10

11

12

13

1415

16

17

18

2021

22

23

24

edges and node methode = 24, n = 15v = 24 - 15 + 2 = 11v = 11

=2

=1

=1

=2

=1

=1

=1

=1predicate methodv = + 1v = 11

(Measure of independent logical decisions in the module)



13

Branching out of a loop Branching in to a loop

Branching into a decision

Branching out of a decision

Essential Complexity - Unstructured Logic



14

Essential Complexity, ev

• Flowgraph and reduced flowgraph after structured constructs have been removed, revealing decisions that are unstructured.

v = 5 Reduced flowgraphv = 3

Therefore ev of the original flowgraph = 3

Superimposedessential flowgraph



15

Essential Complexity, ev

Good designs

Can quicklydeteriorate!

v = 10 ev = 1

v = 11 ev = 10

• Essential complexity helps detect unstructured code.



16

Module Design Complexity, iv

main

progeprogd

iv = 3

Therefore,

iv of the original flowgraph = 3

Reduced Flowgraph

v = 3

proge()

progd()

main v = 5

proge()

progd()

• Example:

main(){

if (a == b) progd();if (m == n) proge();switch(expression){case value_1:

statement1;break;

case value_2:statement2;break;

case value_3:statement3;

}}

do not impact calls



17

Module Metrics Reportv, number of unit test paths for a module

Total number of test paths for all modules

iv, number of integration tests for a module

Average number of testpaths for each module

Page 1 10/01/99 Module Metrics Report

Program: less Module Name Mod # v(G) ev(G) iv(G) File Name------------- ----- ------ ----- ----- ------------------CH:fch_get 118 12 5 6 ..\code\CH.ICH:buffered 117 3 3 1 ..\code\CH.Ich_seek 105 4 4 2 ..\code\CH.Ich_tell 108 1 1 1 ..\code\CH.Ich_forw_get 106 4 1 2 ..\code\CH.Ich_back_get 110 6 5 5 ..\code\CH.Iforw_line 101 11 7 9 ..\code\INPUT.Iback_line 86 12 11 12 ..\code\INPUT.Iprewind 107 1 1 1 ..\code\LINE.Ipappend 109 36 26 3 ..\code\LINE.Icontrol_char 119 2 1 1 ..\code\OUTPUT.Icarat_char 120 2 1 1 ..\code\OUTPUT.Iflush 130 1 1 1 ..\code\OUTPUT.Iputc 122 2 1 2 ..\code\OUTPUT.Iputs 100 2 1 2 ..\code\OUTPUT.Ierror 83 5 1 2 ..\code\OUTPUT.Iposition 114 3 1 1 ..\code\POSITION.Iadd_forw_pos 99 2 1 1 ..\code\POSITION.Ipos_clear 98 2 1 1 ..\code\POSITION.IPRIM:eof_bell 104 2 1 2 ..\code\PRIM.IPRIM:forw 95 15 8 12 ..\code\PRIM.IPRIM:prepaint 94 1 1 1 ..\code\PRIM.Irepaint 93 1 1 1 ..\code\PRIM.Ihome 97 1 1 1 ..\code\SCREEN.Ilower_left 127 1 1 1 ..\code\SCREEN.Ibell 116 2 1 2 ..\code\SCREEN.Ivbell 121 2 1 2 ..\code\SCREEN.Iclear 96 1 1 1 ..\code\SCREEN.Iclear_eol 128 1 1 1 ..\code\SCREEN.Iso_enter 89 1 1 1 ..\code\SCREEN.Iso_exit 90 1 1 1 ..\code\SCREEN.Igetc 91 2 1 2 ..\code\TTYIN.I------------- ----- ------ ----- ----- ------------------Total: 142 93 82Average: 4.44 2.91 2.56Rows in Report: 32



18

Low Complexity Software• Reliable

– Simple logic• Low cyclomatic complexity (v < 10)

– Not error-prone– Easy to test

• Maintainable– Good structure

• Low essential complexity (ev < 4)

– Easy to understand– Easy to modify



19

Moderately Complex Software

• Unreliable– Complicated logic

• High cyclomatic complexity (v >> 10)

– Error-prone– Hard to test

• Maintainable– Can be understood– Can be modified– Can be improved



20

Highly Complex Software• Unreliable

– Error prone– Very hard to test

• Unmaintainable– Poor structure

• High essential complexity (ev >> 10)

– Hard to understand– Hard to modify– Hard to improve



21

McCabe QA

McCabe QA measures software quality with industry-standard metrics

– Manage technical risk factors as software is developed and changed

– Improve software quality using detailed reports and visualization

– Shorten the timebetween releases

– Develop contingency plans to address unavoidable risks



22

Processing with McCabe QA Tools

BUILDLevel

TESTLevel

ANALYSISLevel

Preprocess Compile& Link

Run& Test

CM

ClearCase

PARSE

src files

src

*.E

Battlemap

Flowgraphs

Reports

Text Graphics

Test Plan

Instrumented srcinst-src; inst-lib.c

inst-src

Inst-lib.c

inst.exe

Output

IMPORT

Trace File

CoverageAnalysis

CoverageReport

Project CodeTraditional Procedures

New McCabe’s Procedures



23

Project B: Backbone™ Concentration Node



24

Project B: Backbone Concentration Node

• This system has been designed to support carrier networks. It provides both services of conventional Layer 2 switches and the routing and control services of Layer 3 devices.

• Nine protocol-based sub-trees of the code (3400 modules written in the C-programming language for BGP, DVMRP, Frame Relay, ISIS, IP, MOSPF, OSPF2, PIM, and PPP protocols) have been analyzed.



25

Annotated Source Listing for the OSPF-module Flowgraph for the OSPF-module



26

Cyclomatic Test Paths for the OSPF-module 1st Test Flowgraph for the OSPF-module



27

Module Metrics for the OSPF Protocol Suite Halstead Metrics for the OSPF Protocol Suite



28

Example 1: Reliable and Maintainable Module Example 2: Unreliable Module that difficult to maintain



29

Example 3: Absolutely Unreliable and Unmaintainable Module

Summary of Modules’ Reliability and Maintainability



30

Project-B Protocol-Based Code Analysis

• Unreliable modules: 38% of the code modules have the Cyclomatic Complexity more than 10 (including 592 functions with v > 20);

• Only two code parts (FR, ISIS) are reliable;• BGP and PIM have the worst characteristics (49% of

the code modules have v > 10);• 1147 modules (34%) are unreliable and

unmaintainable with v > 10 and ev > 4;• BGP, DVMRP, and MOSPF are the most unreliable

and unmaintainable (42% modules);• The Project-B was cancelled.



31

Project-B Code Protocol-Based Analysis (continue)

• 1066 functions (31%) have the Module Design Complexity more than 5. The System Integration Complexity is 16026, which is a top estimation of the number of integration tests;

• Only FR, ISIS, IP, and PPP modules require 4 integration tests per module. BGP, MOSPF, and PIM have the worst characteristics (42% of the code modules require more than 7 integration tests per module);

• B-2.0.0.0int18 Release potentially contains 2920 errors estimated by the Halstead approach. FR, ISIS, and IP have relatively low (significantly less than average level of 0.86 error per module) B-error metrics. For BGP, DVMRP, MOSPF, and PIM, the error level is the highest one (more than one error per module).



32

Comparing Project-B Core Code Releases



33

Comparing Project-B Core Code Releases

• NEW B-1.3 Release (262 modules) vs. OLD B-1.2 Release (271 modules);• 16 modules were deleted (7 with v >10);• 7 new modules were added (all modules are reliable with v < 10, ev = 1);• Sixty percent of changes have been made in the code modules with

the parameters of the Cyclomatic Complexity metric more than 20.• 63 modules are still unreliable and unmainaitable;• 39 out of 70 (56%) modules with v >10 were targeted for changing

and remained unreliable;• 7 out of 12 (58%) modules have increased their complexity v > 10;• Significant reduction achieved in System Design (S0) and System

Integration Metrics (S1):S1 from 1126 to 1033; S0 from 1396 to 1294.

• New Release potentially contains less errors: 187 errors (vs. 206 errors) estimated by the Halstead approach.

• The Project-B was cancelled.



34

Project C: Broadband Service Node

• Broadband Service Node (BSN) allows service providers to aggregate tens of thousands of subscribers onto one platform and apply customized IP services to these subscribers;

• Different networking services [IP-VPNs, Firewalls, Network Address Translations (NAT), IP Quality-of-Service (QoS), Web steering, and others] are provided.



35

Project-C Code Subtrees-Based Analysis

• THREE branches of the Project-C code (Release 2.5int21) have been analyzed, namely RMC, CT3, and PSP subtrees (23,136 modules);

• 26% of the code modules have the Cyclomatic Complexity more than 10 (including 2,634 functions with v > 20); - unreliable modules!

• All three code parts are approximately at the same level of complexity (average per module: v = 9.9; ev = 3.89; iv = 5.53).

• 1.167 Million lines of code have been studied (50 lines average per module);

• 3,852 modules (17%) are unreliable and unmaintainable with v > 10 and ev > 4;

• Estimated number of possible ERRORS is 11,460;• 128,013 unit tests and 104,880 module integration tests should be

developed to cover all modules of the Project-C code.



36

Project-C Protocol-Based Code Analysis

• NINE protocol-based areas of the code (2,141 modules) have been analyzed, namely BGP, FR, IGMP, IP, ISIS, OSPF, PPP, RIP, and SNMP.

• 130,000 lines of code have been studied.• 28% of the code modules have the Cyclomatic Complexity more than

10 (including 272 functions with v > 20); - unreliable modules!• FR & SNMP parts are well designed & programmed with few possible

errors. • 39% of the BGP and PPP code areas are unreliable (v > 10).• 416 modules (19.4%) are unreliable & unmaintainable (v >10 & ev >4).• 27.4% of the BGP and IP code areas are unreliable & unmaintainable.• Estimated number of possible ERRORS is 1,272;• 12,693 unit tests and 10,561 module integration tests should be

developed to cover NINE protocol-based areas of the Project-C code.



37

Correlation between the Number of Error Submits, the Number of Unreliable Functions (v > 10), and the Number of Possible Errors for Six Protocols

0

100

200

300

400BGP

FR

IP

ISIS

OSPF

RIP Submits

UnreliableFunctions

Possible Errors



38

Correlation between the Number of Customer Reports, the Number of Unreliable Functions (v > 10), and the

Number of Possible Errors for Five Protocols

0

100

200

300

400BGP

FR

ISISOSPF

RIPCustomer Reports

UnreliableFunctions

Possible Errors



39

Project-C: Code Coverage



40

Project-C: Test Coverage



41

The Structured Testing Methodology (based on the Theory of Graphs) has done for us:

• Identified complex code areas (high v).• Identified unreliable & unmaintainable code (v >10 & ev >4).• Predicted number of code errors and maintenance efforts [Halstead B,

E-, and T-metrics].• Estimated manpower to develop, test, and maintain the code.• Developed strategies for unit/module testing, integration testing.• Provided Test & Code Coverage [paths vs. lines].• Identified “dead” code areas.• Improved Software Design and Coding Standards.• Improved Reengineering Efforts in many other projects.• Validated Automated Test Effectiveness.

paper 13d

Documents

control complexity

complexity boundaries

v essential complexity

integration testing

ev module design complexity

testing quantityunits

software modules logic

code coverage conclusion