the structured testing methodology for software quality analyses of networking systems

The Structured Testing Methodology for Software Quality Analyses of

Networking Systems

Vladimir Riabov, Ph.D.Associate Professor

Department of Mathematics & Computer Science

Rivier College, Nashua, NH

E-mail: [email protected]

56th Northeast Quality Council Conference

Mansfield, Massachusetts, October 17-18, 2006

56th NEQC Conference, 2006

Complexity Metrics for Networking Software Studies

2

Agenda:• Structured Software Testing Methodology & Graph Theory:

Approach and Tools;• McCabe’s Software Complexity Analysis Techniques;• Results of Code Complexity Analysis for two industrial projects in

Networking;• Study of Networking Protocols Implementation;• Predicting Code Errors;• Test and Code Coverage;• Conclusion: What have the Graph Theory and Structured Testing

Methodology done for us?

Developing Complex Computer Systems“If you don’t know where you’re going, any road will do,” - Chinese Proverb

“If you don’t know where you are, a map won’t help,” - Watts S. Humphrey

“You can’t improve what you can’t measure,” - Tim Lister



3

McCabe’s Structured Testing Methodology Approach and Tools

• McCabe’s Structured Testing Methodology is:- a unique methodology for software testing developed in 1976

[IEEE Transactions on Software Engineering, Vol. SE-2, No. 4, 1976, pp. 308-320];

- based on the Theory of Graphs;- approved as the NIST Standard (1996) in the structured testing;- a leading tool in computer, IT, and aerospace industries (HP, GTE, AT&T, Alcatel, GIG, Boeing, NASA, etc.) since

1977;- provides Code Coverage Capacity.

• Author’s Experience with McCabe IQ Tools since 1998:- leaded three projects in networking industry that required Code

Analysis, Code Coverage, and Test Coverage;- completed BCN Code Analysis with McCabe Tools;- completed BSN Code Analysis with McCabe Tools;- studied BSN-OSPF Code Coverage & Test Coverage.



4

McCabe’s Publication on the Structured Testing Methodology (1976)

NIST Standard on the Structured Testing Methodology (1996)



5

McCabe’s Structured Testing Methodology

• The key requirement of structured testing is that all decision outcomes must be exercised independently during testing.

• The number of tests required for a software module is equal to the cyclomatic complexity of that module.

• The software complexity is measured by metrics: - cyclomatic complexity, v - essential complexity, ev - module design complexity, iv - system design, S0 = iv - system integration complexity, S1 = S0 - N + 1 for N

modules - Halstead metrics, and 52 metrics more.

• The testing methodology allows to identify unreliable-and- unmaintainable code, predict number of code errors and maintenance efforts, develop strategies for unit/module testing, integration testing, and test/code coverage.



6

Basics: Analyzing a Software Module

• For each module (a function or subroutine with a single entry point and a single exit point), an annotated source listing and flowgraph is generated.

• Flowgraph is an architectural diagram of a software module’s logic.

1 main()2 {3 printf(“example”);4 if (y > 10)5 b();6 else7 c();8 printf(“end”);9 }

Statement CodeNumber

main Flowgraph

node:statement or blockof sequential statements

condition

end of condition

edge: flow of controlbetween nodes

1-3

4

5 7

8-9

Battlemap

main

b c printf



7

if (i) ; if (i) ; else ; if (i || j) ;

do ; while (i); while (i) ; switch(i) { case 0: break; ... }

Flowgraph Notation (in C)

if (i && j) ;



8

Flowgraph and Its Annotated Source Listing

Module: marketing

Annotated Source Listing

Program : corp4 09/23/99File : ..\code\corp4.iLanguage: instc_nppModule Module Start Num ofLetter Name v(G) ev(G) iv(G) Line Lines------ ----------------------------------------------------------- ----- ------ B marketing 2 1 2 16 10

16 B0 marketing()17 {18 int purchase;1920 B1* B2 purchase = query("Is this a purchase");21 B3 if ( purchase == 1 )22 B4* B5 development();23 else24 B6* B7 B8 support();25 B9 }

0

1*

2

3

4*

5

6*

7

8

9

Origin information

Node correspondence

Metric information

Decision construct



9

Would you buy a used car from this software?

• Problem: There are sizeand complexity boundariesbeyond which softwarebecomes hopeless– Too error-prone to use– Too complex to fix– Too large to redevelop

• Solution: Control complexityduring development andmaintenance– Stay away from the boundaries.



10

Important Complexity Measures

• Cyclomatic complexity: v = e - n + 2 (here: e = edges; n = nodes) – Amount of decision logic

• Essential complexity: ev– Amount of poorly-structured logic

• Module design complexity: iv– Amount of logic involved with subroutine calls

• System design complexity: S0 = iv – Amount of independent unit (module) tests for a system

• System integration complexity: S1 = S0 - N + 1 – Amount of integration tests for a system of N modules.



11

• Cyclomatic complexity, v - a measure of the decision logic of a software module.– Applies to decision logic embedded within written

code.– Is derived from predicates in decision logic.– Is calculated for each module in the Battlemap.– Grows from 1 to high, finite number based on the

amount of decision logic.– Is correlated to software quality and testing quantity;

units with higher v, v > 10, are less reliable and require high levels of testing.

Cyclomatic Complexity



12

Cyclomatic Complexity 1

4

2

6

7

8

9

11

13

14

15

3 5

10 12

region method regions = 11Beware of crossing lines

R1 R2

R3 R4

R5

R6R7

R8R9

R10

R11

19

23

1

23

45

67

89

10

11

12

13

1415

16

17

18

2021

22

23

24

edges and node methode = 24, n = 15v = 24 - 15 + 2 = 11v = 11

=2

=1

=1

=2

=1

=1

=1

=1predicate methodv = + 1v = 11

(Measure of independent logical decisions in the module)



13

Branching out of a loop Branching in to a loop

Branching into a decision

Branching out of a decision

Essential Complexity - Unstructured Logic



14

Essential Complexity, ev

• Flowgraph and reduced flowgraph after structured constructs have been removed, revealing decisions that are unstructured.

v = 5 Reduced flowgraphv = 3

Therefore ev of the original flowgraph = 3

Superimposedessential flowgraph



15

Essential Complexity, ev

Good designs

Can quicklydeteriorate!

v = 10 ev = 1

v = 11 ev = 10

• Essential complexity helps detect unstructured code.



16

Module Design Complexity, iv

main

progeprogd

iv = 3

Therefore,

iv of the original flowgraph = 3

Reduced Flowgraph

v = 3

proge()

progd()

main v = 5

proge()

progd()

• Example:

main(){

if (a == b) progd();if (m == n) proge();switch(expression){case value_1:

statement1;break;

case value_2:statement2;break;

case value_3:statement3;

}}

do not impact calls



17

Module Metrics Reportv, number of unit test paths for a module

Total number of test paths for all modules

iv, number of integration tests for a module

Average number of testpaths for each module

Page 1 10/01/99 Module Metrics Report

Program: less Module Name Mod # v(G) ev(G) iv(G) File Name------------- ----- ------ ----- ----- ------------------CH:fch_get 118 12 5 6 ..\code\CH.ICH:buffered 117 3 3 1 ..\code\CH.Ich_seek 105 4 4 2 ..\code\CH.Ich_tell 108 1 1 1 ..\code\CH.Ich_forw_get 106 4 1 2 ..\code\CH.Ich_back_get 110 6 5 5 ..\code\CH.Iforw_line 101 11 7 9 ..\code\INPUT.Iback_line 86 12 11 12 ..\code\INPUT.Iprewind 107 1 1 1 ..\code\LINE.Ipappend 109 36 26 3 ..\code\LINE.Icontrol_char 119 2 1 1 ..\code\OUTPUT.Icarat_char 120 2 1 1 ..\code\OUTPUT.Iflush 130 1 1 1 ..\code\OUTPUT.Iputc 122 2 1 2 ..\code\OUTPUT.Iputs 100 2 1 2 ..\code\OUTPUT.Ierror 83 5 1 2 ..\code\OUTPUT.Iposition 114 3 1 1 ..\code\POSITION.Iadd_forw_pos 99 2 1 1 ..\code\POSITION.Ipos_clear 98 2 1 1 ..\code\POSITION.IPRIM:eof_bell 104 2 1 2 ..\code\PRIM.IPRIM:forw 95 15 8 12 ..\code\PRIM.IPRIM:prepaint 94 1 1 1 ..\code\PRIM.Irepaint 93 1 1 1 ..\code\PRIM.Ihome 97 1 1 1 ..\code\SCREEN.Ilower_left 127 1 1 1 ..\code\SCREEN.Ibell 116 2 1 2 ..\code\SCREEN.Ivbell 121 2 1 2 ..\code\SCREEN.Iclear 96 1 1 1 ..\code\SCREEN.Iclear_eol 128 1 1 1 ..\code\SCREEN.Iso_enter 89 1 1 1 ..\code\SCREEN.Iso_exit 90 1 1 1 ..\code\SCREEN.Igetc 91 2 1 2 ..\code\TTYIN.I------------- ----- ------ ----- ----- ------------------Total: 142 93 82Average: 4.44 2.91 2.56Rows in Report: 32



18

Low Complexity Software• Reliable

– Simple logic• Low cyclomatic complexity (v < 10)

– Not error-prone– Easy to test

• Maintainable– Good structure

• Low essential complexity (ev < 4)

– Easy to understand– Easy to modify



19

Moderately Complex Software

• Unreliable– Complicated logic

• High cyclomatic complexity (v >> 10)

– Error-prone– Hard to test

• Maintainable– Can be understood– Can be modified– Can be improved



20

Highly Complex Software• Unreliable

– Error prone– Very hard to test

• Unmaintainable– Poor structure

• High essential complexity (ev >> 10)

– Hard to understand– Hard to modify– Hard to improve



21

McCabe QA

McCabe QA measures software quality with industry-standard metrics

– Manage technical risk factors as software is developed and changed

– Improve software quality using detailed reports and visualization

– Shorten the timebetween releases

– Develop contingency plans to address unavoidable risks



22

Processing with McCabe QA Tools

BUILDLevel

TESTLevel

ANALYSISLevel

Preprocess Compile& Link

Run& Test

CM

ClearCase

PARSE

src files

src

*.E

Battlemap

Flowgraphs

Reports

Text Graphics

Test Plan

Instrumented srcinst-src; inst-lib.c

inst-src

Inst-lib.c

inst.exe

Output

IMPORT

Trace File

CoverageAnalysis

CoverageReport

Project CodeTraditional Procedures

New McCabe’s Procedures



23

Project B: Backbone™ Concentration Node



24

Project B: Backbone Concentration Node

• This system has been designed to support carrier networks. It provides both services of conventional Layer 2 switches and the routing and control services of Layer 3 devices.

• Nine protocol-based sub-trees of the code (3400 modules written in the C-programming language for BGP, DVMRP, Frame Relay, ISIS, IP, MOSPF, OSPF2, PIM, and PPP protocols) have been analyzed.



25

Annotated Source Listing for the OSPF-module Flowgraph for the OSPF-module



26

Cyclomatic Test Paths for the OSPF-module 1st Test Flowgraph for the OSPF-module



27

Module Metrics for the OSPF Protocol Suite Halstead Metrics for the OSPF Protocol Suite



28

Example 1: Reliable and Maintainable Module Example 2: Unreliable Module that difficult to maintain



29

Example 3: Absolutely Unreliable and Unmaintainable Module

Summary of Modules’ Reliability and Maintainability



30

Project-B Protocol-Based Code Analysis

• Unreliable modules: 38% of the code modules have the Cyclomatic Complexity more than 10 (including 592 functions with v > 20);

• Only two code parts (FR, ISIS) are reliable;• BGP and PIM have the worst characteristics (49% of

the code modules have v > 10);• 1147 modules (34%) are unreliable and

unmaintainable with v > 10 and ev > 4;• BGP, DVMRP, and MOSPF are the most unreliable

and unmaintainable (42% modules);• The Project-B was cancelled.



31

Project-B Code Protocol-Based Analysis (continue)

• 1066 functions (31%) have the Module Design Complexity more than 5. The System Integration Complexity is 16026, which is a top estimation of the number of integration tests;

• Only FR, ISIS, IP, and PPP modules require 4 integration tests per module. BGP, MOSPF, and PIM have the worst characteristics (42% of the code modules require more than 7 integration tests per module);

• B-2.0.0.0int18 Release potentially contains 2920 errors estimated by the Halstead approach. FR, ISIS, and IP have relatively low (significantly less than average level of 0.86 error per module) B-error metrics. For BGP, DVMRP, MOSPF, and PIM, the error level is the highest one (more than one error per module).



32

Comparing Project-B Core Code Releases



33

Comparing Project-B Core Code Releases

• NEW B-1.3 Release (262 modules) vs. OLD B-1.2 Release (271 modules);• 16 modules were deleted (7 with v >10);• 7 new modules were added (all modules are reliable with v < 10, ev = 1);• Sixty percent of changes have been made in the code modules with

the parameters of the Cyclomatic Complexity metric more than 20.• 63 modules are still unreliable and unmainaitable;• 39 out of 70 (56%) modules with v >10 were targeted for changing

and remained unreliable;• 7 out of 12 (58%) modules have increased their complexity v > 10;• Significant reduction achieved in System Design (S0) and System

Integration Metrics (S1):S1 from 1126 to 1033; S0 from 1396 to 1294.

• New Release potentially contains less errors: 187 errors (vs. 206 errors) estimated by the Halstead approach.

• The Project-B was cancelled.



34

Project C: Broadband Service Node

• Broadband Service Node (BSN) allows service providers to aggregate tens of thousands of subscribers onto one platform and apply customized IP services to these subscribers;

• Different networking services [IP-VPNs, Firewalls, Network Address Translations (NAT), IP Quality-of-Service (QoS), Web steering, and others] are provided.



35

Project-C Code Subtrees-Based Analysis

• THREE branches of the Project-C code (Release 2.5int21) have been analyzed, namely RMC, CT3, and PSP subtrees (23,136 modules);

• 26% of the code modules have the Cyclomatic Complexity more than 10 (including 2,634 functions with v > 20); - unreliable modules!

• All three code parts are approximately at the same level of complexity (average per module: v = 9.9; ev = 3.89; iv = 5.53).

• 1.167 Million lines of code have been studied (50 lines average per module);

• 3,852 modules (17%) are unreliable and unmaintainable with v > 10 and ev > 4;

• Estimated number of possible ERRORS is 11,460;• 128,013 unit tests and 104,880 module integration tests should be

developed to cover all modules of the Project-C code.



36

Project-C Protocol-Based Code Analysis

• NINE protocol-based areas of the code (2,141 modules) have been analyzed, namely BGP, FR, IGMP, IP, ISIS, OSPF, PPP, RIP, and SNMP.

• 130,000 lines of code have been studied.• 28% of the code modules have the Cyclomatic Complexity more than

10 (including 272 functions with v > 20); - unreliable modules!• FR & SNMP parts are well designed & programmed with few possible

errors. • 39% of the BGP and PPP code areas are unreliable (v > 10).• 416 modules (19.4%) are unreliable & unmaintainable (v >10 & ev >4).• 27.4% of the BGP and IP code areas are unreliable & unmaintainable.• Estimated number of possible ERRORS is 1,272;• 12,693 unit tests and 10,561 module integration tests should be

developed to cover NINE protocol-based areas of the Project-C code.



37

Correlation between the Number of Error Submits, the Number of Unreliable Functions (v > 10), and the Number of Possible Errors for Six Protocols

0

100

200

300

400BGP

FR

IP

ISIS

OSPF

RIP Submits

UnreliableFunctions

Possible Errors



38

Correlation between the Number of Customer Reports, the Number of Unreliable Functions (v > 10), and the

Number of Possible Errors for Five Protocols

0

100

200

300

400BGP

FR

ISISOSPF

RIPCustomer Reports

UnreliableFunctions

Possible Errors



39

Project-C: Code Coverage



40

Project-C: Test Coverage



41

The Structured Testing Methodology (based on the Theory of Graphs) has done for us:

• Identified complex code areas (high v).• Identified unreliable & unmaintainable code (v >10 & ev >4).• Predicted number of code errors and maintenance efforts [Halstead B,

E-, and T-metrics].• Estimated manpower to develop, test, and maintain the code.• Developed strategies for unit/module testing, integration testing.• Provided Test & Code Coverage [paths vs. lines].• Identified “dead” code areas.• Improved Software Design and Coding Standards.• Improved Reengineering Efforts in many other projects.• Validated Automated Test Effectiveness.

the structured testing methodology for software quality analyses of networking systems

Documents

integration testing

unitmodule testing

software modulefor

software engineering

cyclomatic complexity

bsn code analysis

code coverageconclusion

code coverage capacity