January 28, 2020 Sam Siewert

SE420Software Quality Assurance

Lecture 3 – Unit Testing & Integration TestingPart-1

RemindersAssignment #1 Submission– Assigned on Thursdays– Dues Sunday 10 days later– Late due date 13 days later

Grading in progress

Assignment #2

Goal to practice code walk-throughs and start class on holding walk-throughs once SQA groups have formed

Sam Siewert 2

Reading This WeekSQA Text Chapt. 1&2 Complete, Start Chapter 3 as Planned, Chapter 4 with Assignment #2

http://softwaretestingfundamentals.com/

Skim SWEBOK, Chapter 4– Note 4.3 White-box, or “Glass-box” – tests based on information about how the

software has been designed and coded [synonyms in SWBOK and ISTQB]– Black-box – test cases relying only on input/output behavior– Gray-box - A test that has both Black-box and White-box qualities– Visibility and/or intrusiveness of a unit test is important consideration

Unit Testing – White-box, Black-box, Coverage CriteriaIntegration Testing – Find Defects in the Interfaces and InteractionBetween Modules and Components of a SubsystemSystem Testing – Does Implementation Meet Requirements?Acceptance Testing – Does Implementation Meet Customer Needs? Regression Testing – Did Anything Get Broken by Code or Configuration Changes?

Sam Siewert 3

Notes on Unit TestingWhite-box = Glass-box, but re-use distinction for COTS is noted in research

We will look at test harness intrusion and visibility

E.g. COTS is not necessarily full-source re-use (white vs. glass box reuse)

Gray-box implies a test with both White-box and Black-box

Sam Siewert 4

https://www.aviation-ia.com/product-categories/600-series,https://www.windriver.com/products/vxworks/certification-profiles/,https://www.windriver.com/universities/

When Does Testing Start?(Design, Development, Test of Test, Use)

Test Plan, Test Case, and Test Support and Policies for Lifecycle Concurrent with Analysis, Design, Development

Acceptance Test Design First (We’ll Look at that Next) – Top Down

For Purpose of Class, We’ll Go Bottom Up for Testing

Sam Siewert 5

Reg

ress

ion

Test

ing

Testing starts with SWE Kick-off and is ConcurrentCross-Validation During Each Phase of Test Design and Dev.Test Execution Starts with Units, With Design if Possible

https://insights.sei.cmu.edu/sei_blog/2013/11/using-v-models-for-testing.html

Software Unit - DecompositionModule, Component, CSU (Computer Software Unit - MIL-STD-498)

– Composes a Subsystem– Requires a Test Driver for Unit Test– Smallest Unit of Re-use– Could be an Object in OOA/OOD/OOP– Cohesion – functions (methods) that belong together– Loose Coupling – reliance on other modules

Subsystem, CSCI (Computer Software Configuration Item – MIL-STD-498)

– Specific Use Cases or Services– Can Stand Alone, Tested Alone Up to System/Subsystem Level

System, CSC (Computer Software Configuration)– A Hierarchy of Subsystems with Interfaces and Interaction Clearly Defined– Interface Control Document– Concurrent Engineering (Subsystem Teams)– Independent Testing (Validation & Verification)

System of Systems - http://sosengineering.org/2014/ (Research Area)

Sam Siewert 6

Goal this Week – Unit TestingSimple Unit Example

– Stick with the Crypto (Simple Substitution and Transposition)– Expand to include RAID-10 and RAID-50? (Simple Stripe and Mirror or Parity)– Option for Simple Image Processing – E.g. 3 transforms

Ideal– Requirements fully Validated (baseline)– System design fully validated– Architecture validated– Module Detailed Design validated and verified– Code Construction Complete According to Coding Standards– Start Testing Unit Test– Verify Module, Debug, Deliver to SQA

Reality– Proof-of-Concept or Exploratory Coding– Requirements, System, Architecture, Design will be Refined Over Time– Adaptation of Previously Developed Modules (Module Re-use)– Sustaining Engineering – Bug Fixes and Performance Improvements

Sam Siewert 7

Black-Box (Outside Code under Test)Expected Functionality of the Unit

– Focus on Input (test case) and Output (expected)– Can We test all Input Combinations? - Not Likely– Strategies for Test Cases (Negative, Positive)

Initial Conditions and Start-up– Test Empty or Minimum Inputs – No Input– Global State at start-up?

Boundary Value - Corner Cases and Pattern Tests– Extreme values (Max, Min, and Combinations)– Empty, Full, Max/Min Alternating, Etc.

Stress Testing and Soak Testing (Stability)– Random Inputs for 1000’s of Calls– Run Continuously Over night

Fitness Testing (Selection of Functional test cases)– Is the Output as Expected– E.g. Random Number Produces Uniform Distribution)– Does the Output meet Mathematical Requirements?

Performance Testing– Operations / second, with/without Compiler Optimization– On Reference Hardware– Matching Expected Algorithmic Complexity?

C or Script Driver for Each Case Sam Siewert 8

Fitness testing

Harman, Mark, and John Clark. "Metrics are fitness functions too." 10th International Symposium on Software Metrics, 2004. Proceedings.. Ieee, 2004.

Ahmed, Amr AbdelFatah, Mohamed Shaheen, and Essam Kosba. "Software testing suite prioritization using multi-criteria fitness function." 2012 22nd International Conference on Computer Theory and Applications (ICCTA). IEEE, 2012.

Boundary value analysis testing• Function with n parameters covered?• Ring buffer (full or empty?)• Tree is balanced?• Database is normalized?

Khan, Mohd Ehmer. "Different approaches to black box testing technique for finding errors." International Journal of Software Engineering & Applications 2.4 (2011): 31.

Nidhra, Srinivas, and Jagruthi Dondeti. "Black box and white box testing techniques-a literature review." International Journal of Embedded Systems and Applications (IJESA) 2.2 (2012): 29-50.

Black-Box Fitness Test [ rand(), srand() ]

Sam Siewert 9

C++ Test Driver

Is this Distribution Uniform?

How Close to Uniform?

Is it a Fair Test?

Should We Test more Iterations?

Is C++ rand() good?

Fit to Ship?

1) Good random number generators are (not so) easy to find, P. Hellekalek2) Random Number Generators: Good Ones are Hard to Find, S. Park, K. Miller

Random Number Generators – Improved?Why are they all suspect?

Not truly random, they emulate randomness –uniformity?

Must seed the sequence –with what?– Clock – same 2x per day or at

least 1x each day– Date and time?– Something else?

The point is to assess how good or how bad?

Repetition in pseudo-random sequence? For any seed? Sam Siewert 10

Randomness

Better left to nature

Groovy Baby!

Methods to seed pseudo-random

Cloudflare – Lava lamp wall

Tom Scott on Cloudflare method

What’s not to like?

Other options – Patent #1, #2, paper

Does Seed Affect Quality of rand()?

Sam Siewert 11

White-Box (Inside Code under Test)Single Step Debug and Examine State

Basic Block – Function Body or Entry to Exit Point– Instruction Path Length– Clock / Time Path Length– Does the Block Return or Terminate?

Coverage Criteria– Function (method)– Path– Statement– Instruction (Instructions Can Be Conditional)– Short-Circuit Logic Coverage

Sam Siewert 12

Automatic Static EvaluationsCode Complexity Based on Number of Paths [Structure]Principle of McCabe Metric (We will study in depth later)

Sam Siewert 13

Most complexDo you agree?

Short Circuit LogicIf (func_A() && func_B()) then do { funct_C() } else do { funct_D()};– If result of func_A() is FALSE (zero), do we need to call

func_B()?– Have we fully tested if we cover “do” and “else do”?– Why or why not?– Why do Compilers Implement Short-Circuit Logic?

If(func_A() or func_B()) then do { funct_C() } else do { funct_D()};

Any conjunctive logic

Sam Siewert 14

Short-Circuit Logic ExampleDo the Test Cases Provide Full Coverage of A, B, C, D?

Sam Siewert 15

#include <stdio.h>

int function_A(void){

static int toggle_A=0;

printf("function_A\n");if(toggle_A == 0)

toggle_A=1;else

toggle_A=0;

return toggle_A;}

int function_B(void){

static int toggle_B=0;

printf("function_B\n");if(toggle_B == 0)

toggle_B=1;else

toggle_B=0;

return toggle_B;}

void function_C(void){

printf("do function_C\n");}

void function_D(void){

printf("do function_D\n");}

int main(void){

int rc;

// Test Case #1 - Call all functionsrc=function_A();rc=function_B();function_C();function_D();

// Test Case #2, Test use in logicif((rc=(function_A() && function_B())))

function_C();else

function_D();

return(1);}

Potential short-circuit

Gcov Analysis of Previous Example(Code Under Test – A, B, C, D)

Sam Siewert 16

-: 0:Source:sclogic.c-: 0:Graph:sclogic.gcno-: 0:Data:sclogic.gcda-: 0:Runs:1-: 0:Programs:1-: 1:#include <stdio.h>-: 2:-: 3:2: 4:int function_A(void)-: 5:{-: 6: static int toggle_A=0;-: 7:2: 8: printf("function_A\n");2: 9: if(toggle_A == 0)1: 10: toggle_A=1;-: 11: else1: 12: toggle_A=0;-: 13:2: 14: return toggle_A;-: 15:}-: 16:

1: 17:int function_B(void)-: 18:{-: 19: static int toggle_B=0;-: 20:1: 21: printf("function_B\n");1: 22: if(toggle_B == 0)1: 23: toggle_B=1;-: 24: else

#####: 25: toggle_B=0;-: 26:1: 27: return toggle_B;-: 28:}-: 29:1: 30:void function_C(void)-: 31:{1: 32: printf("do function_C\n");1: 33:}-: 34:2: 35:void function_D(void)-: 36:{2: 37: printf("do function_D\n");2: 38:}

Never tested

Gcov Analysis of Previous Example(Test Driver Code Coverage)

Sam Siewert 17

-: 39:-: 40:1: 41:int main(void)-: 42:{-: 43: int rc;-: 44:-: 45: // Test Case #1 - Call all functions1: 46: rc=function_A();1: 47: rc=function_B();1: 48: function_C();1: 49: function_D();-: 50:-: 51: // Test Case #2, Test use in logic1: 52: if((rc=(function_A() && function_B())))

#####: 53: function_C();-: 54: else1: 55: function_D();-: 56:1: 57: return(1);-: 58:} Test call never made

Questions for Next TimeIdentification of Test Cases

Unit Test Driver Code Development

Automation?

Is the Test Valid, Correct?

How Do We Verify Coverage Criteria?

What are Unit Test Exit Criteria? - When are We Done?

Observation – Test Code May be More Voluminous and Complex than Code Under Test

Sam Siewert 18

January 21, 2016 Sam Siewert

Linux Skills

Building Unit Test Drivers(Strategy and Methods)

Walk-through and RAID DiscussionRead through Crypto Backgrounder

Browse code

Look at Gcov and Lcov for Crypto example – source coverage results

Discuss Quality– Defects?– Non-functional requirements improvements (readability,

portability, usability, etc.)– Strength of encryption (cryptanalysis durability)– Applications?

Sam Siewert 20

Assignment #2Use Code Example Unit– Crypto Transposition or Substitution Functions– RAID Parity or Mirroring with Striping– Image Processing– Other?

Debug to Determine Paths and To Drive Test Cases

Develop Test Drivers

Test Tests

Run Tests and Produce Test Report

Package for Delivery (Report)

Sam Siewert 21