place your school logo here verification (cs340 john knight 2004) 1 verification
TRANSCRIPT
Verification (CS340 John Knight 2004) 2 Place your School Logo Here
Validation vs. Verification Validation:
“Did We Build The Right Thing?” Does The Software Do What User Wants?
Verification: “Did We Build The Thing Right?” Does The System Implement The Specification?
Actual SoftwareIdea
Development
Validation Verification
Validation
SoftwareSpecification
Verification (CS340 John Knight 2004) 3 Place your School Logo Here
Verification Techniques Inspection:
Examine The Work Product By Eye Detailed Discussion—Laboratory Topic
Static Analysis: Have A Program Examine The Work Product Summary
Testing: For An Executable Work Product, Try It Out Summary
Formal Proof: Use Mathematics To Demonstrate The Desired Property Summary
Inspections are the best return on investment.
Always include inspection in your software processes.
Verification (CS340 John Knight 2004) 4 Place your School Logo Here
Reviews, Walkthroughs, & Inspections
A Variety Of Techniques Tried: Reviews Walkthroughs Inspections
All Based On Same Premise:Humans looking at work products
find defects quite well.
Very Effective In PracticeApplicable To All Work productsImportant Observation:
You can inspect all of aprogram but you cannot
test all of a program.
Verification (CS340 John Knight 2004) 5 Place your School Logo Here
Fagan Inspections Developed By Michael Fagan At IBM Extremely Successful, Very Popular Hard To Do Well, Easy To Botch Steps:
Planning - Fairly Brief Overview - About One Hour Preparation - Several Hours Over Several Days Inspection - About A Two-Hour Meeting Rework - Depends On What Is Found Follow-Up - Check That Necessary Changes Made
People: Moderator - Manages Process Author - Developer Of Artifact Reader - Provides Inspection Focus Inspectors - General Participants Specialists - Specific Technical Areas
Verification (CS340 John Knight 2004) 6 Place your School Logo Here
Fagan Inspections
Specifications Designs Implementations Test Plans Process Concepts Etc.
Author’sOverview
Inspector’sReading
Inspector’sReading
Inspector’sReading
Insp
ectio
n
Author’sRework
Verification (CS340 John Knight 2004) 7 Place your School Logo Here
Inspection Activities Goal:
Detect Defects In Work Product Defects Might Be Any Deviation From Desired Quality Not Just Source Code "Bugs"
Checklist-driven: Usually A Corporate Standard Set Of Checks Determines Specialists Who Have To Inspect
Inspectors Assumed To Have Read/Inspected Work Product
Moderator: Schedules Inspection Determines Length Monitors/Manages Meeting Determines Which Part Of Artifact To Inspect
Verification (CS340 John Knight 2004) 8 Place your School Logo Here
Inspection Activities Reader:
“Guides” Inspection By Reading Aloud, Line-by-line Author:
Provides Overview Answers Questions Undertakes Rework
Inspectors: Apply Product Knowledge Language/Tools Knowledge Technical Experience
Specialists—Attention To: Portability Maintenance Real-time Use Of Language Etc.
Verification (CS340 John Knight 2004) 9 Place your School Logo Here
Review—Issues With Fagan Inspections Why The Overview? Documents? No Enforced Preparation—People Tend Not To Bother... Inspection Based On Paper Documents Not Rigorous:
Driven By The Clock Dominated By Strong-Willed Individuals No Precise Format No Precise Schedule
No Matching Of Inspector Skill Levels: Senior Engineers Argue About Comments And Spelling
No Conclusions EnsuredRemains A Controversial Technology
Done Well, Inspection Is The Best Value For Money Available
Verification (CS340 John Knight 2004) 10 Place your School Logo Here
Active Reviews Developed By Parnas And Weiss At NRL Attempt To Deal With Problems Of Fagan Inspections Sophisticated Technique Some Key Points:
Inspectors Work On A Single Technical Area—Their Area Of Expertise
Inspectors Work Alone Authors Supply Questions To Inspectors To Check
Comprehension Results Collected And Product Reworked As Necessary
Found To Be Very Effective At NRL On A7
Verification (CS340 John Knight 2004) 11 Place your School Logo Here
Phased Inspections Developed By Knight And Myers At UVa Attempt To Deal With All The Problems Of Fagan Inspections Goal—A Very Rigorous, Repeatable Software Inspection Process Premise:
There is far too much to check in a single inspection.Inspections should be in tightly focused, separate steps.
Steps Are Called Phases—Each Phase Building On Prior Phases Two Types Of Phase—Single- And Multi-Inspector Use Questions And Isolated Inspection From Active Reviews Multi-inspector Phase:
Several Inspectors Work In Parallel—Try To Find All Defects In Certain Class
Inspectors Meet To Reconcile Results Experiments At UVa Indicate Considerable Benefits In Approach
Verification (CS340 John Knight 2004) 12 Place your School Logo Here
Phased Inspections
Inspector(s) A---
CheckCompliance
WithSpecificQualityGoal
Inspector(s) B---
CheckCompliance
WithSpecificQualityGoal
Assume AllPrior Checks
Inspector(s) I---
CheckCompliance
WithSpecificQualityGoal
Assume AllPrior Checks
Computer Support
Signature Signatures
Phase 1 Phase 2 Phase N
Verification (CS340 John Knight 2004) 13 Place your School Logo Here
Types Of Phases
SpecificQualityCheck Specific
QualityCheck
SpecificQualityCheck
SpecificQualityCheck
Reconciliation
(a) Individual (b) Group
Verification (CS340 John Knight 2004) 14 Place your School Logo Here
Phased Inspection—Example
DefectInspection
By Individual
DefectInspection
By Individual
DefectInspection
By Individual
ReconciliationDesign
DocumentationInspection
GlobalProgramming
PracticesInspection
InternalDocumentInspection
Source CodeFormat
Inspaction
LocalProgramming
PracticesInspection
Verification (CS340 John Knight 2004) 15 Place your School Logo Here
Cleanroom Development Developed By Mills At IBM Premise:
Provide a software development environment, a “cleanroom”,that would not permit defects to enter the software.
Draws On Analogy With Hardware Engineering Sophisticated Technique—Some Key Points:
Software Developed Without Execution Authors Not Permitted To Execute Software Some Variants Do Not Permit Compilation Software Tested By Independent Group Test Results Returned To Authors For Software Repair Quality Assessment By Statistical Estimation
Results Include: Better Productivity (Actually, Much Better) Better Quality (Actually, Much Better)
Verification (CS340 John Knight 2004) 16 Place your School Logo Here
(Insert Course #) Inspections Goal Is To Try Inspection Exercise Three-phase Phased Inspection:
Source-code Layout Programming Practices Functional Correctness
Author-supplied Questions For Comprehension Serious Measurement Of Activities Laboratory Activity To Be Presentation Of Results Post-lab Is To Be Inspection Report
Read the lab manual for lab 8.
Verification (CS340 John Knight 2004) 17 Place your School Logo Here
Static Analysis
Examine Work Product Looking For Legal But Probably Erroneous Things
Typical Uses: Set/Use Anomalies Coding Standard Checking Design-rule Checking
Static Analyzer Report
Work Product
(Any Type, NotJust Source Code
Parse InputCreate Internal RepresentationConduct Check(s)Produce Report
Verification (CS340 John Knight 2004) 18 Place your School Logo Here
Difficulties With Static Analysis Not A Panacea—Many Things Cannot Be Checked:
if a > b thenc := 1;...if d < e then
f := g / c;
-- Is c initialized on every execution? Program Complexity:
...very_large_function(a, b, c, d, e, f, g, h);
-- What is the effect of the function on the prgrm variables? Language Specific, Difficult To Construct—Hence Not As Common As They
Should Be Often Supplement Compilers (Finish The Compiler’s Job), E.g. Lint. Very High Performance System Developed By Dave Evans—Splint
Verification (CS340 John Knight 2004) 19 Place your School Logo Here
Testing Summary Testing As Verification Technique Testing And Validation Testing Compared To Debugging Testing Phases:
Unit - Integration - System Acceptance - Regression
Black Box vs. White (Clear) Box Testing Coverage:
Functional Statement Branch Path
Automation Of Test Execution Checking Correctness Of Output
Verification (CS340 John Knight 2004) 20 Place your School Logo Here
Testing In Engineering
Hydraulic Rams
AEROSPACE ENGINEERING SOFTWARE ENGINEERING
Structural Analysis By Finite Element Technique
Testing Confirms Analysis
Almost No Comprehensive Analysis Possible
Testing Establishes Conclusion
Verification (CS340 John Knight 2004) 21 Place your School Logo Here
Some Testing Terminology Confusing Terms—Use Them Properly:
Error And Erroneous State Fault And Latent Fault Failure Commission vs. Omission
Test Oracle—Hardest Part Of Testing: Beethoven Exxon
Integration Strategies: Bottom Up Top Down
Test “Case”? Software Reliability?
Verification (CS340 John Knight 2004) 22 Place your School Logo Here
Functional Testing Black Box Approach Function/Test Case Matrix Coverage Metric:
All Functions Covered Singly Function Covered In Combinations
Equivalence Partitioning Boundary Value Analysis Cause-Effect Graphing Special Cases:
Derive Cases From Conditions In Software Base Cases On Intuition
Verification (CS340 John Knight 2004) 23 Place your School Logo Here
Structural Testing White/Glass Box Approach Base Test Case Selection On Software Structure Control Flow Based Testing—Try To Meet Coverage
Metrics: Statement Coverage Branch Coverage (All Flavors) Path Coverage
How Can These Be Measured? Mutation Testing:
Test The Test Cases Benign Mutants Killing Mutants Other Defects?
Verification (CS340 John Knight 2004) 24 Place your School Logo Here
Testing Special Types Of Program Object-Oriented Programs:
Tricky... How Do You Test A Template? Programs Manipulate Objects, Not Classes (Or Templates) What Can You “Trust” When Inheriting A “Tested” Class?
User Interfaces: How Should The Input Be Generated?
Concurrent Programs: Order Of Events Test Harnesses
Real-Time Programs: Correctness Includes Timing Timing Affected By Measurement....