verification and validation: a quick introduction 1-2 lectures
TRANSCRIPT
Verification and Validation: Verification and Validation: A Quick IntroductionA Quick Introduction
1-2 Lectures1-2 Lectures
Learning ObjectivesLearning Objectives1.1. The student will be able to define both verification and The student will be able to define both verification and
validation, and to know the difference between them.validation, and to know the difference between them.
2.2. The student will be able to identify the appropriate activities The student will be able to identify the appropriate activities and their classification as either verification or validation.and their classification as either verification or validation.
3.3. The student will be able to distinguish between verification The student will be able to distinguish between verification and validation, given a set of activities.and validation, given a set of activities.
4.4. Given a set of software artifacts, the student will understand Given a set of software artifacts, the student will understand the appropriate verification or validation activity to that the appropriate verification or validation activity to that artifact.artifact.
5.5. Given a set of projects in different categories, the student will Given a set of projects in different categories, the student will be able to determine which category of projects would be able to determine which category of projects would require more V&V activity, and which would require less.require more V&V activity, and which would require less.
6.6. Given a set of defects and the phases of development in Given a set of defects and the phases of development in which they were found, the student will be able to estimate which they were found, the student will be able to estimate the relative cost of correcting errors early or late in the the relative cost of correcting errors early or late in the lifecycle.lifecycle.
Why Invest in Quality?Why Invest in Quality? Cost effectiveCost effective Provides competitive edgeProvides competitive edge Essential for business survivalEssential for business survival Essential for international marketingEssential for international marketing Helps to retain customers, and Helps to retain customers, and
increase profitsincrease profits Hallmark of world class businessHallmark of world class business
Ways to Improve QualityWays to Improve Quality Prevention of DefectsPrevention of Defects
• Process ImprovementProcess Improvement• Complexity ReductionComplexity Reduction• Risk ManagementRisk Management• Causal AnalysisCausal Analysis
Detection and Correction of DefectsDetection and Correction of Defects• VerificationVerification• ValidationValidation• ReworkRework• Causal AnalysisCausal Analysis
VerificationVerification
Verification – Are we building the product Verification – Are we building the product right?right?
Verification is any Verification is any checking process checking process conducted on software artifacts in an conducted on software artifacts in an attempt to determine if they work as attempt to determine if they work as specified by the specified by the designers of the systemdesigners of the system..
Includes reviews, inspections, Includes reviews, inspections, walkthroughs, unit testing and integration walkthroughs, unit testing and integration testing.testing.
ValidationValidation
Validation – Are we building the right Validation – Are we building the right product?product?
Validation is the process of evaluating Validation is the process of evaluating software artifacts during the software software artifacts during the software development process in an attempt to development process in an attempt to determine if the system works as required by determine if the system works as required by the the customerscustomers. Any evaluation activity that . Any evaluation activity that involves the customer can be used for involves the customer can be used for validation purposes.validation purposes.
Includes program reviews, system testing, Includes program reviews, system testing, customer acceptance testing.customer acceptance testing.
Verification vs. ValidationVerification vs. Validation VerificationVerification
• Main purpose is to detect defects in the Main purpose is to detect defects in the artifacts of the system under artifacts of the system under development.development.
ValidationValidation• Main purpose is to show that the system Main purpose is to show that the system
under development meets user needs, under development meets user needs, requirements, and expectations.requirements, and expectations.
Verification & Validation Verification & Validation TechniquesTechniques
Static MethodsStatic Methods• Techniques applied to artifacts without Techniques applied to artifacts without
execution.execution. Dynamic MethodsDynamic Methods
• Techniques applied to artifacts through Techniques applied to artifacts through execution.execution.
Mathematically Based MethodsMathematically Based Methods
Static: ReviewsStatic: Reviews WalkthroughsWalkthroughs
• CodeCode verificationverification• DocumentDocument
ConOps, SRSConOps, SRS validationvalidation STEP, SAD, SDDSTEP, SAD, SDD verificationverification
InspectionsInspections• CodeCode verificationverification• Document AuditsDocument Audits verificationverification
Program ReviewsProgram Reviews• Customer involvedCustomer involved validationvalidation• No customerNo customer verificationverification
Effectiveness of Static VerificationEffectiveness of Static VerificationEffectiveness of Static VerificationEffectiveness of Static Verification
More than 60% of program defects can be More than 60% of program defects can be detected by program inspections.detected by program inspections.
More than 90% of program defects may be More than 90% of program defects may be detectable using more rigorous detectable using more rigorous mathematical program verification.mathematical program verification.
The defect detection process is not The defect detection process is not confused by the existence of previous confused by the existence of previous defects.defects.
Dynamic: Testing (Verification)Dynamic: Testing (Verification)
Unit Test (Detailed Design): • Testing the individual software modules,
components, or units. Integration Testing (Architectural Design):
• After unit test, the system is put together in increments. Integration testing focuses on the interfaces between software components (OO thread-based, cluster-based testing)
System Testing (Requirements Spec): • One goal of system testing is to ensure that
the system functions as specified in the specification.
Dynamic: Testing (Validation)Dynamic: Testing (Validation)
System Testing (Requirements Spec):• Another goal of system testing is to ensure
that the system functions as the client expected in a controlled environment.
User Acceptance Test (ConOps):• A set of formal tests run for the client, and
specified by the client. When the system passes these tests, the software has been accepted by the client as meeting the requirements.
Verification and Validation in the Verification and Validation in the Development LifecycleDevelopment Lifecycle
CodeExecute Unit
Tests
White-Box &Black-Box
Testing
Black-BoxTesting
RequirementsAnalysis
ExecuteSystem Tests
Execute Integration
TestsDesign
Validate the System
Verify Design
Verify Implementation
Review
s
Mathematics-Based VerificationMathematics-Based Verification
Verification is based on mathematical Verification is based on mathematical arguments which demonstrate that a arguments which demonstrate that a program is consistent with its program is consistent with its specification.specification.
Programming language semantics Programming language semantics must be formally defined.must be formally defined.
The program must be formally The program must be formally specified.specified.
Why Inspection is important?Why Inspection is important? Relative cost to fix a defectRelative cost to fix a defect Phase in which found Cost ratio Phase in which found Cost ratio
(hours) (hours)– – Requirement Requirement 1 1– – DesignDesign 3-6 3-6– – CodingCoding 10 10– – TestingTesting 15-70 15-70– – OperationOperation 40-100040-1000
Data derived from Capers Jones.
Costs of Finding and Fixing LateCosts of Finding and Fixing Late
Delays in identifying and fixingdefects gets geometrically more
expensive as the lifecycle progresses!
$1
$10
$100
$1000
Cost
of
Fix
ing
a D
efe
ct
Defe
cts
Time in (Phase of) Development
Cost of QualityCost of Quality
Includes all costs of quality-related Includes all costs of quality-related activities.activities.
Quality costs =Quality costs =+Prevention costsPrevention costs+Detection and Appraisal costsDetection and Appraisal costs+Failure costsFailure costs
• Internal failure costsInternal failure costs• External failure costsExternal failure costs
Quality Cost ComponentsQuality Cost Components Direct CostDirect Cost
– – Reviews/inspectionsReviews/inspections
– – Unit testingUnit testing
– – System testingSystem testing
– – Acceptance testingAcceptance testing
– – Test planning and Test planning and
designdesign
– – Computer timeComputer time
– – Resources (terminals,Resources (terminals,
staffs, etc.)staffs, etc.)
Indirect CostIndirect Cost
– – ReworkRework
– – RecoveryRecovery
– – Corrective action costCorrective action cost
– – FailuresFailures
– – Analysis meetingAnalysis meeting
– – DebuggingDebugging
– – RetestingRetesting
– – Legal feesLegal fees
Verification or… Validation?Verification or… Validation?
ReviewsReviews EitherEither Unit testingUnit testing
VerificationVerification Integration TestingIntegration Testing
VerificationVerification System testingSystem testing ValidationValidation Acceptance testingAcceptance testing ValidationValidation
Balancing Trade-OffsBalancing Trade-Offs What is the real requirement?What is the real requirement? What are the expected benefits of What are the expected benefits of
V & V vs.V & V vs.– – cost of increasing V&V activitycost of increasing V&V activity– – additional time requiredadditional time required– – technical feasibilitytechnical feasibility– – risk of not finding defectsrisk of not finding defects
» delivery» delivery» operation» operation
““When is it good enough?”When is it good enough?”(Bach, 1997)
SummarySummary Verification and Validation are both Verification and Validation are both
important to the quality of software.important to the quality of software. Each are different:Each are different:
• Verification determines that what we are doing Verification determines that what we are doing is done correctly.is done correctly.
• Validation determines whether we are doing Validation determines whether we are doing the right thing.the right thing.
Sometimes, less quality is okay. Some Sometimes, less quality is okay. Some software systems require less V&V, as the software systems require less V&V, as the cost is not supportable. Some require cost is not supportable. Some require more no matter what it costs.more no matter what it costs.
The earlier defects are discovered, the less The earlier defects are discovered, the less it will cost to correct them.it will cost to correct them.
•Design • Coding
• System Requirements• Contractor Management
• Evaluate Requirements• Evaluate Design
• Monitor Tests• Integration
TestedProducts
Discrepancies
Subs
Project Manager
IV&V
• Testing
Independent Verification and Independent Verification and Validation (IV&V)Validation (IV&V)
Static and Dynamic Static and Dynamic Verification & ValidationVerification & Validation
Formalspecification
High-leveldesign
Requirementsspecification
Detaileddesign
Program
PrototypeDynamicvalidation
Staticverification
Sommerville, p. 421
Test Case Development in the Test Case Development in the V ModelV Model
CodeExecute Unit
Tests
Test cases designed and executed by developers
Test cases designedand executed by
independent testers
RequirementsAnalysis
ExecuteSystem Tests
Execute Integration
TestsDesign
Validate Requirements
Verify Design
Verify Implementation