osma sas 08 infusion of perspective-based inspections for nasa iv&v (technical briefing)

1PBI_SAS_08_Tech_Shull September 2008MAC-T IVV-08-162

Dr. Forrest Shull, FCMDKurt Woodham, L-3 Communications

OSMA SAS 08

Infusion of Perspective-Based Inspections for

NASA IV&V

(Technical Briefing)


Introduction to Perspective-Based Inspection (PBI)

Tailoring of PBI for the IV&V Context

Evaluation Plan

Outline


A long history of research & application shows that structured human inspection is one of the most cost-effective practices for achieving quality software “Cost savings rule” – Cost to find & fix software defects is about 100x

more expensive after delivery than in early lifecycle phases, for certain types of defects.

IBM: 117:1 between code and use Toshiba: 137:1 between pre- and post-shipment Data Analysis Center for Software: 100:1

“Inspection effectiveness rule” – Reviews and inspections find over 50% of the defects in an artifact, regardless of the lifecycle phase applied.

50-70% across many companies (Laitenberger) 64% on large projects at Harris GCSD (Elliott) 60% in PSP design/code reviews (Roy) 50-95%, rising with increased discipline (O’Neill) … many others

Software Inspection


Planning

Preparation

DefectReportForm

Meeting

Follow-through

SoftwareArtifact

PlanningForm

Defect Correction

Form

1

2

3

4

organizer

inspector

moderatorinspectorsauthor

author

CorrectedSoftwareArtifactSoftware

Inspection

DefectCollection

Form

Roles

Activities

Products

Improving Inspections

Software Reading Techniques: Procedural guidance incorporating tested best practices for individual preparation


A procedural technique incorporating three best practices:1. Giving each reviewer a particular and unique focus (or

perspective) on the document under review Base perspectives on document stakeholders to help

understand necessary participants, relate inspection tasks to development skills

2. Making individual review of a document an active (rather than passive) undertaking

3. Articulating the quality aspects of interest

Perspective-Based Inspection (PBI)


History 2001-2003: SARP research funding

Refined basic approach, implemented on NASA projects and in classrooms, measured results

2004: Tech infusion with Flight Software Branch / GSFC Goal: Produce new Branch standards Success metric: Based on experiences on pilot(s), how much of the recommended

process is adopted by Branch?

2004: Tech infusion with United Space Alliance / JSC Goal: Reduce defect slippage over current increment Success metric: Show reduced defect slippage on PB-inspected artifacts

2007-2009: SARP funding for improved general inspection planning

Other Results and Benefits Improved defect detection effectiveness substantially (NASA & elsewhere) Helped re-invigorate inspection practices Forms the basis of industrial and academic training

Perspective-Based Inspection (PBI)


Pathfinder: Infuse PBI into the IV&V practices being applied to JWST JWST Project/Developer artifact review JWST IV&V System Reference Model (SRM) validation

Technology Provider… Adapts PBI to IV&V domain and JWST-specific context Administers training Facilitates initial reviews Monitors efficacy of PBI and documents results/recommendations

JWST IV&V Team… Receives training Conducts reviews in-line with IV&V execution Provides ancillary feedback to Technology Provider (minimal disruption)

IV&V Facility Research… Monitors progress and determines next-step expansion based on JWST

pathfinder

Adopting PBI for IV&V


JWST System Overview

Integrated Instruments• Mid-Infrared Instrument (MIRI) • Near-Infrared Spectrograph (NIRSpec)• Near-Infrared Camera (NIRCam)• Fine Guidance Sensor (FGS)

• Launch in 2013• Collaboration between NASA, ESA, CSA• Managed by GSFC• NGST – prime contractor• STScI – operator after launch


JWST Observatory Diagram

Image credit: http://www.jwst.nasa.gov/observatory.html


JWST IV&V Overview IV&V Scope

Observatory and Ground System segments

SRM-based approach starting with top-level user needs flowed systematically down through segment, elements, subsystem, and component levels Specific focus areas driven by IV&V criticality/risk analysis with close

coordination with JWST project

WBS driven SLP 09-1 Rev J (http://www.nasa.gov/centers/ivv/ims/slps/index.html) Defines validation and verification tasking to be performed at various

levels of system decomposition• Emphasis on early validation of requirements with respect to system

needs, goals, and objectives in light of three validation questions• Validated SRM plays pivotal role in validation and verification

activities throughout IV&V life-cycle


Motivation: Focus the responsibilities of each inspector Minimize overlap among inspector responsibilities Maximize union of defects found

Document with defects

High overlap Low coverage(Ad hoc)

Low overlapHigh coverage(PBI)

Adopting PBI for IV&V: Perspectives


Tailoring required for new domain, new issues, new artifacts

Adopting PBI for IV&V: Perspectives

Needs, goals, objectives elucidation

Analysis val./ver. target

Model construction

System Ref Model

Fanny May : Loan Arranger

Borrower : Borrower

A Lender : Specified Lender

Loan : Loan

verify_report()

new_loan(lender, borrowers)

new_

look_for_a_lender(lender)

look_for_a_loan(loan)

look_for_a_

update_loan(lender, borrower)

update_

lender :

new_lender(name,contact, phone_number)

update(lender)

monthly_report(lender, loans, borrowers)

identify_report_format()

Receive Monthly Report


Borrower : Borrower


Loan : Loan

verify_report()


new_



look_for_a_


update_

lender :


update(lender)





Borrower : Borrower


Loan : Loan

verify_report()


new_



look_for_a_


update_

lender :


update(lender)




Model validation

Do sources adequately define user needs?

Is model complete, correct, consistent, and testable?

Do behaviors defined in val./ver. targets map to equivalent definition in model?


Interpretation of “defect”

Avoid focusing on correctness as the one and only quality property

Any issue that would cause a problem for a stakeholder downstream.

Defect Type Application to System Reference Model

Completeness One or more diagrams or use cases are missing a representation for a key concept.

Correctness/ Compliance

A diagram or use case contains a misrepresentation of a concept intrinsic to the domain described in the general requirements.

Consistency A representation of a concept in one diagram or use case disagrees with a representation of the same concept in either the same or another diagram / use case.

Clarity A representation of a concept is unclear, and could cause a user of the document (developer, low-level designer, etc.) to misinterpret or misunderstand the meaning of the concept.

Redundancy The diagram or use case includes information that, while perhaps true, does not apply to this domain and should not be included.

Adopting PBI for IV&V: Defect Types


Adopting PBI for IV&V: Defect Types

Other sources of defect categories: Checklists and defect categories from previous work Experienced inspectors

What is it that the most experienced people are looking for?Can get from:

• Asking them!

• Noting recurring questions at inspection meetings

Defect/discrepancy report databases What types of problems seem to be common?What types of problems take the longest to close?


Reading through a document one page at a time is not the most effective approach.

In PBI every inspector receives a scenario to guide his/her work.

Based on your perspective…

Imagine you are creating some other artifact you need as part of your job…

E.g. for tester perspective, think about building a test plan

Then read through the document as you normally would to get the necessary information.

Inspectors have questions about quality to answer while they follow the scenario.

Adopting PBI for IV&V: Active Review


Initial perspectives for this domain: Modeler

Uses requirements & existing models to create the next iteration Quality foci: Appropriate level of detail in requirements; completeness

and clarity of requirements & models; consistency and correctness of models

Domain expert Assesses whether model accurately and usefully captures domain Quality foci: Correctness of requirements (independent of models);

identification of stakeholders and evaluation of usability from their POV; checking flow of control & use of reusable components

Quality assurance Assesses whether models and system can be validated properly Quality foci: Handling of exception cases / unexpected system conditions;

robustness of system; testability

Adopting PBI for IV&V: Summary


Infusion Plan Understand team-specific quality concerns Understand likely perspectives for the team Refine set of perspectives and define

procedures for each. E.g., What quality concerns map to which

perspectives What is a feasible/effective process for checking

those concerns

Finalize procedures and provide training Analyze ongoing inspection results (and

update procedures if necessary) Monitor downstream defect profile Write final report

Status: Reviewed sample models & processes;

Constructed draft PBI techniques;

Created draft training course


Evaluation Success metric: Effectiveness of PB inspection in comparison to past

inspections

Current JWST approach Project documents

Issues written against missing/incorrect/incomplete/inconsistent content in source artifacts

SRM Reviews IV&V team maintains SRM change logs, action item lists, and other sources for the

infusion team to evaluate relative effectiveness IV&V team documents discrepancy when comparing SRM against val/ver target

Strategy is to maintain current approach (issue reports, review records), but to track how PBI assisted in identifying the items Likely to involve follow-up conversation/interviews to assess results and

collect ancillary observations from the IV&V team Will balance this with objective to minimize impact to IV&V activities

Infusion Plan: Path Forward


Use measurement results to further optimize, e.g., Consistently missing a certain type of defect?

Add a new defect type and associated questions to relevant perspectives

Missing a whole set of defects?Consider whether a new perspective should be added.

Do we never find defects of type x? Are there any perspectives that don’t seem to add much that the other perspectives didn’t find?

Consider deleting the relevant defect type or perspective. Do we consistently face a lot of problems of type x?

Add more reviewers using relevant perspectives

Future Considerations


Forrest Shull

[email protected]

301-403-8970

Contact information

Kurt Woodham

[email protected]

757-644-5807

osma sas 08 infusion of perspective-based inspections for nasa iv&v (technical briefing)

Documents