measurement and analysis m&a - 1 24 oct 01 practical software and systems measurement a...

Measurement and Analysis

M&A - 1 24 Oct 01

Practical Software and Systems MeasurementA foundation for objective project management

COCOMO Forum

October 24, 2001PSMPSM


M&A - 2 24 Oct 01

Outline

• Objectives of Measurement and Analysis Initiatives

• Practical Software and Systems Measurement

• ISO/IEC 15939, Software Measurement Process

• CMMI Measurement and Analysis Process Area

• ISO 9000-3, Guidelines for Software


M&A - 3 24 Oct 01

Objectives of Measurement Initiatives

• Experienced-Based Measurement to Support Management and Engineering Requirements

• Consistent Measurement Process Across Measurement Initiatives

• Consistent Measurement Terminology


M&A - 4 24 Oct 01

ISO/IEC15939

CMMIM&A

Practical Software and SystemsMeasurement

ISO9000-3

INFORMATION DRIVEN

MEASUREMENTPROCESS


M&A - 5 24 Oct 01

Measurement Process Scope

Core Measurement Process

Technical &ManagementProcesses

EstablishCommitment

PlanMeasurement

Experience Base

Measurement Requirements User Feedback

InformationProducts

InformationNeeds

Improvement Actions

Plan PerformanceInformation

PerformMeasurement

EvaluateMeasurement

EvaluationResults

Measurement Process Model


M&A - 6 24 Oct 01

Practical Software and Systems Measurement (PSM)


M&A - 7 24 Oct 01

What is PSM?

• A Primary Measurement and Analysis Process Used by DoD, Government, and Commercial Programs

• Technology Based on Actual Experience on Real Programs

• A Fully Integrated Approach Including Guidance, Tools, Training, Implementation Workshops, Support Products, and a Qualified Technical Team


M&A - 8 24 Oct 01

Key Measurement Concepts

• Measurement is a Process - Not a Pre-Defined List of Graphs or Reports

• The Measurement Process is Flexible - Adapted To Meet Specific Project and Organizational Objectives and Information Needs

• The Measurement Requirements are Integrated Into the Project Technical and Management Processes


M&A - 9 24 Oct 01

Common Information Needs

• Schedule and Progress

• Resources and Cost

• Product Size and Stability

• Product Quality

• Process Performance

• Technology Effectiveness

• Customer Satisfaction


M&A - 10 24 Oct 01

Problem Report StatusPriority 1, 2, and 3 Problem Reports

Build 1

0

100

200

300

400

500

600

700

800

Nu

mb

er o

f P

rob

lem

Rep

ort

s

Discovered

Closed

30 Aug 98

Start ofBuild 1 I&T

Oct 97 Jan 98 Apr 98 Jul 98 Jan 99 Apr 99Oct 98

900

1000


M&A - 11 24 Oct 01

Types of Analysis

Estimation

FeasibilityAnalysis

PerformanceAnalysis

DefinedInformation

Needs


M&A - 12 24 Oct 01

Estimation

Project Characteristics

Project DataHistorical Data

Assum ptions

Estimates Lack of Inform ation

Adjustm ents

ObjectivesConstraints

M ap and Calibrate

Com puteEstim ate

EvaluateEstim ate

SelectApproach


M&A - 13 24 Oct 01

Integrated Analysis Model

TechnologyEffectiveness

ProcessPerformance

Product Size and Stability

Resourcesand Cost

Schedule andProgress

ProductQuality

CustomerSatisfaction


M&A - 14 24 Oct 01

Integrated Analysis ExampleTest Progress

0

100

200

300

400

500

600

700

800

900

1000

1 Jan 22 Jan 12 Feb 5 Mar 26 Mar 16 Apr

Nu

mb

er

of

Re

qu

ire

me

nts

Plan

Actual

Problem Report StatusOpen By Priority

0

10

20

30

40

50

60

70

80

90

100


Nu

mb

er

of

Pro

ble

m R

ep

ort

s

High

Medium

Low

CPU Utilization

0

10

20

30

40

50

60

70

80

90

100


Pe

rce

nt

of

CP

U C

yc

les

Threshold

Software Reliability

0

20

40

60

80

100

120


Ho

urs

Be

twe

en

Fa

ilu

res

Threshold


M&A - 15 24 Oct 01

EnterpriseManagement

OrganizationalManagement

ProgramManagement

• Program Estimation and Planning• Program Performance Tracking• Program Tradeoff Analysis• Resource Management

• Process Improvement• Program Planning Guidelines• Performance Based Acquisition• Organizational Norms & Benchmarks

• Performance Measurement• Normative Performance Baselines• Technical and Business Policy• Investment Decisions & Analysis

RiskManagement

Process

Issue-DrivenMeasurement

Process

Multi-Level Requirements


M&A - 16 24 Oct 01

PSM Project Strategy

PSMProject

Approach

ComprehensiveTransition Support

Proven MeasurementTechnical Guidance

and Tools

Practical

Software

MeasurementGovernment - Industry -Academia Team

Technical Consensus of Best Practices

Incremental ProductDevelopment andImprovement

Empowered TransitionOrganizations

Integrated With OtherKey Initiatives


M&A - 17 24 Oct 01

New Technology Areas

• Measurement and New Technologies- Object-oriented developments- Spiral/evolutionary approaches- Interoperability- Product lines

• Initial Recommendations Were Draft Drafted in Early 2001

• Field Trials Are Underway - Additional Trials Will be Conducted in 2002


M&A - 18 24 Oct 01

PSM Participating Organizations

• US Navy NUWC• US Navy OPTEVFOR• ODUSD (S&T)• OSD C31• OSD PA&E• Aerospace Corp.• DCMA• DFAS • DISA• DLA• DSMC• FAA• IDA• MITRE Corp.• NDU IRM College• NASA• National Park Service• Sandia National Lab• SEI• Soc. Sec. Admin.

DoD and Government• US Air Force AFMC• US Air Force SSSG• US Air Force STSC• US Air Force CRSIP• US Air Force STRATCOM• US Army AMC• US Army ATEC • US Army CECOM• US Army ODISC4• US Army SMDC• US Army TACOM - ARDEC• USMC MCTSSA• US Navy AEDC• US Navy ASN RDA• US Navy FMSO• US Navy NAVAIR• US Navy NAVSEA• US Navy NAWC• US Navy NRL• US Navy NSWC

Overseas• Defence Materiel Org.

(Australia)• LiveWare (Argentina)• NRC (Canada)• Software and Systems

Engineering (Israel)• Software Improvements

(Australia)• Software Quality Institute

(Australia)


M&A - 19 24 Oct 01

PSM Participating Organizations

• Independent Engineering • ITABHI• James Gregory Assoc. • Kodak Health Imaging• Litton/TASC• Lockheed Martin• L3 Communications• Northrop Grumman• OAO Corporation• PRC - Litton• PRC/TASC• QSM• Quality Plus Tech.• Raytheon - Hughes• Robbins, GIOIA• Rockwell Collins• SAIC

• Sallie Mae• Sentel• Software Metrics • SPC• Sverdrup Tech.• Tecolote Research• TeraQuest Metrics• Tivoli• TRW• United Defense • Univ. Southern CA• User Trust Network• US West• VPI - State University• West Virginia High Tech.

Cosortium• West Virginia University

Industry• ACS GSG• American Sys. Corp.• Averstar• Bloodworth Int. Tech.• Carnegie Mellon Univ.• CSC• CTA• Distributive Software• Federal Reserve Bank• General Dynamics• General Scientific Corp.• Graeme & Garland• GTE• Hawaiian Electric• IEEE• IFPUG• IITRI• INCOSE


M&A - 20 24 Oct 01


M&A - 21 24 Oct 01

ISO/IEC 15939, Software Measurement Process


M&A - 22 24 Oct 01

ISO/IEC 15939

• PSM Was Base Document for This Standard

• Specifies Required Activities and Tasks of a Measurement Process

• FCD Ballot Passed in May 2001 With Only 1 Dissenting Vote - All Comments Resolved at ISO Plenary

• Terminology Was Adjusted for Consistency With ISO/IEC 9126, 15498, and 14143 Series

• DIS Ballot Will be Released in September 2001 - Ballot Will Close in November 2001 - International Standard in Late 2001


M&A - 23 24 Oct 01

Basic Measurement “Architecture”

ISO/IEC15939

MeasurementProcessModel

MeasurementInformation

Model

Information DrivenMeasurementRequirements

Consistent, TailorableImplementation

Process


M&A - 24 24 Oct 01

Attribute

MeasurementConstruct

Entities

InformationNeeds Information

Product

MeasurableConcept

Measurement Information Model


M&A - 25 24 Oct 01

Quantity Defined as a Function of Two or More Measures

Attribute

Measure that Provides an Estimate or Evaluation With Respect to Information Needs

Operations Mapping an Attribute to a Scale

Meas.Method

BaseMeasure

Attribute

Meas.Method

BaseMeasure

DerivedMeasure

Meas.Function

Indicator

DerivedMeasure

Algorithm Combining Two or More Measures

Measure of a Single Attribute Defined By a Specified Measurement Method

Property Relevant to Defined Information Needs

Algorithm Combining Measures and Decision Criteria

AnalysisModel

Entities

Information Needs


M&A - 26 24 Oct 01

InformationNeeds

InformationProduct

DecisionCriteria

Measure that Provides an Estimate or Evaluation With Respect to Information Needs

BaseMeasure

Indicator

DerivedMeasure

Algorithm Combining Measures and Decision CriteriaAnalysis

Model

Information Product


M&A - 27 24 Oct 01

CMMI - Measurement and Analysis (M&A)


M&A - 28 24 Oct 01

CMMI M&A

• ISO/IEC 15939 Was Base Document for the Measurement and Analysis Process Area

• CMMI M&A Describes How to Evaluate a Measurement Process

• M&A Is a Level 2 Process Area

• Pilot Assessments Are Being Conducted in 2001


M&A - 29 24 Oct 01

ISO 9000-3Quality management and quality assurance standards - Part 3: Guidelines for the application of ISO 9001 to the development, supply, installation, and maintenance of computer software


M&A - 30 24 Oct 01

ISO 9000-3

• 9000-3 Is Being Updated by SC7 as Part of the ISO 9000:2000 Update

• Part 8, “Measurement, Analysis and Improvement”, Was Updated for Consistency With 15939

• Standard Is at Working Draft Stage - Expected Publication Date Is 2002/2003


M&A - 31 24 Oct 01

For More Information

PSM Support Center

Cheryl Jones - PSM Project ManagerTACOM - ARDECAMSTA-AR-QATBuilding 62Picatinny Arsenal, NJ 07806-5000

(973) 724-2644 (Voice)(973) 724-2382 (FAX)

[email protected] Home Page: http://www.psmsc.com

measurement and analysis m&a - 1 24 oct 01 practical software and systems measurement a...

Documents