a family of scampi appraisal methods · pdf filedriven by appraisal scope, organizational...

Pittsburgh, PA 15213-3890

CMMISM

Sponsored by the U.S. Department of Defense© 2003 by Carnegie Mellon University

SM CMMI, CMM Integration, and SCAMPI are service marks of Carnegie Mellon University® Capability Maturity Model and CMM are registered with the U.S. Patent and Trademark Office

Pittsburgh, PA 15213-3890

CMMI

This material is approved for public release. Distribution is limited by theSoftware Engineering Institute (SEISM) to attendees.

Sponsored by the U.S. Department of Defense© 2003 by Carnegie Mellon University

SM CMM Integration, SCAMPI, SCAMPI Lead Assessor, SCAMPI Lead Appraiser, and SEI are service marksof Carnegie Mellon University. CMM and CMMI are registered in the U.S. Patent and Trademark Office by Carnegie Mellon University.

A Family of SCAMPI AppraisalMethods

CMMI

© 2003 by Carnegie Mellon University Module N SLAT V1.1 page 2

ObjectivesIntroduce the three classes ofSCAMPI appraisal methods,different appraisal needs andtypes, and the SCAMPI B & CProject.

CMMI


Three Classes of Appraisals

LargeMediumSmallTeam Size

HighMediumLowResource needs

YesNoNoRatings generated

HighMediumLowAmount of objective evidence

Class AClass BClass CCharacteristic

CMMI


Appraisal Needs

Getting Started, Understanding the CMMI

Familiarization or Expectation Setting for Future Appraisal

Broad-based Information on Implementation of Practices

In-depth Analysis of Implemented Practices

Gap Analysis Focused on Findings of Previous Appraisal

Incremental Data to Track Improvement Progress

Benchmarking Leading to a Public Statement

CMMI


Appraisal Types

Getting-Started Intervention

*Mini Appraisal

*Gap Analysis

*Improvement Monitoring

Delta Appraisal

Incremental Appraisal

Benchmarking Appraisal

CMMI


Getting-Started InterventionPurpose: Provide information toan organization learning aboutCMMI.

Description: Expert-directed,presentation & discussion-intensive, broad scope, shallowcoverage method thatemphasizes participation ofaffected groups in facilitatedsessions - with no focus onsufficiency of coverage.

Rough Effort Estimate: One totwo experts on site for 1 to 5days. Planning and coordinationeffort driven by appraisal scope.

CMMI


Mini AppraisalPurpose: An abbreviated versionof a benchmarking appraisalused for experience and non-rating related outputs.Description: Team-based, dataintensive, narrow scope, deepcoverage method thatemphasizes the use of ObjectiveEvidence and interviews - withreduced focus on datasufficiency.Rough Effort Estimate: Oneexpert on site for 3 to 5 days.Planning and coordination effortdriven by appraisal scope, andexperience level of local teammembers.

CMMI


Gap AnalysisPurpose: Identify specificdeficiencies in implementedpractices relative to practiceswhich would supportachievement of the goals - for aspecified scope.

Description: Expert-based,interview-intensive, broad scope,shallow coverage method thatemphasizes identification ofweaknesses - with more limitedfocus on sufficiency of coverage.

Rough Effort Estimate: One totwo experts on site for 1 to 3days. Planning and coordinationeffort driven by appraisal scope.

CMMI


Improvement MonitoringPurpose: Track implementationof process improvement actionsin an organizational unit that haspreviously undergone abenchmarking appraisal.Description: Expert-based,interview & document-intensive,narrow scope, deep coveragemethod that emphasizesidentifying status of changes inprocesses - with more limitedfocus on sufficiency of coverage.

Rough Effort Estimate: One totwo experts on site for 1 to 3days. Planning and coordinationeffort driven by scope andoutcomes of previousappraisal(s).

CMMI


Delta AppraisalPurpose: Confirm correctiveactions resulting from abenchmarking appraisal.

Description: Expert-based,interview & document-intensive,very narrow scope, deepcoverage method thatemphasizes confirming specificchanges in processes - withlimited focus on sufficiency ofcoverage for the model.

Rough Effort Estimate: One totwo experts on site for 1 or 2days. Planning and coordinationeffort driven by the nature andmagnitude of weaknessesidentified in the benchmarkingappraisal.

CMMI


Incremental AppraisalPurpose: Building a profile ofresults across a broad modeland/or organizational scope.

Description: Expert-directed orteam-based, data intensive,narrow scope, deep coveragemethod that emphasizesuse of objective evidence - withstrong focus on sufficiency ofcoverage (in narrowincrements).

Rough Effort Estimate:Appraisal events distributedover time. One expert on site for3 to 8 days. Planning andcoordination effort driven byappraisal scope, and experiencelevel of team members.

CMMI


Benchmarking AppraisalPurpose: Document a frequentlysought, high level, benchmarkfor an organizational unit.Description: Expert-directed andteam-based, data intensive,broad scope, deep coveragemethod that emphasizes use ofobjective evidence - with strongfocus on sufficiency of coverage.

Rough Effort Estimate: Oneexpert on site for 5 to 15 days.Planning and coordination effortdriven by appraisal scope,organizational maturity, andexperience level of teammembers.

CMMI


Sources of Objective EvidenceInstruments

• Organizational assets reflecting evidence ofimplementation of model practices (e.g., mappingtables)

• QuestionnairesInterviews

• Standard structured interviews; on-call interviews;follow-up interviews

• Exploratory or focused questions targeted at manager,practitioners and/or users

Presentations• Briefings, demonstrations

Documents• Hardcopy, softcopy, hyperlinks

CMMI


Appraisal Scope

Defining the“Organizational Unit”

Selecting CMMI Model,Representation, and Scope

CMMI-SE/SW

2345 PA1

PA2PA3PA4PA5PA6

0 1 2 3

CMMI


Broad Scope, Shallow CoverageTake advantage of low cost data collection methods• achieve broader coverage of the organization

- include all projects, not just 4 for benchmarking• address a larger portion of the model

- match information sources to information needs

Limitations of low cost data collection methods• very limited opportunity to explain nuances

- yes/no questions in interviews don’t pay off• difficult to assure validity of data

- ambiguous questionnaire items don’t help

CMMI


Narrow Scope, Deep CoverageMaximize insight with a small sample of the organization• detailed insight about a selected unit in the organization

- potentially include every staff member• full coverage of limited model material

- minimal risk of undiscovered weaknesses

Limitations of using a small sample• limited ability to make generalizations

- process of one stellar project may be misleading• limited coverage of the lifecycle in use

- implementation of future phases may differ fromintended practices

CMMI


Required Outputs: SCAMPI B&CCompleted AppraisalDisclosure Statement (ADS)

Identification of data collectiontechniques used

Identification of data collectionsessions conducted

Appraisal plan, annotated withactual data for• time, effort and cost• model coverage• organizational coverage

Official

J. Hancock

CMMI


Expected Outputs: SCAMPI B&CStatements of strengthsand/or weaknesses relativeto model practices, goals orother components.

Characterizations ofdifferences in practiceimplementation• across model content

covered in the appraisal• across organizational

units included in theappraisal

Your Results

0

50

100

• Findings statements• Strengths/Weaknesses• Recommendations• ...

CMMI


Optional Outputs: SCAMPI B&CDetailed data summariesrelated to a pre-existing:• process improvement target• performance contract• remedial improvement action• set of risk areas in a contract

Orientation and/or training for:• improvement champions• technical and managerial staff• sponsors new to CMMI

A ‘watershed’ event for a groupof co-workers.

CMMI


Appraisal Reporting TechniquesFormal presentations using“slides” - words and graphics.

Facilitated, interactive,presentations with specificparticipants.

Group or individual meetingsinvolving appraisers andparticipants.

Written material, ranging from‘bulleted’ summaries inpresentation slide format to afull detailed report.

CMMI


Appraisal Use/Purpose: Examples

Formulating OurWinning Strategy

Fixing ImmediateProblems

Exploring Goals forImprovement &Training Staff

CMMI


Appraisal Outcomes: Examples -1

Making Sure OurHouse is in Order

Charting ProgressAgainst Our Plan

CMMI


Appraisal Outcomes: Examples -2

Find NewChallengesto Conquer

Recognize Progressand Celebrate Success

CMMI


The SCAMPI B&C ProjectSEI working collaboratively with appraisal stakeholders:• prototyping and field trials with collaborator organizations• communication with stakeholders about the project• periodic dissemination of draft material for review• publication/presentation of project results

CMMI


SCAMPI B&C Project ObjectivesDevelop, Field Test, and Deploy an integrated suite ofappraisal methods that are:• Developed iteratively with extensive field tests• Based on requirements from users• Integrated and upwardly compatible• Well specified and documented

CMMI


Project Goals1. Jointly Develop proven B&C appraisal methods2. Transition B&C competencies to the participating

organizations3. Advance the (SCAMPI) authorization of current and

future lead appraisers4. Deliver value to the participating organizations’

improvement efforts through the delivery of pilotappraisals

5. Jointly produce: technical reports, papers,presentations

CMMI


Appraisal Usage ModesInternal process improvement

Acquisition

• Source selection

• Contract/Process monitoring

CMMI


Options for the Pilot Appraisals• Appraisal types and timing sequence• Definition of the OU scope (what parts of the organization

are included in the appraisal)• Definition of the reference model scope• Findings

- At the OU level- Discipline specific- Project specific

• These options drove the derivation of the pilot use casesand the pilot enactments.

CMMI


Estimated ResourcesTeam Members• Class B (2-9) 3-8 days on-site; 1-2 days training• Class C (1-6) 1-5 days on-site; 0-1 days training

Appraisal participants• 1-5 hours/participant, including (as applicable)

- Attending the opening meeting- Completing a questionnaire or survey- Participating in an interview session- Participating in a validation session- Attending the findings presentation

CMMI


SEI’s RoleSteward of the family of SCAMPI appraisal methodsLead developer of SCAMPI B and C methodsDefiner of pilot data to be collected and data analyses tobe performedLeader/Coach of pilot appraisals

CMMI


ConclusionNot all appraisal needsmake sense using theSCAMPI A method.

The SEI is developingSCAMPI B & C methods.