Appraisal Assistant Appraisal Assistant WorkshopWorkshop

Evidence collection (CMMI Evidence collection (CMMI Appraisals)Appraisals)

OverviewOverview• Terms and definitionsTerms and definitions• Evidence related screensEvidence related screens• Getting evidences into the databaseGetting evidences into the database

– Importing a list of documentsImporting a list of documents– Using spreadsheet to collect initial PIIsUsing spreadsheet to collect initial PIIs– Using evidence registry to collect PIIsUsing evidence registry to collect PIIs

• Characterize PIIsCharacterize PIIs• Using cross-instantiation evidence panel to Using cross-instantiation evidence panel to

collect interview notescollect interview notes• Reviewing evidence coverage on model Reviewing evidence coverage on model

elementselements• Transfer evidence data between appraisal Transfer evidence data between appraisal

projectsprojectsWorkshop exercise setup: Workshop exercise setup: project : CMMI cont, with RD/PI on target CL5, with instantiation: project : CMMI cont, with RD/PI on target CL5, with instantiation: Org.Process Org.Process and and PSP wizardPSP wizard

Using Evidence Registry Using Evidence Registry – Terms and definitions– Terms and definitions

• Single Instantiation EvidenceSingle Instantiation Evidence: Evidences that : Evidences that are owned by ONE instantationare owned by ONE instantation– Project plan, design spec, test cases etcProject plan, design spec, test cases etc– Can be entered through Can be entered through Single Instance spreadsheetSingle Instance spreadsheet, or , or

Evidence registryEvidence registry• Multi-Instantiation EvidencesMulti-Instantiation Evidences:: Evidences that Evidences that

are related to more than one instantiationsare related to more than one instantiations– Appraisal interviews, organization wide standards, policiesAppraisal interviews, organization wide standards, policies– Sometimes referred as Sometimes referred as Cross-instantiation evidencesCross-instantiation evidences in in

user manualuser manual– Can be entered through Can be entered through Multi-Instance spreadsheetMulti-Instance spreadsheet, or , or

Evidence registry, Evidence registry, or or cross-instantiation evidence cross-instantiation evidence panelpanel

• Evidence and PII:Evidence and PII:– Evidence:Evidence: the individual record of observations, the individual record of observations,

documents from the organization unitdocuments from the organization unit– PII:PII: A piece of evidence is mapped to a model practice to A piece of evidence is mapped to a model practice to

indicate the implementation of that model practice.indicate the implementation of that model practice.

Evidence Related Screens Evidence Related Screens – evidence registry panel– evidence registry panel

• Evidence listEvidence list• Evidence detail Evidence detail

InfoInfo• Relate evidence to Relate evidence to

model practicemodel practice• Comment and Comment and

characterized how characterized how evidence is evidence is mapped to the mapped to the practicepractice

• Evidence coverage Evidence coverage based on the 50% based on the 50% affirmation ruleaffirmation rule

Evidence Screen – Evidence Screen – cross-instantiation evidence cross-instantiation evidence

panelpanel

• Evidence ListEvidence List• Evidence notes can be related to multiple Evidence notes can be related to multiple

instantiationsinstantiations

Evidence Related Screen Evidence Related Screen – practice characterization – practice characterization

tabtab• Evidence grouped Evidence grouped

by Instantiationby Instantiation• Can not Can not

add/remove add/remove evidence in the tabevidence in the tab

• Can modify Can modify indicator type/PII indicator type/PII characterization characterization and PII commentsand PII comments

Importing a list of Importing a list of documents - 1documents - 1

• Documents are Documents are NOTNOT mapped to any practices mapped to any practices (unlike the initial PII spreadsheets, discuss (unlike the initial PII spreadsheets, discuss later)later)

• Use the included Use the included DocumentList.XLSDocumentList.XLS to ensure to ensure correct file formatcorrect file format

• Can be extracted from the OU’s document Can be extracted from the OU’s document databasedatabase

• Instantiation name must be the same as defined Instantiation name must be the same as defined in appraisal scopein appraisal scope– The row will be ignored if the Appraisal Assistant The row will be ignored if the Appraisal Assistant

can not find a matched instantiation in the appraisal can not find a matched instantiation in the appraisal scopescope

• Save the Save the Evidence listEvidence list tab as tab as TAB delimited TAB delimited TextText before importing before importing

Importing a list of Importing a list of documents - 2documents - 2

• 3 documents of 2 instantiations in 3 documents of 2 instantiations in this examplethis example

• Not mapped to any practiceNot mapped to any practice• Check error log for rows that are Check error log for rows that are

ignored after importingignored after importing

List of Document Evidences

Instantiation Evidence Name Document Link Source/Affirmation Description

org. processes evidence bah! www.gu.edu.au moo moo's siteorg. processes Evd 1 must import

Test project 1 EVD 2 import once

Importing a list of Importing a list of documents - 3documents - 3

Result of importing the previous document list

Using Spreadsheet to Using Spreadsheet to Collect Initial PIIs - 1Collect Initial PIIs - 1

• Appraisal Assistant generates one spreadsheet for Appraisal Assistant generates one spreadsheet for each instantiation in the scope and an additional each instantiation in the scope and an additional spreadsheet for multi-instantiation evidencesspreadsheet for multi-instantiation evidences

• Each evidence can be mapped to multiple practicesEach evidence can be mapped to multiple practices• Each evidence can be mapped to the same practice Each evidence can be mapped to the same practice

once and only onceonce and only once• One row per PII, insert rows for more PIIOne row per PII, insert rows for more PII• Leave the row Leave the row EMPTYEMPTY if there is if there is NONO PII for that PII for that

practicepractice• User ensures data accuracy and consistencyUser ensures data accuracy and consistency• XML spreadsheets can be opened by Excel 2002 or XML spreadsheets can be opened by Excel 2002 or

newernewer• Save as ‘Save as ‘Tab Delimited TextTab Delimited Text’ file before being ’ file before being

importedimported• Program merges evidences only if they have the Program merges evidences only if they have the

same name same name ANDAND the same document link strings the same document link strings

Using Spreadsheet to Using Spreadsheet to Collect Initial PIIs - 2Collect Initial PIIs - 2

• 3 evidences here, associated to 3 practices3 evidences here, associated to 3 practices• Single instantiation spreadsheet, instantiation name in the top most cellSingle instantiation spreadsheet, instantiation name in the top most cell• Detail description of process areas and practices in cell commentsDetail description of process areas and practices in cell comments

Using Spreadsheet to Using Spreadsheet to Collect Initial PIIs - 3Collect Initial PIIs - 3

• 3 evidences 3 evidences imported imported from the from the previous previous spreadsheetspreadsheet

• All 3 All 3 evidences evidences belong to belong to the same the same instantiatioinstantiationn

Using Spreadsheet to Using Spreadsheet to Collect Initial PIIs - 4Collect Initial PIIs - 4

• 3 multi-instantiation evidences here3 multi-instantiation evidences here• MI evd 1MI evd 1 is mapped to: PI/SP1.1 - is mapped to: PI/SP1.1 - Org.Processes/Test project 1,Org.Processes/Test project 1,

PI/SP1.2 - PI/SP1.2 - Org.ProcessesOrg.Processes//Project 2Project 2• MI evd 2MI evd 2 is mapped to: PI/SP1.1 - is mapped to: PI/SP1.1 - Project 2Project 2 , PI/SP1.3 - , PI/SP1.3 - Test Test

project 1project 1• MI evd 3MI evd 3 is mapped to: PI/SP1.1 - is mapped to: PI/SP1.1 - Test project 1Test project 1//Project 2Project 2

Using Spreadsheet to Using Spreadsheet to Collect Initial PIIs -5Collect Initial PIIs -5

• 3 evidences 3 evidences imported imported from the from the previous previous spreadsheetspreadsheet

• All imported All imported evidences evidences are labelled are labelled as as Multi-Multi-InstInst

ExerciseExercise

• Import 2 evidences of Import 2 evidences of Org.ProcessesOrg.Processes from from a spreadsheeta spreadsheet– Evidence 1: Evidence 1:

• NameName: : Project plan Project plan • Document LinkDocument Link: : www.sqi.gu.edu.auwww.sqi.gu.edu.au• Associated toAssociated to: : RD-GP 2.1, RD-GP2.2 as RD-GP 2.1, RD-GP2.2 as

Direct artifactDirect artifact

– Evidence 2:Evidence 2:• NameName: : Test planTest plan• Document LinkDocument Link: : www.cit.gu.edu.auwww.cit.gu.edu.au• Associated toAssociated to: : RD-GP 2.2, RD-GP 2.4 as RD-GP 2.2, RD-GP 2.4 as

Indirect artifactIndirect artifact

Using Evidence Registry - Using Evidence Registry - OverviewOverview

Using Evidence Registry Using Evidence Registry – single instantiation – single instantiation

evidenceevidence1.1. Click Click AddAdd Button Button

(Ctrl-A) to add a (Ctrl-A) to add a new recordnew record

2.2. Enter the Name Enter the Name of the evidenceof the evidence

3.3. Specify the Specify the instantiationinstantiation

4.4. Associate it to Associate it to the practice(s)the practice(s)

5.5. Specify the type Specify the type of the PIIof the PII

6.6. Comment on Comment on why/how the why/how the evidence is evidence is related to the related to the practicepractice 11

22 33 44 55

66

ExerciseExercise

• Add 1 evidence to the registryAdd 1 evidence to the registry– NameName: Design Review Record: Design Review Record– InstantiationInstantiation: Org.Processes: Org.Processes– Associated toAssociated to::

• RD – GP2.1 as Direct Artifact, comment: RD – GP2.1 as Direct Artifact, comment: bah!bah!

• RD – GP2.2 as Indirect Artifact, comment: RD – GP2.2 as Indirect Artifact, comment: doh!doh!

Using Evidence Registry Using Evidence Registry – multi-instantiation – multi-instantiation

evidencesevidences

11

1. Click Add button to add a new a new record

2. Name the new evidence record

3. Select Multi-Inst from the instantiation column

4. Locate the practice and the practice-instantiation, Tick the check box to map the evidence

5. Specify the Indicator Type

6. Comments on how/why the evidence is considered a PII

22 33 44 55

66

Using Evidence Registry – Using Evidence Registry – Additional evidence Additional evidence

informationinformation• Only Only evidence evidence

namename and and instantiationinstantiation are are mandatorymandatory

• Evidence FileEvidence File: : Link to a file or Link to a file or web pageweb page

• LabelLabel: short : short string defined by string defined by appraisal team to appraisal team to identify evidence identify evidence

• Source/Source/AffirmationAffirmation: the : the owner of the owner of the evidenceevidence

• Last Modified Last Modified by /Last Modified by /Last Modified onon: auto fields: auto fields

Using Evidence Registry - Using Evidence Registry - Characterize P.I.I.Characterize P.I.I.

• Click on the Click on the ‘Characteristic’ ‘Characteristic’ column of the column of the evidence under evidence under the practicethe practice

• Select the Select the characteristic characteristic from the from the dropdown listdropdown list

• Though non-Though non-mandatory, mandatory, useful for useful for characterizing characterizing practice practice implementationimplementation

Using Cross-Instantiation Using Cross-Instantiation Evidence PanelEvidence Panel

• Evidences covers more than one Evidences covers more than one instantiationinstantiation

• Can be launched from evidence Can be launched from evidence registry and from the main U.I.registry and from the main U.I.

• Only consists of process areas that Only consists of process areas that are within the appraisal scopeare within the appraisal scope

• Process areas are grouped by their Process areas are grouped by their categorycategory

Using Cross-Instantiation Using Cross-Instantiation Evidence PanelEvidence Panel

• Data will be automatic passed into the Data will be automatic passed into the evidence registryevidence registry

• If the PII note on an instantiation of a practice If the PII note on an instantiation of a practice is cleared, the evidence is disassociated from is cleared, the evidence is disassociated from the practice on that instantiationthe practice on that instantiation

• If an evidence is deleted, ALL related If an evidence is deleted, ALL related interview notes will also be deletedinterview notes will also be deleted

• You can also associate/disassociated multi-You can also associate/disassociated multi-instance evidence to a practice through the instance evidence to a practice through the evidence registry (discussed above)evidence registry (discussed above)

Using Cross-Instantiation Using Cross-Instantiation Evidence PanelEvidence Panel

1.1. Click Click AddAdd button button to add a new to add a new recordrecord

2.2. NameName the the evidenceevidence

3.3. Find the Find the interested interested practice from practice from the tabs and the tabs and clickclick on its on its instantiationinstantiation columncolumn

4.4. Comment Comment on on how evidence is how evidence is related to the related to the instantiation of instantiation of the practicethe practice

5.5. Specify Specify Indicator TypeIndicator Type

11

22 33

4455

ExerciseExercise

• Enter notes for “Interview Enter notes for “Interview 22””– PI SP1.2-2 :PI SP1.2-2 :

• Instantiation: Org.ProcessesInstantiation: Org.Processes• Characteristic: StrengthCharacteristic: Strength• Notes: “Testing interview 2 comment on PI SP1.2-2, Notes: “Testing interview 2 comment on PI SP1.2-2,

we thing it is a strength”we thing it is a strength”

– PI SP 1.3-3:PI SP 1.3-3:• Instantiation: PSP wizardInstantiation: PSP wizard• Characteristic: StrengthCharacteristic: Strength• Notes: “Testing interview 2 comment on PI SP 1.3-3, Notes: “Testing interview 2 comment on PI SP 1.3-3,

it is a strength”it is a strength”

Using Evidence Registry – Using Evidence Registry – Review evidence coverageReview evidence coverage

• Automatic calculate affirmation coverage based on the 50% ruleAutomatic calculate affirmation coverage based on the 50% rule• Example indicators /work products are shown in the edit boxes Example indicators /work products are shown in the edit boxes

belowbelow• BLUEBLUE triangle: At least ONE direct artifact from that practice- triangle: At least ONE direct artifact from that practice-

instantiation insertion, instantiation insertion, REDRED triangle: No direct artifact from that triangle: No direct artifact from that practice-instantiation insertion.practice-instantiation insertion.

Using Evidence Registry – Using Evidence Registry – Shortcut keysShortcut keys

• Ctrl – ACtrl – A : Add new evidence record : Add new evidence record• F1F1: Online help (: Online help (under constructionunder construction))• F2 – F5F2 – F5 : Expand/Collapse the model hierarchy : Expand/Collapse the model hierarchy

tree nodes (tree nodes (if the tree has the input focusif the tree has the input focus))• F7F7 : Locate all evidence-practice associations : Locate all evidence-practice associations

that are related to the selected evidencethat are related to the selected evidence• Arrow keys/TabArrow keys/Tab : Navigate the model : Navigate the model

hierarchy tree (hierarchy tree (if the tree has the input focusif the tree has the input focus))• Right click anywhere to bring up the context Right click anywhere to bring up the context

menumenu

Using Evidence Registry Using Evidence Registry - Icons- Icons

Not enough affirmation on a practice/goalNot enough affirmation on a practice/goal

Enough affirmation on a practice/goalEnough affirmation on a practice/goal

Has no direct artifact on a practiceHas no direct artifact on a practice

Has at least one direct artifact on a practiceHas at least one direct artifact on a practice

Process areaProcess area

Direct artifactDirect artifact

Indirect artifactIndirect artifact

Direct affirmationDirect affirmation

Unknown indicator typeUnknown indicator type

Evidence is not reviewedEvidence is not reviewed

Evidence is reviewedEvidence is reviewed

Transfer Evidence Between Transfer Evidence Between Appraisal ProjectsAppraisal Projects

• Evidences that are out of destination scope Evidences that are out of destination scope are NOT importedare NOT imported

• Evidences that are not mapped to the Evidences that are not mapped to the practices are NOT exportedpractices are NOT exported

• Source appraisal model representation can Source appraisal model representation can be different from the destination be different from the destination representationrepresentation

• Import routine does NOT check for Import routine does NOT check for duplication with the existing evidencesduplication with the existing evidences– Import the database once and ONLY once.Import the database once and ONLY once.– Import the database before generating any mini Import the database before generating any mini

teamsteams

Workshop revision Workshop revision historyhistory

VeVerr

AuthorAuthor DateDate CommentComment

0.10.1 Fred LiangFred Liang 09/Apr/0509/Apr/05 Initial versionInitial version

0.20.2 Fred LiangFred Liang 16/July/0516/July/05 Update evidence registry screen, evidence coverage, Update evidence registry screen, evidence coverage, collect evidence by model elementcollect evidence by model element

0.30.3 Fred LiangFred Liang 7/Dec/057/Dec/05 Add import document list, restructured slidesAdd import document list, restructured slides

0.40.4 Fred LiangFred Liang 19/Mar/0719/Mar/07 Update for BETA 3Update for BETA 3