csc proprietary 9/14/2015 4:15:28 pm 008_fmt_wht 1 german workshop on experience management 2003...

CSC Proprietary 04/21/23 03:15 AM 008_fmt_wht 1

German Workshop on Experience Management 2003

Experience Management within Project Management Processes

Maya Kaner and Reuven Karni

Industrial Engineering and ManagementTechnion – Israel Institute of Technology



Project management is a knowledge intensive activity. Managers use their skills and experience to make decisions during the execution of a development process. Better results in terms of meeting schedule budget and functionality are usually achieved due to the manager’s accumulated experiences and knowledge.

(Barros, Werner and Travassos)


Aims of the research

• To propose an process oriented architecture for project management experience management

• To define metrics for interpreting knowledge and transferring experience along the process

• To study the effectiveness of a knowledge-based architecture in terms of these metrics


Definitions

• Project management

• Project management is the application of knowledge, skills, tools and techniques to project activities in order to meet or exceed stakeholder expectations (PMBOK)

• Project management processes

• Those processes within a project which are concerned with planning, organizing, managing and controlling a project in the areas of integration, scope, time, cost, quality, human resources, communications, risk and procurement (PMBOK)


The Project Management Body of Knowledge

• The PMBOK provides a knowledge perspective of project management

• It divides project knowledge into an hierarchy of project knowledge areas and project management processes

• It details nine knowledge areas and 39 project management processes

•This stresses the wide range of knowledge and experience required by the project manager


Definitions

• Experience management

• A special form of knowledge management which deals with task-based and experiential knowledge that is both gained and applied when carrying out business-related tasks (Minor)

• Experience-based knowledge

• Knowledge is not so much a capacity for specific action, but the capacity to use information, learning and experience resulting in an ability to interpret information and ascertain what information is necessary in decisionmaking (Watson)


Definitions

• Knowledge focus

• The compilation and correlation of knowledge relevant to the execution of a knowledge-intensive activity

• Metric

• A measure of a particular characteristic of performance or efficiency

• In our case this is the ability to interpret knowledge and transfer experience


Scope

• Business-related tasks

• Our specific tasks are those concerned with knowledge application in decisionmaking within project management

• Project-related experience

• The specific experience investigated is that obtained and employed by project managers when making decisions (our “target”)


Research theory

• Based on a schema proposed by March and Smith

“Design and natural science research in information technology”

• Research deliverables or outputs are

• Constructs – concepts or vocabulary of the domain

• Model – propositions expressing relationships amongst the constructs

• Method – a set of steps to perform a task


Research theory

•Research activities include

• Build

• Evaluate

“We build an artifact to perform a specific task. The basic question is: does it work?

We evaluate artifacts to determine if we have made any progress. The basic question is: how well does it work?

Evaluation requires the development of metrics and the measurement of artifacts according to those metrics”


Research theory

•Research metrics include

• Constructs – completeness, simplicity, understandability, ease of use

• Model – completeness, level of detail, consistency

• Method – operability, efficiency, generality, ease of use


Our research outputs – constructs

Knowledge area level (K)

Process level (P)

Issue level (I)

Case level (C)

Entity level (E)

Value level (V)


Our research outputs – constructs

Knowledge area level (K)

Process level (P)

Issue level (I)

Case level (C)

Entity level (E)

Value level (V)

Project management body of knowledge (PMBOK)

Case base


Our research outputs – model (I)(processes, issues and cases)

Humanresources

Riskmgt.

Timemgt.

Staff acquisition

Riskresponse

Activitydefinition

Select Candidate

Corrective action

Duration estimating

SystemExperience

ProblemResponse

Duration Cost

Budget Beginner

Experience Delay

1.5 – 3 wks 30 – 40K

K P I/C E V

C

R

E


Our research outputs – model (II)(dyads and paths)

C 16

R 14

E 18

E 18

(+,+,+,+,-,-)

(+,+,+,+,-,-)

(+,+,+,+,+,+)

(+,+,+,+,+,+)

+ value recorded

- no value


Our research outputs – method(creation of a knowledge focus)

• Five case-based and path-based activities

• Define case retrieval input from a decision scenario using “case-by-example”

• Retrieve all corresponding cases

• Select relevant cases to retrieve paths

• Retrieve all path dyads and interpret them

• Synthesize cases and path dyads to create a knowledge focus related to the decision scenario

• Carry out any feedback required to retrieve other cases or paths


Our research – analysis of a decision scenario(creation of a knowledge focus)

• Scenario (text)

"The manager of an information systems

implementation project is required to select the best

candidate for analyzing a human resource

management system. He can choose a beginner,

someone with experience, or an expert. The decision

should be based on knowledge from past projects,

concentrating on the trade-off between inexperience

risks versus salary costs"


Knowledge and experience metrics

• Mappability

• Definition – the ability to map project processes (PMBOK) into issues

• Experience – associating project process outputs with the decisions (usually selection) needed to create them

• Measure – percentage of project process outputs (PMBOK) mapped into issues



• Compliance

• Definition – the ability to map a textual decisionmaking situation into cases-by-example expressed by the issue-case-entity lexicon

• Experience – associating textual formats with ontological formats

• Measure – number of cases-by-example created from the decisionmaking situation description



• Accordance

• Definition – the ability to map a graphical representation of a path into a textual interpretation of the relationships expressed by the path

• Experience – associating nodes and arcs with specific successive actions

• Measure – graph traverse mapped into causality



• Convergence

• Definition – the ability to map a set of retrieved cases and paths onto a knowledge focus

• Experience – synthesizing knowledge from a set of retrieved sources

• Measure – cases and paths mapped into knowledge focus



• Cover

• Definition – the knowledge resources compiled to create a knowledge focus

• Experience – the ability to retrieve and compile knowledge (completeness)

• Measure – non-redundant cases and paths compiled towards a knowledge focus



• Efficiency

• Definition – the time required to create a knowledge focus

• Experience – the ability to retrieve and compile knowledge (speed)

• Measure – the time required to create a knowledge focus


Measures (scoring)

• We score performance by counting the number of errors in the “knowledge items” required to successfully carry out each step in the process.

• In this way we can learn from the experiences of the subjects in order to reduce errors and thus improve the methodology


Measures – Mappability(issues connected with process outputs)

• Errors: outputs not mapped into issues

• (out of scope of this presentation)


Measures – Compliance(decomposition of a text into issues)

• Errors: faulty search for relevant cases

• Redundant search – CBE values not directed unneeded knowledge

• Incomplete search – CBE values not relevant unneeded knowledge

• Constrained search – too many CBE values missing knowledge

• Null search – case/issue not identified missing

knowledge


Measures – Accordance(interpretation of a path graph)

• Errors: arcs or nodes overlooked or misinterpreted

• Null – arc overlooked missing causality

• Null – case overlooked missing knowledge

• Incomplete – arc misinterpreted missing causality

• Incomplete – case misinterpreted missing knowledge


Measures – Convergence(creation of a knowledge focus)

• Errors: cases or paths overlooked or misinterpreted

• Null – focus component overlooked missing focus

• Incomplete – focus component misinterpreted misleading focus


Measures – Cover(cases/paths required for knowledge focus)

• Errors: excessive cases or paths

• Cases – too many cases retrieved blurred knowledge focus

• Paths – too many paths retrieved blurred knowledge focus


Measures – Efficiency(total time required to derive knowledge focus)

• Errors: excessive time required

• Time – extra time required misdirected knowledge focus


Experiment – Compliance(decomposition of a text into issues)

• Scenario (text)


implementation project is required to select the

best candidate for analyzing a human resource


someone with experience, or an expert. The

decision should be based on knowledge from

past projects, concentrating on the trade-off

between inexperience risks versus salary costs"



• Solution (cases-by-example)

• C (candidates)• Specialization = “Analysis” • System = “Human resource management”• Aim = Experience and salary of past HRM

candidates

• E (time and cost estimates)• Specialization = “Analysis” • System = “Human resource management”• Aim = If and when original estimates updated

• R (risks)• Category = “Project team member(s)” • Type = “Experience”• Aim = Risks of team inexperience and responses



• Experiment

• Groups • Control given case lexicon at start of experiment• Treatment given case lexicon before start of experiment

• Results – average errors (p < 0.01)• Control n = 30; average = 3.8; variance = 2.9• Treatment n = 30; average = 2.0; variance = 1.0

• Inference• Treatment group makes fewer errors• The experience of studying the lexicon contributes to better performance



• Experiment

• Errors – misdirected input • Redundant search 46%• Constrained search 24%• Null search 20%• Incomplete search 9%

• Errors – cases involved• Risks 87%• Estimates 9%• Candidates 3%

• Interpretation of main errors• Fear of missing knowledge leads to redundancy• Complexity of risk case causes problem with decomposition


Experiment – Accordance(interpretation of a path graph)

• Scenario (graph)

C 16

R 14

E 18

E 18

(+,+,+,(+,+,+,+,-,-)+,-,-)

(+,+,+,(+,+,+,+,-,-)+,-,-)

(+,+,+,+,+,(+,+,+,+,+,+)+)

(+,+,+,+,+,(+,+,+,+,+,+)+)

++ value recorded

- - no value



• Solution (textual interpretation)

• C 16 (+,+,+,+): A beginner with salary 10-15K was selected for coding the maintenance system E 18 (+,+,+,+,-,-): This led to an estimate of the corresponding activity duration (1.5 – 2 wks) and cost (30 – 40K) R 14 (+,+,+,+,-,-): This also led to anticipating a problem of lack of experience which could be triggered by a change in requirements E 18 (+,+,+,+,+,+): Foreseeing this problem then led to a revision of the estimated activity duration (2 – 3 wks) and cost (50 – 60K) R 14 (+,+,+,+,+,+): This in turn led to an expectation of project delay



• Experiment

• Groups • Control none• Treatment given case details

• Results – average errors• Treatment n = 30; average = 1.9; variance = 1.6

• Inference• The graph represents “experience” in linking decisions; the subjects were require to reproduce this experience



• Experiment

• Errors – incorrect case interpretation• Risks 16%• Estimates 9%• Candidates 9%

• Errors – incorrect arc interpretation/overlook• Arc E – R 30%• Other arcs (3) 12%• Candidates 3%

• Interpretation of main errors• Inability to interpret the evolution of the risk case along the path sequence• Inability to interpret the apparent “loop” in the path


Experiment – Convergence(creation of a knowledge focus)

• Scenario (text)


implementation project is required to select the

best candidate for analyzing a human resource


someone with experience, or an expert. The

decision should be based on knowledge from

past projects, concentrating on the trade-off

between inexperience risks versus salary costs"



• Five case-based and path-based activities

• Define case retrieval input from the decision scenario using “case-by-example”




• Synthesize cases and path dyads to create a knowledge focus related to the decision scenario



Experiment – Convergence(control – no guidance given)

• Four CBR and PBR actions (icons)

• Define case retrieval input from the decision scenario using “case-by-example”




• Press any icon to activate the corresponding action


Experiment – Convergence(treatment – process map given)

• Four linked CBR and PBR actions (icons)

1. Define case retrieval input from the decision scenario using “case-by-example”

2. Retrieve all corresponding cases

3. Select relevant cases to retrieve paths

4. Retrieve all path dyads and interpret them




• Experiment

• Groups • Control carry out any of the activities, in any order• Treatment carry out all four activities in a given order

(process)• Results – average errors (p < 0.01)

• Control n = 15; average = - 0.9; variance = 0.5

• Treatment n = 15; average = - 0.5; variance = 0.3• Inference

• The treatment group is better able to synthesize the knowledge retrieved and to relate it to the decision to be made• A systematized process contributes to better performance• Experience – knowledge about the sequence in which to retrieve and interpret knowledge – contributes to better performance


Experiment – Efficiency(total time required to derive knowledge focus)

• Experiment


(process)• Results – average errors (p < 0.05)

• Control n = 15; average = 23 minutes; variance = 37 • Treatment n = 15; average = 19 minutes; variance = 21

• Inference• The treatment group is faster in generating the knowledge focus• A systematized process contributes to faster and more uniform performance• Experience – knowledge about the sequence in which to retrieve and interpret knowledge – contributes to better performance


Experiment – Cover (cases)(cases required for knowledge focus)

• Experiment


(process)• Results – average errors (lower bound = 3) (p < 0.05)

• Control n = 15; average = + 1.47; variance = 2.6 • Treatment n = 15; average = + 0.40; variance = 1.4

• Inference• The treatment group requires less to find a meaningful input • Systematization gvies more effective and uniform performance• Systematization gives more confidence and thus less scope• Experience – knowledge about the sequence in which to retrieve and interpret knowledge – contributes to better performance


Experiment – Cover (paths)(paths required for knowledge focus)

• Experiment


(process)• Results – average errors (lower bound = 3)

• Control n = 15; average = + 3.27; variance = 2.8 • Treatment n = 15; average = + 3.47; variance = 3.3

• Inference• Both groups tend to search the same number of paths• However, the treatment group retreieved these pathns in an organized fashion• Experience – knowledge about the sequence in which to retrieve and interpret knowledge – contributes to better performance


Conclusions

• Business process oriented knowledge management (BPOKM) means that knowledge is structured according to process requirements

• Experience oriented knowledge management means that knowledge is created and updated through the experience of those applying the knowledge

•The project manager looks for suitable candidates (process) and seeks the effects of previous decisions (experience) on time, costs and risks as stored in the knowledge base


Conclusions

• Our aim is to create an effective knowledge focus – the right number, identification and interpretation of the retrieved cases and paths

•At each step of the process, knowledge focus errors may occur

• These have been measured by a uniform metric – a count of the errors incurred

• Errors reflect redundant or missing or misinterpreted knowledge elements (cases or paths) as compared to an “ideal” search for the knowledge required


Conclusions

• In general we find that systematization of the decision process and the link between process and knowledge required leads to better performance

• Our methodology allows us to locate areas and reasons for non-performance, such that the project management decisionmaking process can be improved


To sum up …

• Knowledge bases can be structured by adopting a process description as one specific foundation for the schema of the knowledge base. This facilitates access since in a process oriented application the user knows his exact position in the process and can therefore easily navigate the knowledge base.

• Since the process description functions as a structuring criterion for a knowledge base, all the relevant information and knowledge for a process step – that goes beyond the capability of a conventional process model – can be captured.

(Jablonski, Horn and Schlundt)


To sum up …

We hope that our methodology has

contributed a useful approach, based on

error metrics at each process step, to be

able to measure the effectiveness of this

knowledge capture

CSC Proprietary 04/21/23 03:15 AM 008_fmt_wht 52

German Workshop on Experience Management 2003


Thank you!

csc proprietary 9/14/2015 4:15:28 pm 008_fmt_wht 1 german workshop on experience management 2003...

Documents