mtat.03.243 software engineering management - ut · • homework 2 – solution ... • have a look...
TRANSCRIPT
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
MTAT.03.243
Software Engineering Management
Lecture 11:
SPI & Measurement –
Part B
Dietmar Pfahl
email: [email protected] Spring 2015
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
Structure of Lecture 11
• Homework 2 – Solution Sketch
• Exercise
• GQM (Goal/Question/Metric) Process
• Homework 3
• Standards
– Measurement Process
– Product Quality
– Process Capability/Maturity
• KPI (Key Performance Indicator)
• Balanced Scorecard
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
Homework 2: Software Process Modeling
Background information:
• Recall Exercise 2 from the lecture where you modeled a process using one of the
(equivalent) notations shown in Figure 1 below. Then perform the tasks described on
the next page.
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
Homework 2: Software Process Modeling
Task T1 [3 marks]:
• Have a look at the description of ‘Scrum’ published in the Eclipse Process
Framework (EPF) Wiki at http://epf.eclipse.org/wikis/scrum/index.htm.
• Create a process model using the table notation shown in Figure 1.
• Make a note/comment, if you think, information is missing (incomplete) or
inconsistent in the process description provided by the EPF Wiki.
• Note: Marking will consider completeness and correctness of the model as
well as appropriateness of your notes/comments
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
Homework 2 –
Task 1:
Solution
Sketch
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
Homework 2: Software Process Modeling
Task T2 [2 marks]:
• In the model you created in Task T1, select at least three activities that are
directly connected via product-flow (i.e., one activity consumes at least one
artifact produced by another activity). Then create a graphical representation
of this subset of the Scrum process model. You can choose one of the
following three notations (and tools):
– Notation from Figure 1
– Spearmint (you find information on how to download and install
Spearmint on the course wiki – Lecture 4)
– Bizagi (you find information on how to download and install Bizagi
Process Modeler on the course wiki – Lecture 4)
• Note: Marking will consider completeness and correctness of the model and
consistency with the model developed in Task T1.
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
Homework 2 – Task 2: Solution Sketch
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
Homework 2: Software Process Modeling
Task T3 [1 mark]:
• Explain your choice of the notation/tool in Task T2. What was the reason
why you chose it and what are the advantages and disadvantages of the
notation you chose? To get full marks, you need to define and discuss at
least two criteria that guided your decision. For example, you may use a
table notation as shown below:
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
Homework 2 – Task 3: Solution Sketch
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
Homework 2: Distribution of Marks
Students enrolled: 26 HW1 submitted: 25 Penalty of -1: 1 Penalty of -2: 1 Average marks: 4.8 No name on sheet: >3 Not PDF format: 1 Late submitted: 1 Not submitted: 1 Mean: 4.8 Median: 5.3 Mode: 6
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
Structure of Lecture 11
• Homework 2 – Solution Sketch
• Exercise
• GQM (Goal/Question/Metric) Process
• Homework 3
• Standards
– Measurement Process
– Product Quality
– Process Capability/Maturity
• KPI (Key Performance Indicator)
• Balanced Scorecard
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
Measurable Entities in a SW Process (Model)
An entity can represent any of the following:
• Process/Activity: any activity (or set of
activities) related to software
development and/or maintenance (e.g.,
requirements analysis, design, testing) –
these can be defined at different levels of
granularity
• Product/Artifact: any artifact produced or
changed during software development
and/or maintenance (e.g., source code,
software design documents)
• Resources: people, time, money,
hardware or software needed to perform
the processes
Activity
Productout
Productin
Ressourcetool
Ressourcerole
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
Process Measure – Avg. Sprint Productivity
Name: ’Average Sprint Productivity’ (~ ’Process Efficiency’)
• Name: Avg. Sprint Prod. ???????? ...
• Entity: Software Sprint
• Attribute: Size Effort
• Unit:
• Range: [0, ∞) [0, 1, 2, ..., n] [0, ∞)
[0, ∞)
• [ph] = TM [p] * TD [d] * WHD [h/d]
Productivity = Output / Input
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
Process Measure – Avg. Sprint Productivity
Name: ’Average Sprint Productivity’ (~ ’Process Efficiency’)
• Name: Avg. Sprint Prod. ???????? ???????? ????????
• Entity: are Sprint
• Attribute: Size Effort
• Unit:
• Range: [0, ∞) [0, 1, 2, ..., n] [0, ∞)
[0, ∞)
• [ph] = TM [p] * TD [d] * WHD [h/d]
Productivity = Output / Input
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
Process Measure – Avg. Sprint Productivity
Name: ’Average Sprint Productivity’ (~ ’Process Efficiency’)
• Name: Avg. Sprint Prod. Sprint Count Sprint Output Sprint Input
• Entity: Dev. Process
• Attribute: Efficiency Sprint count Size
• Unit: LOC/person-hour dimensionless person-hour
• Range: [0, ∞) [0, 1, 2, ..., n] [0, ∞)
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
Process Measure – Avg. Sprint Productivity
Name: ’Average Sprint Productivity’ (~ ’Process Efficiency’)
• Name: Avg. Sprint Prod. Sprint Count Sprint Output Sprint Input
• Entity: Dev. Process Dev. Process Software Sprint
• Attribute: Efficiency Sprint Number Size Effort
• Unit: LOC/person-hour dimensionless LOC person-hour
• Range: [0, ∞) [0, 1, 2, ..., n] [0, ∞) [0, ∞)
• Formula (Model): ASP [loc/ph] = (SUMsprint (SO [loc] / SI [ph])) / SC [-]
– Question: Does this formula make sense?
• Note: Since a sprint has a fixed time in days (TD) and a fixed number of fulltime team-members
(TM) with a fixed number of work-hours per day (WHD), one could ’Sprint Input’ (SI) consider a
derived measure, calculated as SI [ph] = TM [p] * TD [d] * WHD [h/d]
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
Structure of Lecture 11
• Homework 2 – Solution Sketch
• Exercise
• GQM (Goal/Question/Metric) Process
• Homework 3
• Standards
– Measurement Process
– Product Quality
– Process Capability/Maturity
• KPI (Key Performance Indicator)
• Balanced Scorecard
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
GQM Core Elements
GQM has three elements:
• Goals
• Questions (and associated Models)
• Measures
Goal
Q1 Q2 Q3 Q4
M1 M2 M3 ...
Implicit
models
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
GQM Core Elements: Goals
• GQM goal (or: Measurement
Goals) are derived from
business or improvement
goals
• A GQM goal defines
–which object is
measured,
–for which purpose,
–with respect to which
quality focus (aspect),
–from which viewpoint,
–and in which context
(environment).
GQM Goal Template
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
GQM Goal – Example
Analyze test process
for the purpose of characterization (understanding)
with respect to (quality aspect) effectiveness
from the viewpoint of the test team
in the environment of project X, organization Y.
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
GQM Question – Examples
• Goal: Analyze the test process for the
purpose of characterization with respect
to (quality aspect) effectiveness from the
viewpoint of the test team in the
environment of project X, organization Y.
• Question 1: How many failures are
detected during testing?
• Question 2: When are failures detected
(time)?
• Question 3: What types of failures are
detected?
• Question 4: How much testing effort is
spent?
• Question 5: Which test techniques/tools
are applied?
• Etc.
Test
Effectiveness,
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
Developing the GQM Hierarchy
Example GQM Hierarchy (incomplete):
• Question 3: What is the distribution of failures by criticality?
• Model: D = F(x, y) = x[y]/x[all], x = Measure 1.1, y = Measure 3.1,
where D: distribution of # failures per criticality class
• Measure 1.1: Failure count (ST: absolute; U: n/a; S: positive
integer; O: product version 1.0)
–Hypothesis: 120 failures
• Measure 3.1: Failure criticality (ST: nominal; U: n/a; S: {critical =
complete breakdown of system, uncritical = unable to perform
one or more of the functions F1, ..., F6, other}, O: failure report)
–Hypothesis: 5% critical failures, 15% major failures, 80%
minor failures
Q3
Goal1
M3.1 M1.1
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
GQM Plan
• The models and measures are
defined by answering ”What kind of
information do we need in order to
answer the questions?"
• The GQM-tree is documented in
tabular form
• Each measure is defined by:
–Name, ID
–Scale, unit, etc.
–Hypotheses
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
Measurement Plan – Example
• Table for tracing Measurement Plan entries to GQM Plan, Project Plan
(based on Process Model) and Data Collection Forms
GQM Plan Project Plan Data Collection Forms
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
The GQM Process Business Goals
Measurement Goals
(Why ?)
GQM Plan (What ?)
Measurement Data
Measurement Plan
(How ?)
Initiate Measurment Project / Set Goals
Define Measures
Define Measurement (and Analysis)
Execute Measurement (Collect data)
Analysis &
Feedback
Development Process
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
Structure of Lecture 11
• Homework 2 – Solution Sketch
• Exercise
• GQM (Goal/Question/Metric) Process
• Homework 3
• Standards
– Measurement Process
– Product Quality
– Process Capability/Maturity
• KPI (Key Performance Indicator)
• Balanced Scorecard
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
Homework 3: SW Measurement
Administrative information:
• This homework has to be done individually. Maximum number of marks: 6
• Submission deadline is Wednesday, 08-Apr-2015, at 20:00 sharp. If you don’t submit in time,
you receive a penalty as follows:
– Late delivery until Thursday, 08-Apr-2015, at 20:00 2 marks penalty (-33%)
– Late delivery after Thursday, 08-Apr-2015, at 20:00 6 marks penalty (-100%)
• No exceptions from the penalty-rules will be made!
• Submit your homework using the ‘Submission’ function provided on the course web-page:
https://courses.cs.ut.ee/2015/SE_Man/spring/Main/Submission
– Make sure to select the correct task, i.e., ‘Homework 3’, when you submit your file.
• IMPORTANT: Only files in PDF format will be accepted! I won’t look at files that are not PDF.
– Please make sure that you have put your name and student ID somewhere at the top of
the first page of the file you are submitting.
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
Homework 3: SW Measurement
• Task T1 [1 mark]
– Understand
measurement
definitions
• Task T2 [1.5 marks]
– Develop a measure
• Task T3 [1 mark]
• Task T4 [1.5 marks]
• Task T5 [1 mark]
– Subjective Meas. &
Reliability
Tasks 3 and 4 are
about ’Abstraction
Sheets’
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
Structure of Lecture 11
• Homework 2 – Solution Sketch
• Exercise
• GQM (Goal/Question/Metric) Process
• Homework 3
• Standards
– Measurement Process
– Product Quality
– Process Capability/Maturity
• KPI (Key Performance Indicator)
• Balanced Scorecard
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
Measurement Process -> ISO 15939
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
SW Product Quality -> ISO 9126
Software Quality Model (ISO 9126 – Part 1)
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
SW Product Quality -> ISO 9126 (cont’d)
External and Internal Measures (ISO 9126 – Parts 2 & 3)
Proposals for measures in ISO9126 - Appendix C (-> reading materials)
Example: Maturity – internal: actual defect detection/correction during development, test adequacy, ... Maturity – external: freedom of the SW system from failures caused by defects in the SW
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
ISO 9126 – Example Application
Measures how many defects (faults, bugs) remain in the SW system that may emerge as future failure.
Formula: X = (abs(A1 - A2)) / B
with: A1 is the total number of predicted latent defects in the system. A2 is the total number of actually occurring failures. B is the system size.
Note: A1 is an internal measure derived from a reliability growth model.
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
Internal vs. External Measures
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
ISO 9126 –
Software Product Quality
Effectiveness relates to how capable is the software in enabling users to achieve specific goals with accuracy and completeness. Productivity relates to how capable is the software in enabling users to reach effective goals with appropriate amounts of resources. Safety relates to the software ability to operate within acceptable level of risk of harm to people, business, property or the environment. Satisfaction relates to the ability of the software product to satisfy the needs of users.
Quality in Use Measures (ISO 9126 – Part 4)
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
ISO 9126 – Future Developments: ISO 25000
• New series of standards currently under development
– Name: Software Product Quality Requirements and Evaluation (SQuaRE - ISO 25000)
• This series of standards will replace the current ISO 9126 (and ISO 14598) series of standards
– Note: the new standard will replace the word ”metric” by “measure”
• Link: http://maisqual.squoring.com/wiki/index.php/ISO/IEC_SQuaRE
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
Software Process & Product Quality
process
quality
Process
Process measures
(ISO 15504)
internal
quality
attributes
external
quality
attributes
quality
in use
attributes
influences
depends on
influences
depends on
influences
Software products Effect of software
product
Internal
measures
External
measures
Quality in use
measures Context
of use
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
Software Process Assessment
CMMI / ISO 15504 (SPICE)
Staged
ML 1
ML2
ML3
ML4
ML5
Defines 5 maturity levels (MLs); in order to
achieve a maturity level all process areas
associated to this level, plus all process areas
associated with levels below must have a certain
minimal capability.
Continuous
A maturity profile is established
based on the capabilities of
individual process areas
PA PA PA
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
Causal Analysis and Resolution Organizational Innovation and Deployment
5 Optimizing
4 Quantitatively Managed
3 Defined
2 Managed
Quantitative Project Management Organizational Process Performance
Requirements Development Technical Solution Product Integration Verification Validation Organizational Process Focus Organizational Process Definition Organizational Training Risk Management Integrated Project Management (for IPPD*) Integrated Teaming* Integrated Supplier Management** Decision Analysis and Resolution Organizational Environment for Integration*
Requirements Management Project Planning Project Monitoring and Control Supplier Agreement Management Measurement and Analysis Process and Product Quality Assurance Configuration Management
1 Performed
Process Areas Level CMMI Levels and
Process Areas
(staged)
* Integrated Product/Process
Development (IPPD) –
add-on to the Engineering
processes
** Acquisition – add-on to the
Engineering processes
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
CMM Assessment Results (continuous)
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
Structure of Lecture 11
• Homework 2 – Solution Sketch
• Exercise
• GQM (Goal/Question/Metric) Process
• Homework 3
• Standards
– Measurement Process
– Product Quality
– Process Capability/Maturity
• KPI (Key Performance Indicator)
• Balanced Scorecard
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
KPI (Key Performance Indicator)
A set of quantifiable measures that a company or
industry uses to gauge or compare performance in
terms of meeting their strategic and operational goals.
Similar to GQM – but:
1. Stronger focus on traceability of
Goals to Business Objectives
2. Explicit targets
3. Lack of guidance on how to find
the right measures (no ’Question’
step)
4. Lack of explicit baselining (where
are we today?)
Example:
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
KPI – A Software Example
Context:
• Ericsson R&D, Zagreb
• Projects on CPP platform
Objective:
• Improve project performance
Key areas of concern:
• Schedule and Progress
• Resources and Cost
• Product Size and Stability
• Product Quality
• Process Performance
• Technology Effectiveness
• Customer Satisfaction
Example KPIs:
Cost Adherence: [1 – (ECost - CCost) / CCost] x 100% measured monthly ECost: expected cost (actual + forecast of remaining cost) CCost: commited cost
Fault Slip Through: [1 – FT Faults / All Faults] x 100% measured monthly FT: Function Test I&V: Intgration & Validation Test All = FT + I&V
Source: An Example of Using Key Performance Indicators for Software Development Process Efficiency Evaluation by Ž. Antolić, R&D Center Ericsson Nikola tesla d.d. (available in reading materials)
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
KPI – A Software Example (cont’d)
Project Benchmarking
Projects:
D1, ..., D6
DFU1, DFU2
M3
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
Structure of Lecture 11
• Homework 2 – Solution Sketch
• Exercise
• GQM (Goal/Question/Metric) Process
• Homework 3
• Standards
– Measurement Process
– Product Quality
– Process Capability/Maturity
• KPI (Key Performance Indicator)
• Balanced Scorecard
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
Balanced Scorecard – 4-Box Model
• A strategic performance
management tool
• A mixture of financial and non-
financial measures each compared
to a 'target' value within a single
concise report
• Four strategic dimensions
• Financial
• Customer
• Internal Business
• Learning and Growth
More info in materials on course-web: Cobbold, I. and
Lawrie, G. (2002a). "The Development of the Balanced
Scorecard as a Strategic Management Tool".
Performance Measurement Association 2002
See also: http://www.balancedscorecard.org/ http://www.ap-institute.com/Balanced%20Scorecard.html
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
Balanced
Scorecard –
Strategy Map
Cause-and-Effect Logic:
• A Strategy Map highlights that
delivering the right performance
in the one perspective (e.g.
financial success) can only be
achieved by delivering the
objectives in the other
perspectives (e.g. delivering
what customers want).
• You basically create a map of
interlinked objectives.
See also: http://www.balancedscorecard.org/ http://www.ap-institute.com/Balanced%20Scorecard.html
MTAT.03.243 / Lecture 11 / © Dietmar Pfahl 2015
Next Lecture
• Topic:
– Lean & Flow-based (KANBAN) Principles and
Processes
• For you to do:
– Finish Homework 3 – Deadline: Wednesday, Apr 8, 20:00
– Continue working on Project
– Optional: Read articles posted on course wiki (Lecture 11)