accounting for non functional and project requirements - cosmic and ifpug development - talmon...

Accoun&ng for Non-‐Func&onal and Project requirements

in so9ware project performance measurement, benchmarking and es&ma&ng:

COSMIC and IFPUG developments

IWSM/Mensura Conference, Krakow, October 2015

Talmon Ben-‐Cnaan (IFPUG), Charles Symons (COSMIC)

IWSM MENSURA Krakow, October 2015

Agenda

•  IntroducJon: Why the need for standards for Non-‐FuncJonal Requirements (NFR) and for Project Requirements and Constraints (PRC)?

•  The joint COSMIC/IFPUG development of a Glossary of NFR and PRC terms

•  The COSMIC Guideline on how to consider NFR and PRC

•  The IFPUG SoTware Non-‐FuncJonal Assessment Process (SNAP) to measure NFR

•  Conclusions and next steps •  QuesJons and debate


The ac&vi&es of so9ware project performance measurement, benchmarking and es&ma&ng

need consistent data and terminology

Measuring project

performance

Project estimating

Benchmarking

Projectdata &

benchmark repository

Recording & measuring software system

project requirements


Func&onal size measurements are used consistently across all ac&vi&es

•  There are three types of requirements for a soTware system project

•  FuncJonal User Requirements (FUR) •  Non-‐funcJonal Requirements (NFR) •  Project Requirements and Constraints (PRC)

•  FuncJonal size measurement methods, e.g. the COSMIC or IFPUG methods, are used consistently to measure the size of FUR across all acJviJes

But what about NFR and PRC?


There is no good exis&ng defini&on of Non-‐Func&onal Requirements (NFR)

Example: ISO 24756 definiJon 1): “A so%ware requirement that describes not what the so%ware will do but how the so%ware will do it. Example: so%ware performance requirements, so%ware external interface requirements, so%ware design constraints, and so%ware quality a>ributes.” Comment: only ‘design constraints’ define ‘how the soTware will do it’


Next, each ac&vity has defined its own data and terminology for NFR and PRC 2)

Requirements Recording (50)

IEEE 804, ISO 9126 Wikipedia

Requirements Sizing (36)

VAF, TCA, SNAP

Benchmarking (48)

ISBSG, SEI

Project Estimating (39)

COCOMO II

One common NFR (= “Interfaces”)


Past surveys have found varying numbers of NFR terms

•  ‘at least’ 161 terms (Chung et al 3))

•  122 terms in a structured hierarchy (Saito et al 4))

•  108 terms (Symons 2))

•  ISO/IEC SQuaRE 5) standard lists 32 Quality terms

A complete, standard list of NFR and PRC terms may be impossible


Past aSempts to measure a collec&ve size of NFR are now discredited

Albrecht’s 10 components of the ‘Value Adjustment Factor’ 6)

14 components (IFPUG) 7)

19+ components (MkII FPA) 8)

….. but they did have value when they were first invented.


Conclusions

•  There is no common understanding in the soTware industry of what is meant by NFR

•  ExisJng standards for NFR and PRC are not coherent •  These issues handicap acceptance of methods for:

•  quanJfying requirements, •  developing project performance benchmarks, •  project esJmaJng.

These were the drivers for COSMIC and IFPUG to develop the joint Glossary of NFR and PRC terms


Agenda

•  IntroducJon: Why the need for standards for Non-‐funcJonal Requirements (NFR) and for Project Requirements and Constraints (PRC)?






Words

What do we mean when saying • A date is a date.

• Right!

•  It is not my type


We have agreed many aspects of NFR and PRC over the past year

•  Scope of the Glossary 9) •  Clarity on:

•  ‘Requirements’ versus ‘constraints’ •  The ‘things’ that NFR and PRC apply •  DisJnguishing and defining NFR and PRC

•  Structuring NFR (aligned with SQuaRE) and PRC to classes à groups à terms

•  The terms: •  67 NFR terms, mainly from ISO and ISBSG (27 COSMIC/IFPUG)

•  19 PRC terms, mainly from PMI and ISBSG


We have clarified the terms Requirements and Constraints

Thesaurus: Requirement: “a thing demanded or obligatory” Constraint: “limitaJon or restricJon”

The difference is not always clear: •  A requirement that the so%ware shall use C# is

also a constraint •  ‘Latency’ can be a requirement for some real-‐Fme

systems but a constraint for the Mars Rover system

•  Staffing skills can be a project requirement or a constraint

Constraints

Requirements

We use requirements for convenience. We use constraints only when it is helpful to disAnguish constraints from requirements.


The scope of requirements and constraints

Requirements and constraints can apply to •  The soCware; •  The data maintained or used by the soTware; •  The technology to be used, e.g. the planorms; •  Other deliverables, e.g. documentaJon or training; •  The combined hardware/soCware system, e.g. a response Jme;

and some constraints come from the environment


Data

Technology

Hardware/Software System

Project

Environment

Other Deliverables

Software

Project, System, Product

A project: a temporary endeavor to achieve defined objecJves of delivering a product by defined dates.

Following a Project, there is a Product in place (A new Product or a Product that was changed by the project). A product is a hardware/ soTware system or an item of soTware such as a soTware package

Product A

Product C

Product B


Requirements for a SoTware System Project

Project Requirements and Constraints (PRC)

System and SoTware Non-‐FuncJonal

Requirements (NFR)

SoTware FuncJonal User

Requirements (FUR)

System Environment Requirements

Technical Requirements

Quality Requirements

(SoTware System)


(Data)

Classifica&on of Requirements


System and SoTware Non-‐FuncJonal Requirements (NFR)

System Environment Requirements

Technical Requirements


(SoTware System)


(Data)

Classifica&on of NFR

1  Quality of the data maintained by the soTware

2  System performance 3  CompaJbility 4  Ease of use 5  System reliability 6  Control of access 7  Maintainability 8  Ease of deployment 9  System or soTware architecture or design

1  Context 2  ApplicaJon Domain 3  ImplementaJons 4  User Base

1  OperaJonal Planorm 2  Database 3  OperaJonal Planorm

constraints 4  Development

requirements

Class

Group

Type


COSMIC and IFPUG defini&ons

FuncAonal User Requirements (FUR) -‐ ISO/IEC 14143-‐1 DefiniAon

Adopted by COSMIC and IFPUG A sub-‐set of the user requirements. Requirements that describe what the soCware shall do, in terms of tasks and services.

NOTE: FuncJonal User Requirements relate to but are not limited to:

•  data transfer (for example Input customer data; Send control signal) •  data transformaJon (for example Calculate bank interest; Derive

average temperature) •  data storage (for example Store customer order; Record ambient

temperature over Jme) •  data retrieval (for example List current employees; Retrieve latest

aircraT posiJon)



Non-functional requirement – COSMIC and IFPUG Definition Any requirement for a soCware system or for a soCware product, including how it should be developed and maintained, and how it should perform in operaAon, except any funcAonal user requirement for the soCware. Non-‐funcJonal requirements concern: •  the soTware system or soTware product quality; •  the environment in which the soTware system or soTware product

must be implemented and which it must serve; •  the processes and technology to be used to develop and maintain the

soTware system or soTware product, and the technology to be used for their execuJon.



Project Requirements and Constraints - Definition Requirements that define how a soCware system project should be managed and resourced, or constraints that affect its performance. Requirements may include:

•  The targets the project should achieve (e.g. budget, delivery date, product quality);

•  The project management processes that should be used; •  How the project should be governed and resourced.

Constraints may include: •  LimitaJons on the project resources; •  Dependencies on other projects outside the control of the project concerned.


67 Terms

Examples

COSMIC / IFPUG DefiniJon

NFR sub-‐type (Quality, System Environment Requirements, Technical Requirements) Quality: 9 groups

Environment: 4 groups Technical: 4 groups

ISO DefiniJon


COSMIC and IFPUG strongly recommend their joint Glossary

• Developed through > 20 iteraJons over a year • Reviewed by many experts (academic and pracJJoners) from around the world

• Available for free download from: •  www.cosmic-‐sizing.org •  www.ifpug.org

• We welcome feedback and comments


Agenda







The COSMIC Guideline builds on the joint COSMIC/IFPUG Glossary

Contents •  Terminology, NFR/PRC definiJons

• ClassificaJon of NFR/PRC terms • Glossary of NFR/PRC terms •  EvoluJon of NFR in a project • How to deal with NFR/PRC in performance measurement, benchmarking & esJmaJng

• Measurement of NFR (interim)

Glossary9) ü  ü  ü 

Guideline10) ü  ü  ü  ü  ü 

ü 


Requirements that first appear as NFR may evolve, wholly or partly, into FUR

Outline Funct- -ional

Requts.

Outline NFR

Project Requts. & Constraints

Require- ments

Analysis

Definition & Design

Build, Test &

Implement

Architecture

Approx. Funct- -ional

Requts.

Detailed NFR

Imple- mented software system

or software product

Detailed FUR

…… as demonstrated by Al Sarayreh, Abran and others, e.g. [11]


The ‘addi&onal’ FUR can be measured by approximate or standard COSMIC sizing

Outline Funct- -ional

Requts.

Outline NFR

Project Requts. & Constraints

Require- ments

Analysis

Definition & Design

Build, Test &

Implement

Architecture

Imple- mented software system

or software product

Approx. Funct- -ional

Requts.

Detailed NFR

Detailed FUR

Size by analogy or expert judgement

Approx. COSMIC size measurement

Precise COSMIC size measurement


The Guideline gives examples for all Quality NFR how they may evolve as a

project progresses

FuncAonality that may evolve from the NFR, that can be measured

‘Middleware’ funcJonality to enable portability across mulJple DBMS or OS or hardware

Graphical User Interface funcJons; and FuncJonality to assist users, e.g. ‘Help’ funcJons

FuncJonality to import external data in real-‐Jme so that it is available for immediate use

IniAal NFR term

Portability

Usability

Response Jme

‘Residual’ or ‘Real’ NFR statement

The specific environments across which the soTware must be portable

The specific usability requirements (e.g. ‘must be usable by public with no training’)

-‐ The response Jme target; -‐ Specific hardware; -‐ Low-‐level programming language.

Examples:


COSMIC has no plans to develop a collec&ve size measure for NFR

Inherent difficulJes:

1.  How to form a collecJve size measure that will add pracJcal value, long-‐term?

2.  The large number and variety of NFR

3.  How to collect effort data related to NFR when NFR evolve into FUR?


1. Collec&ve size measures are common and o9en valuable

Examples of valuable collecJve size measures. (All are mathemaJcally invalid, but they work.) •  The IFPUG Value Adjustment Factor (when first introduced) •  Indices that affect stock market senJment, e.g. the PMI •  Measures of ‘biological age’ •  Total Factor ProducJvity

CondiJons for a usable collecJve size measure: •  A finite, stable, well-‐defined set of components •  If possible, a common unit of measure for all components •  A simple formula for the index (avoid complex maths)


2. There is a large number and variety of NFR. Many overlap in meaning

How many separate NFR should we include in a collecJve size measure? (10, 14, 19+, 67, 108, 122, 161+ …?)

Maintain-‐ ability

Test-‐ ability

Flex-‐ ibility

Adapt-‐ ability

Extend-‐ ability

Evolv-‐ ability

Modula-‐ rity

Modifia-‐ ability

Reus-‐ ability

Port-‐ ability


3. How to capture the effort associated with implemen&ng NFR?

…. given that NFR evolve as a project progresses into FUR ….. ?

.... & noJng that effort to implement NFR varies with:

•  soTware domain, •  type of NFR, •  the parJcular ‘soluJon’ for the NFR, •  technology, re-‐use, etc.


Instead, we propose basic sets of NFR/PRC to record and measure for performance

measurement, benchmarking, es&ma&ng

NFR and PRC Class

Terms

Quality NFR Response time, Transaction rate Security, Privacy Maintainability, Reusability Interfaces, Operational processing mode

System Environment NFR

Application type, sub-type Implementations (numbers of) Maximum number of concurrent users

Technical NFR Operational platform type Programming language DBMS software

Project Requirements and Constraints (PRC)

Many, but not all of the 19 project requirements and constraints are worth recording. Example: if staff experience levels, processes and methods and tools are the same for all projects, then they need not be recorded for internal purposes.


…. and advise how to deal with NFR in performance measurement, benchmarking and es&ma&ng

•  EsJmate and measure total funcJonal size, including funcJonality that evolves from NFR

• Within a defined ‘environment’, i.e. •  soTware domain (business, real-‐Jme) •  soTware architecture, technology •  project type (new, enhancement, etc)

…. collect and record data on ‘real’ NFR and PRC so that you can make like-‐for-‐like comparisons (This is normal pracJce for benchmarking acJviJes, plus a few addiJonal parameters.)


Guideline Conclusions

The COSMIC ‘Guideline for dealing with Non-‐FuncFonal Requirements in project performance measurement & esFmaFng’ • builds on the joint COSMIC/IFPUG Glossary • provides pracJcal advice on how to deal with manageable sub-‐sets of NFR and PRC

and will be available in October for free download from www.cosmic-‐sizing.org


Agenda







FPA and SNAP covers functional and non-functional requirements

PRC – do not impact soTware size

NFR – measured with SNAP Points

FUR – measured with FP

Guidelines to prevent doubled counJng


PRC affects produc&vity and not size

•  Effort depends on PRC classes. No standard / industry mathemaJcal formula

•  Examples: Project type; (same size different effort) •  Effort may be formulated for some PRC (“producJvity drivers”)

•  Examples: Skill level; LocaJon; Schedule compression (“crashing”) (same size different effort)

•  Effort (or deviaJon from esJmaJon) can be explained by some PRC

•  Examples: Risk that was materialized; scope creep


SNAP

•  SNAP idenJfies fourteen non-‐funcJonal characterisJcs (“sub categories”) that classify the way NFR are met.

•  In each sub-‐category, the size is assessed within a counJng unit (SCU). The SCU is part of the definiJon of the sub-‐category.

•  Non funcJonal size is determined by a set of parameters / complexity levels. The parameters that define each sub-‐category answer the quesJon “what factors affect the effort to deliver a NFR”

•  Example: The effort to add input methods (e.g. barcode reader, fax server, reading CSV file) with same funcJonality depends on number of added input methods


SNAP

• Parameters can be used as a table of values (e.g. “High, medium, low”, “<10, 10 to 15, 16+”) or as a mulJplier (“SP = 3*# addiJonal input methods), or a combinaJon of the above

• Once the parameters are assessed, the size of the SCU is calculated. SP size is the sum of the SPs of the SCUs


SNAP sub categories are independent of the way NFR are classified / defined

Accuracy

Performance

Ease of use / customer satisfaction

Response Time

Requ

iremen

ts

CharacterisJcs


SNAP sub-‐categories are independent of the way NFR are classified

Accuracy

Ease of use / customer satisfaction

Response Time

ISO

912

6 ISO

250

10

CO

SM

IC /

IFP

UG

Glo

ssar

y

Requ

iremen

ts

CharacterisJcs

Time Behavior

Performance


Data Func&ons and Transac&onal Func&ons are not sufficient to size NFR

The effort to deliver NFR is also affected by • Database ac&vi&es • Dealing with UI proper&es •  Extensive Logical / Mathema&cal opera&ons • Batch processes that do not cross the boundaries and are not exposed to users

• Mul&ple inputs/output • …..


Example 1 – The requirements

NFR: Improve CRM system performance for “Retrieve and View Customer Details“-‐ by loading the ‘customer details’ screen in 3 seconds or less. (Currently it takes 0.5 sec to 6 sec.) Problem analysis: The system is slow when the user (call center representaJve) opens a big customer with many products and many interacJons. It also happens when users’ virtual memory is low


Example 1 -‐ the design

1.  Increase the virtual memory of ‘Windows’ to maximum; 2.  Create a database view, which includes data from three tables

(Assigned Products, TransacJons, and Payments) (Assuming 30 different DETs are fetched).

3.  Add Logic to idenJfy which customers will be loaded to the new view

4.  Add a field to ‘Customers’ table to indicate whether recent informaJon moved to the new view and add logic to idenJfy which transacJons will be copied (not seen by the users, hence non-‐funcJonal change).

5.  A batch process runs in the background every 2 hours, refreshing this view with large and most acJve users


Example 1 – SNAP count

•  Increasing virtual memory: InstallaJon of a larger size virtual memory is a hardware configuraJon and not a soTware change. It is not counted using soTware sizing methods.

• CreaJng a database view (CUST_DETAIL_SUMMARY_ TEMP) and adding an indicator (field) to ‘Customer’ table:

•  Count SP using 3.2 Database changes; SNAP count is based on two database changes, 20-‐50 DETs, 2-‐5 RETs

• Adding a batch process: (this batch job does not cross the boundary, hence it is not counted using FP)

•  Count SP using 3.3 Batch Processes. SNAP count is based on 30 DETs and 3 FTRs

• Adding logic to meet the NFR: Count SP using 1.2 Logical and MathemaJcal OperaJons. SNAP count is based on 3 FTRs, type = ‘Logical’ and 10 DETs used for the added logic


IFPUG Coun&ng Process

Gather available documentaJon

Determine counJng purpose, scope, boundaries and

parJJons

IdenJfy requirements as FUR, NFR. Separate mixed requirements to FUR and NFR

Measure Data FuncJons

Measure TransacJonal FuncJons

Associate NFR to sub-‐categories

Calculate funcJonal size

Determine SNAP size of each sub

category

Calculate SNAP size

Document and report

FuncJonal

Non-‐FuncJonal


Es&ma&on and benchmark

Es&ma&on 𝑬𝒇𝒇𝒐𝒓𝒕=

[𝑷𝑫𝑹↓𝟏 (𝑷𝒓𝒐𝒋𝒆𝒄𝒕↑′ 𝒔 𝑪𝒉𝒂𝒓𝒂𝒄𝒕𝒆𝒓𝒊𝒔𝒕𝒊𝒄𝒔)×𝑭𝒖𝒏𝒄𝒕𝒊𝒐𝒏𝒂𝒍 𝑺𝒊𝒛𝒆]+[ 𝑷𝑫𝑹↓𝟐 (𝑷𝒓𝒐𝒋𝒆𝒄𝒕↑′ 𝒔 𝑪𝒉𝒂𝒓𝒂𝒄𝒕𝒆𝒓𝒊𝒔𝒕𝒊𝒄𝒔)×𝑵𝒐𝒏−𝒇𝒖𝒏𝒄𝒕𝒊𝒐𝒏𝒂𝒍 𝑺𝒊𝒛𝒆]

PDR – ProducJvity Delivery Rate Note: Feedback from users show that PDR2 diverges in 3 of the 14 sub-‐categories. NSFFC will reformulate them based on collected data Benchmarking •  Based on linear regression to extract PDR1 and PDR2: •  Two different baselines, two business values to customers


Agenda







COSMIC and IFPUG have come closer over the past year

Close exchange of ideas has led to: • Agreeing to disJnguish NFR and PRC • AdopJng PMI definiJons for PRC terms • Many refinements to the classificaJon structure and Glossary contents – through > 20 iteraAons

Further, we agree that: • NFR increase the size of the soTware, and should be measured

• Performance measurement, benchmarking and project esJmaJng must consider both FUR and NFR


Conclusions

Whilst COSMIC & IFPUG sJll have different views on how to measure NFR, we have collaborated very well to agree:

• DefiniJons and classificaJon of NFR and PRC •  The Glossary of NFR and PRC terms, v1.0

We strongly recommend these to the soTware

engineering community


Next steps

•  Enhance the Glossary to take into account ‘Data Quality’ requirements

•  Enhance the Glossary (and define measurements?) to take into account ‘Other Deliverables’ such as training and documentaJon This is a quesJon to YOU: Should we?????

Thank you for your a^enAon

Talmon Ben-‐Cnaan (www.ifpug.org );

[email protected]

Charles Symons (www.cosmic-‐sizing.org ); [email protected]


References

1.  ISO/IEC/IEEE 24765:2010 ‘Systems and soTware engineering – vocabulary’. 2.  Symons, C.R., ‘AccounJng for Non-‐FuncJonal Requirements in ProducJvity Measurement, Benchmarking and

EsJmaJng’, UKSMA/COSMIC InternaJonal Conference on SoTware Metrics & EsJmaJng, London, 27/28 October 2011.

3.  L. Chung, B. Nixon, E. Yu, and J. Mylopoulos, "Non-‐funcJonal Requirements in SoTware Engineering,“ Kluwer Academic Publishing, 2000.

4.  Saito, Y., Monden A., Matsumoto K., ‘EvaluaJon of non-‐funcJonal requirements in a request for proposal (RFP)’, Nara InsJtute of Science & Technology, Japan, at InternaJonal Workshop on SoTware Measurement (IWSM), Nara, 2012..

5.  ISO/IEC FCD 25010: Systems and soTware engineering, –System and soTware product Quality Requirements and EvaluaJon (SQuaRE) –System and soTware quality models

6.  Albrecht, A. J., ‘Measuring applicaJon development producJvity’, In Proc. of the IBM ApplicaJons Development Symposium, pp. 83-‐-‐92. Monterey, California (1979)

7.  IFPUG, CounJng PracJces Manual, Release 4.3.1 – Appendix C

8.  Symons, C.R., SoTware sizing & esJmaJng: Mk II FPA’, John Wiley & Sons, 1991, ISBN 0 471 92985 9

9.  ‘Glossary of terms for Non-‐FuncJonal Requirements and Project Requirements used in soTware project performance measurement, benchmarking and esJmaJng’, Version 1.0, September 2015

10.  ‘Guideline on Non-‐FuncJonal & Project Requirements: How to consider non-‐funcJonal and project requirements in soTware project performance measurement, benchmarking and esJmaJng’, version 1.0, October 2015

11.  Khalid T. Al-‐Sarayreh, Alain Abran and Juan J. Cuadrado-‐Gallego, ‘A Standards-‐based model of system maintainability requirements’, Journal of SoTware: EvoluJon and Process, 2013, Vol. 25, no. 5, pp. 459-‐505