empirical software engineering

Rahim LotfiAhmadReza Ghasemian

1

Table of Contents

Introduction Need for empirical studies in the context of software engineering.

Goals and Benefits

Basic Concepts: Measurement, Collection, Analysis

Basic Concepts for Empirical Software Engineering

Empirical Strategies: survey, case study, controlled experiment

Process of an empirical study

Tools

State of the art

Empirical Software Engineering

• Using other people’s experiences in management and development of softwares

• Empirical, included in all areas of science

• “I could have told you that from the start” or “That’s obvious.”

• Software engineering needs to follow the model of other physical sciences and

develop an experimental paradigm for the field.

• Judging empirical work by the standards of statistics texts would lead to

the suppression of many interesting results

ESE in the Literature

• “Empirical Software Engineering”

• “Experimentation in Software Engineering”• “ESEM - Empirical Software Engineering and Measurement “

History of Empirical Software engineering

• Victor Basili(1996)

• QuotesLike physics, medicine, manufacturing, and many other disciplines, software engineering requires the same high level approach for evolving the knowledge of the discipline; the cycle of model building, experimentation, and learning. We cannot rely solely on observation followed by logical thought.

History of Empirical Software engineering

Key figure

http://academic.research.microsoft.com

Key figures in Iran

Conferences (64) Empirical Software Engineering

Journals (29) Empirical Software Engineering

Table of Contents

Introduction Need for empirical studies in the context of software engineering.

Goals and Benefits

Basic Concepts: Measurement, Collection, Analysis

Goal Question Metric

Empirical Strategies: survey, case study, controlled experiment

Process of an empirical study

Tools

State of the art

Need of Empirical Studies

Why should we conduct empirical studies?

Product evaluation, e.g., prototyping.

Process evaluation1. Prototypes are not possible (simulation based on models).

2. A process is just a description until it is used by people

Important for research: experimentation is mandatory in other disciplines(e.g., medicine, physics, etc.)

Need of Empirical Studies

Empirical studies provides a systematic, disciplined, quantifiable and controlled way of evaluating human-based activities.

You will learn how to select, plan, conduct and analyse an empirical study.

Table of Contents

Introduction

Need for empirical studies in the context of software engineering.

Goals and Benefits Basic Concepts: Measurement, Collection, Analysis

Basic Concepts for Empirical Software Engineering

Empirical Strategies: survey, case study, controlled experiment

Process of an empirical study

Tools

State of the art

Goals and Benefits• The purpose of a study is

• The purpose of a study isto explore ...

to find out what’s happening

to seek for new insights

to ask questions and to find answers

Measurement: usually qualitative

• to describe ...

portray accurate profile of situations, events, projects , technologies

Measurement: quantitative/qualitative

• to explain ...

seek explanation of a situation/problem, usual in the form of causal

Relationships

Measurement: quantitative/qualitative

• relationships, differences, changes

Empirical Studies in SE

• Empirical Studies is research to improve Software Engineering Practice.

– Apply theoretical concepts in SE practice.

– Add experiences on their appliance to the SE ‘body of knowledge’

– Improve processes, methods and tools (SPPI approach).

– Verify theories and models.

Table of Contents

Introduction

Need for empirical studies in the context of software engineering.

Goals and Benefits

Basic Concepts: Measurement, Collection, Analysis Basic Concepts for Empirical Software Engineering

Empirical Strategies: survey, case study, controlled experiment

Process of an empirical study

Tools

State of the art

Some Basic Concepts• Measurement

- is the process of capturing data which are connected to real-world attributes to

describe them.

- Why is measurement important?

• Data CollectionCollection of qualitative / quantitative data according to research questions

• Data AnalysisAnalyzing the results according to the research questions ;

statistical tests

Measurement

• Quotes- “You can’t manage what you can’t measure”, Tom DeMarco

– “What is not measurable make measurable”, Galileo Galilei

• ObjectivesOne objective of science is to find ways to measure attributes of entities

we are interested in.

– Measurement makes concepts more visible and thus more

understandable and controllable.

• Definition

Measurement is the process by which numbers or symbols are mapped

to attributes of entities in the real world in such a way as to describe

them according to clearly defined rules.

Introduction

Need for empirical studies in the context of software engineering.

Goals and Benefits

Basic Concepts: Measurement, Collection, Analysis

Basic Concepts for Empirical Software Engineering Empirical Strategies: survey, case study, controlled experiment

Process of an empirical study

Tools

State of the art

Table of Contents

Quality Improvement Paradigm

Characterize & understand

Setgoals

Chooseprocesses,methods,techniques,and tools

Package &store experience

Analyzeresults

Executeprocess

Provide processwith feedback

Analyzeresults

Corporatelearning

Projectlearning

The Experience Factory Organization

A Different Paradigm

Project Organization Experience FactoryProblem Solving Experience Packaging

Decomposition of a problem Unification of different solutionsinto simpler ones and re-definition of the problem

Instantiation Generalization, Formalization

Design/Implementation process Analysis/Synthesis process

Validation and Verification Experimentation

Product Delivery within Experience / RecommendationsSchedule and Cost Delivery to Project

Introduction

Need for empirical studies in the context of software engineering.

Goals and Benefits

Basic Concepts: Measurement, Collection, Analysis

Basic Concepts for Empirical Software Engineering

Empirical Strategies: survey, case study, controlled experiment

Process of an empirical study

Tools

State of the art

Table of Contents

Table of Contents

Introduction

Need for empirical studies in the context of software engineering.

Goals and Benefits

Basic Concepts: Measurement, Collection, Analysis

Basic Concepts for Empirical Software Engineering

Empirical Strategies: survey, case study, controlled experiment

Process of an empirical study

Tools

State of the art

Table of Contents

Introduction

Need for empirical studies in the context of software engineering.

Goals and Benefits

Basic Concepts: Measurement, Collection, Analysis

Goal Question Metric

Empirical Strategies: survey, case study, controlled experiment

Process of an empirical study

Tools State of the art

Table of Contents

Introduction

Need for empirical studies in the context of software engineering.

Goals and Benefits

Basic Concepts: Measurement, Collection, Analysis

Basic Concepts for Empirical Software Engineering

Empirical Strategies: survey, case study, controlled experiment

Process of an empirical study

Tools

State of the art

State of the art

• Finding better ways to experiment and integrate the results of the studies

• Propagate the empirical discipline

• Build an empirical research engine for software engineering

• Build test beds for experimentation and evolution of processes

• Build product models that allow us to make trade-off decisions

• Build decision support systems offering the best empirical advice for selecting and tailoring the right processes for the problem

• References• V. Basili, G. Caldiera, D. Rombach: „The Goal Question Metric Approach“,2000.

• B. Boehm, H.D. Rombach, M.V. Zelkowitz: „Foundations of Empirical

• Software Engineering – The Legacy of Victor R. Basili“, Springer, 2005.

• Freimut et al.: "Empirical Studies in Software Engineering", Tutorial, VISEK

• Technical Report, 2002.

• IESE Tutorials on Empirical Software Engineering.

• A. Jedlitschka and D. Pfahl: "Reporting Guidelines for Controlled

• Experiments in Software Engineering", ISESE, 2005.

• B. Kitchenham: “Evidence-Based Software Engineering and Systematic

• Literature Review”, Profes, 2006.

• C. Wohlin, P. Runeson, M. Höst, M.C. Ohlsson, B. Regnell and A. Wesslen:

• "Introduction to Experimentation in Software Engineering", Kluver, 2000.

• M. V. Zelkowitz, D. R. Wallace: “Experimental Models for Validating

• Technology”, IEEE Computer, 1997.

• Site : http:// www.microstrategy.com

• Site : http:// www.jamasoftware.com

• Site : http://academic.research.microsoft.com

AcknowledgmentsBahman Zamani

Thank you for your attention

This research work has been supported by University of Isfahan