verification and validation · 26.11.14 b. wolff - gla - resume 10 some relevant standards (recall)...

2014-2015

Verification and ValidationPart 15 : A Resume

Burkhart WolffDépartement Informatique

Université Paris-Sud / Orsay

26/11/14 B. Wolff - GLA - Resume 2

What is Software Engineering ?

" Methods, techniques and tools for# design: requirement analysis, models,

specifications# development: programmation, integration# validation: prototypes, testing# verification: formal proof of required properties# maintenance: reusability, improvements

" A slightly longer answer:

26/11/14 B. Wolff - GLA - Resume 3

What is Software Engineering ?

" ... slightly longer answer:

The discipline of software engineering was created to address poor quality of software, get projects exceeding time and budget under control, and ensure that software is built systematically, rigorously, measurably, on time, on budget, and within specification. [Wikipedia [en]]

" Or much shorter:

SE addresses the problems of

« Development in the Large » ... so for teams with 100 or 1000 of developers, and budgets of sometimes billions of dollars.

26/11/14 B. Wolff - GLA - Resume 4

What is Software Engineering ?

" ... slightly longer answer:

Reminder: What is it, when I talk about " Or much shorter:

SE addresses the problems of

« Development in the Large » ... so for teams with 100 or 1000 of developers, and budgets of sometimes billions of dollars.

26/11/14 B. Wolff - GLA - Resume 5

What is Software Engineering ?

" ... slightly polemically:

Reminder: What is it, when I talk about Software Engineering ? Writing:

! “Write-once, throw away” programs ?! Programs written by a small team with 50 KLoCs?! Our LRI Website-Service

OR:

! programs in industrial context with large institutions (states, companies) commissioning it ?

26/11/14 B. Wolff - GLA - Resume 6

Answers to the Problem of Software Engineering

" Development Processes

" Standards on Processes, Documentation, and organization

" Process-support# Version management

(automatic merge problem)# Configuration management# Build Management# Analysis-Techniques of the merges

26.11.14 B. Wolff - GLA - Resume 7

2013-14 L2-GL - Intro 12

One way to view Software Engineering Project

MANAGEMENT PROCESS

Development Mgt. Risc Management

ConfigurationManagement

« PeopleWare »(Staff, Sub-contractors)

PROCESSUS QUALITE

QUALITYASSURANCE

QUALITYCONTROL

METRICS

FEASIBILITY STUDIES (BEFORE PROJET)

SPECIFICATION & DESIGN

PRODUCTION

INTEGRATION & VALIDATION

EXPLOITATION

TECHNICAL PROCESS

Development

Maintenance & Support

26/11/14 B. Wolff - GLA - Resume 8

How can software be «built systematically»?" A Variant: The VPM3-Model (Daimler)

26/11/14 B. Wolff - GLA - Resume 9

Answers to the Problem of Software Engineering

" Development Processes

" Standards on Processes, Documentation, and organization

" Process-support# Version management

(automatic merge problem)# Configuration management# Build Management# Analysis-Techniques of the merges# Verification and Validation Techniques

26.11.14 B. Wolff - GLA - Resume 10

Some Relevant Standards (Recall)

" Many attempts to control development processes andsoftware products by standards (norms)

" Some attempts to assure and certify software quality.# Most serious and relevant (in France):# DO 178B (Avionics)# ISO 29119 (Software Test) # ISO 15408 « Common Criteria» for computer

security certification requiring formal models as well as proof techniques for EAL 6 and EAL 7.

26/11/14 B. Wolff - GLA - Resume 11

Answers to the Problem of Software Engineering

" Development Processes

" Standards on Processes, Documentation, and organization

" Process-support# Version management

(automatic merge problem)# Configuration management# Build Management# Analysis-Techniques of the merges# Verification and Validation Techniques

26/11/14 B. Wolff - GLA - Resume 12

Concepts of Central. Version Control(Recall)

" Working copises(in user space)

" Repository(on the server-side)

" update:syncing with therepository

" commit: creating a new revision of a document(involves new registration, inclusion in documents, consistency checks)

" operations lock, checkout, import, ...

26/11/14 B. Wolff - GLA - Resume 13

Build Management: A Build-Server (Recall)

"

26/11/14 B. Wolff - GLA - Resume 14

Build Management: A Build-Server (Recall)

"

26/11/14 B. Wolff - GLA - Resume 15

The Problem of Software Engineering

" In a large process, automatic support formerges of documents and code becomes vital

" This raises the problem of consistency of products:documents and code

" Answers we know of:# . . .

26/11/14 B. Wolff - GLA - Resume 16

Answers to the Problem of SE

" Answers we know of:# more Formal Modeling (in Analysis and Design)# Static Analysis-Techniques of the merges

(from type-checking in UML and code,to MISRA code style checkers)

# Verification and Validation Techniques! Black box tests! White box tests! Deductive Verification

26/11/14 B. Wolff - GLA - Resume 17

Answers to the Problem of SE

" Answers we know of:# more Formal Modeling (in Analysis and Design)# Static Analysis-Techniques of the merges

(from type-checking in UML and code,to MISRA code style checkers)

# Verification and Validation Techniques! Black box tests! White box tests! Deductive Verification

26/11/14 B. Wolff - GLA - Resume 18

A first Glance to an Example: Bank

Opening a bank account. Constraints:" there is a blacklist" there is a present of 15 euros in the initial account" account numbers must be distinct.

26/11/14 B. Wolff - GLA - Resume 19

A first Glance to an Example: Bank (2)

" context Compte:inv unique: Compte.allInstances->isUnique(c | c.no)

context Banque::ouvrirCompte(nomC: String) : Integer

pre: Personne.allInstances->forall(p | p.nom <> nomC)post: Personne.allInstances->isUnique(p | p.nom = nomC

and p.oclIsNew() and )

and Compte.allInstances ->isUnique(c | c.titulaire.nom = nomC and c.oclIsNew() and c.solde = 15)and Personne.allInstances ->includesAll(Personne.allInstances@pre)and Compte.allInstances ->includesAll(Compte.allInstances@pre)

26/11/14 B. Wolff - GLA - Resume 20

Answers to the Problem of SE

" Answers we know of:# more Formal Modeling (in Analysis and Design)# Static Analysis-Techniques of the merges

(from type-checking in UML and code,to MISRA code style checkers)

# Verification and Validation Techniques! Black box tests! White box tests! Deductive Verification

26/11/14 B. Wolff - GLA - Resume 21

Static Analysis (Recall)

" Modern Static Analysis Methodsare typically a combination of the following techniques# Data-Flow, Control-flow, Patterns, (“Classics”)# Symbolic Execution (“forward analysis”)# Deductive Verification (wp like, “backw. ana.”)# Model-Checking# Abstract Interpretation # Predicate Abstraction,# Interpolant Construction

26/11/14 B. Wolff - GLA - Resume 22

Static Analysis (Recall)

" Modern Static Analysis Methodsare typically a combination of the following techniques# Data-Flow, Control-flow, Patterns, (“Classics”)# Symbolic Execution (“forward analysis”)# Deductive Verification (wp like, “backw. ana.”)# Model-Checking# Abstract Interpretation # Predicate Abstraction,# Interpolant Construction

26/11/14 B. Wolff - GLA - Resume 23

Answers to the Problem of SE

" Answers we know of:# more Formal Modeling (in Analysis and Design)# Static Analysis-Techniques of the merges

(from type-checking in UML and code,to MISRA code style checkers)

# Verification and Validation Techniques! Black box tests! White box tests! Deductive Verification

26/11/14 B. Wolff - GLA - Resume 24

Testing Methods (Recall)

" A taxonomy on types of tests# Static Test / Dynamic (Runtime) Test # Structural Test / Functional Test# Statistic Tests

" Functional Test; Link to UML/OCL# Dynamic Unit Tests, Static Unit Tests,

# Coverage Criteria

" Structural Tests# Control Flow and Data Flow Graphs

# Tests and executed paths. Undecidability.

# Coverage Criteria

26/11/14 B. Wolff - GLA - Resume 25

Testing Methods (Recall)

" It is an approximation to full verification (for ex. by proof)" Main emphasis: finding bugs early,

# either in the model# or in the program# or in both

" A systematic test is:# process programs and specifications

and to compute a set of test-cases under controlled conditions.

# ideally: testing is complete if a certain criteria,the adequacy criteria is reached.

26/11/14 B. Wolff - GLA - Resume 26

Testing Methods (Recall)

" Answers we know of:# more Formal Modeling (in Analysis and Design)# Static Analysis-Techniques of the merges

(from type-checking in UML and code,to MISRA code style checkers)

# Verification and Validation Techniques! Black box tests! White box tests! Deductive Verification

26/11/14 B. Wolff - Ingé. 2 - Proof-Based Verification II 27

Hoare – Logic: A Proof System for Programs

" Hoare revisited (i):

" ... this part is actually highly deterministic

26/11/14 B. Wolff - Ingé. 2 - Proof-Based Verification II 28

Hoare – Logic: A Proof System for Programs

" Hoare revisited (ii):

" ... this part needs some work, and some new ideas.

26/11/14 B. Wolff - Ingé. 2 - Proof-Based Verification II 29

Tools: gwhy and Squareroot

26/11/14 B. Wolff - GLA - Resume 30

Resume : Test vs. Proof

Test

# Requires Testability of Programs (initializable,reproducible behaviour, sufficient control over non-determinism)

# Can be also Work-Intensive !!!

# Requires Test-Tools, whose details can be very complicated

# Requires a Formal Specification

# Makes Test-Hypothesis, which may be hard to justify !

26/11/14 B. Wolff - GLA - Resume 31

Resume : Test vs. Proof

Deductive Verification

# Needs knowledge / assumptions on the precise execution of the language and the the environment(may be difficult in practice ... gcc -o3 !!!)

# Is work-intensive and intellectually expensive !!!

# Requires Tools, whose details can be complicated

# Requires a Formal Specification

# Currently restricted to abstract models or realistic C with code-size 10kloc.)

26/11/14 B. Wolff - GLA - Resume 32

An Example for an Extended Static Checking (Combi-) Method: SAL Annotations

# Due to heavy machinery(heuristics, patterns, abstract interpretation ...)

the annotation of loops with invariantsis not necessary by the user for the SALlanguage.

The programmer has just to provide contracts.

Since 2006, Microsoft annotates the entireWindows and Word code-base with SAL.

26/11/14 B. Wolff - GLA - Resume 33

An Example for an Extended Static (Combi-) Checking Method: SAGE

# ... a white box fuzz - tester.

26/11/14 B. Wolff - GLA - Resume 34

Static Analysis : (recall)

" Lots of Methods with the same Goal:

More automation, better prediction of “real errors”

(i.e. less false-positives)better explanation of “real errors”

(very difficult)less constraints on the supported language subset

(“real” C, “real” Java) trend to “dirty languages”such as JavaScript

26/11/14 B. Wolff - GLA - Resume 35

Global Summary

" Current Research and Trends:

# Combination of Methods

# Test and Proof have a lot in common

# A lot of theory needs to combined with heavy technology