approaches to improve system dependability – from formal...

C O

R P

O R

A T

E

T E

C H

N O

L O

G Y

Approaches to Improve System Dependability – From Formal Verification to Model-Based Testing

Andreas Ulrich, Peter Amthor, Marlon VieiraSiemens AG, Corporate Technology, CT SE/[email protected]

C O

R P

OR

A T

E

T E

C H

N O

L O

G Y

2Approaches to Improve System Dependability 7. Bieleschweig Workshop, 4./5. Mai 2006

>300

>2000

>200

Research &Development

IntellectualProperty & Functions

others

>1900 >500

(Distribution of employees by functions)

Munich ErlangenBerlin

St. Petersburg MoscowRomsey (RMR)Princeton

Bangalore

Shanghai

Beijing

Tokyo

Berkeley

Siemens Corporate TechnologyPresent in all leading markets and technology hot spots

C O

R P

O R

A T

E

T E

C H

N O

L O

G Y

DiscreteOptimization

SE 6

DiscreteOptimization

SE 6

SystemsEngineering

SE 5

SystemsEngineering

SE 5

ProjectManagement

SE 4

ProjectManagement

SE 4 Software Initiative

Software Initiative

Software Processes

SE 3

Software Processes

SE 3

ArchitectureSE 2

ArchitectureSE 2

Information Broker

Information Broker

Software &Engineering

DevelopmentTechniques

SE 1

DevelopmentTechniques

SE 1

System and Software Processes

Software Architecture for Distributed,

Mobile und Embedded Systems

Siemens Software InitiativeProject Management and Innovation

Information Brokers and Technical Liaison Managers

Quality and Efficiency inSoftware Development

Optimization of Planning, Decision, and Production Processes

Analysis and Engineering of Complex Systems

Corporate Technology, Software & EngineeringC

O R

P O

R A

T E

T

E C

H N

O L

O G

Y


Our focus

Dependability CompetenceTeam at Siemens CT SE

Focus of Work in Dependability Engineering

Dependability

Attributes

Means

Threats

Availability

Reliability

Safety

Confidentiality

Integrity

Maintainability

Fault prevention

Fault tolerance

Fault removal

Fault forecasting

Faults

Errors

Failures

[source: J.-C. Laprie et al., 2000]

C O

R P

OR

A T

E

T E

C H

N O

L O

G Y


Fault Prevention by Model Driven Designand Formal Verification

Application Domain

Requirements

FormalVerification

Formal Model(refined)

formalize

Propertiesto be checked

Results

Informal

Design Model

Correct Model

refine

C O

R P

OR

A T

E

T E

C H

N O

L O

G Y


Formal Verification

Formal model

Typically extracted manually from an informal model and requirements

But possible reuse of models from model-driven design

• Matlab/Simulink

• Lustre, Esterel (Scade)

• Statecharts, UML models (Rhapsody etc.)

Requires transformation to the input language of a model-checker

Set of properties

What properties to be checked?

• Structural properties, reachability

• Derived from requirements

Requires in-depth system knowledge and knowledge in formal languages(e.g. LTL) hard!

C O

R P

OR

A T

E

T E

C H

N O

L O

G Y


Example: Verification of the PROFIsafe Protocol

Profibus DP

Standard-Host/PLC

Repeater

Standard-I/O

Master-SlaveAssignment

Engineering Tool

PG/ES withsecure accesse.g. Firewall

TCP/IP

F = Failsafe

F-Gate-way

otherSafety-

BusF-Sensor F-Field-

Device

DP/PA

F-Actuator

Peer Slave F Communication

F-Actuator

Coexistence of standard and failsafe communication

F-Host/FPLC

Standard-I/O

F-I/O

Emergencypush buttons

C O

R P

OR

A T

E

T E

C H

N O

L O

G Y


PROFIsafe Protocol Architecture

F-HostF-Host F-InputSlave

F-InputSlave

Host Application

Process

Host Application

Process

Slave Application

Process

Slave Application

Process

Grey Channel

FailsafeCommunication

C O

R P

OR

A T

E

T E

C H

N O

L O

G Y


PROFIsafe Modeling Approach

3 await slave ack

4 slave ack check

2 message prepare

5 message prepare

6 await slave ack

ifhost CRCor slave timeoutor slave CRC/cons.Nr.or not operator ack.thenstore faults,x=x+1, use FV

if ack. received with cons.Nr.=0 and not hosttimeoutthenrestart host- timer

7 slave ack check

if ack. received with old cons.Nr. and not host timeout

if messagepreparedthen send

10 slave ack check

9 await slave ack

8 message prepare

if host timeoutthenstore fault,x=x+1, use FV,restart host- timer

if not faults and operator ack.then reset stored faults,old cons.Nr. = x, x=x+1, if slave FV activated or ipar then use FV else use PV

if host CRC or host cons.Nr. or slave timeout or slave CRC/cons.Nr.then store faults, x=x+1, use FV

if host timeoutthenx=x+1, use FV

if message preparedthen send

ifhost CRCor slave timeoutor slave CRC/cons.Nr.thenstore faults,x=x+1, use FV,restart host- timer

if host timeoutthenstore fault,x=x+1, use FV,restart host- timer

if message preparedthen send

if not stored faults before system startthen x=0, use FV

1 system start

if ack. received not with old cons.Nr and not host timeoutthen restart host- timer

if ack. received with cons.Nr.=x and not host timeoutthen restart host- timer

if stored faults before/during system startthen x=1, use FV

if not faultsthenold cons.Nr. = x, x=x+1, if slave FV activated oripar then use FV else use PV

if not faultsthenx=x if slave FV activated or ipar then use FV else use PV

if wait delay timethen store fault, restart host- timer

11 wait delay time

parametrization ok configuration ok initial values = 0 restart host-timer

1. PROFIsafeStatechart

in_cons_num == out_cons_num &&in_CRC == ok &&in_ps_status_bit3_TO == 0 &&in_ps_status_bit2_CRCNO == 0

]{old_cons_num = out_cons_num;inc_cons_num;use_FV_slave;

}

2. RefinedStatechart

3. MC Input Model(in-house MC used)

C O

R P

OR

A T

E

T E

C H

N O

L O

G Y


PROFIsafe Properties

Structural properties

Absence of livelocks and deadlocks

External inputs can be handled in all states

Deterministic behavior

Specific properties

F-host activates FV after Timeout

F-host activates FV after CRC Fault

F-slave activates FV after Timeout

F-slave activates FV after CRC Fault

Possible improvements

Check CRC in initial operation of F-host

Clarify how F-slave should handle status bits 2 and 3

Fault identified

C O

R P

OR

A T

E

T E

C H

N O

L O

G Y


Correct Model – What Next?

Generation of production code from verified model

How feasible is the approach?

Automatic code generation is still a challenge, e.g. in the embedded domain on special hardware

Verified model mostly domain-specific, i.e. no general-purpose code generator

Efficiency of auto-generated code?

Generated code must run in an unknown, i.e. not verified, environmentIf environment is unknown, how sure can one be that the verification results

are preserved?

Product certification does not like auto-code generation

Is there an alternative to production code generation?Yes, generation of test code, model-based testing!

C O

R P

OR

A T

E

T E

C H

N O

L O

G Y


Beware of bugs in the above code;

I have only proved it correct, not tried it.[Donald E. Knuth, 1977]

Proving vs. Testing

C O

R P

OR

A T

E

T E

C H

N O

L O

G Y


PROFIsafe Test Case Generation(Fault Removal)

Structuraltest coverage

criterion

Model ofF-Slave

F-SlaveImplementation

Test Sequence #1

1. rcv[pv][0][0][nok]2. resume3. send[1][0][1][0]4. rcv[pv][0][1][ok]5. resume6. send[1][0][1][1]7. rcv[pv][0][2][ok]8. resume9. send[1][0][0][2]10. rcv[pv][0][3][ok]11. resume12. send[1][0][0][3]13. rcv[pv][0][4][ok]14. resume15. send[0][0][0][4]16. ...

Test Sequence #1

1. rcv[pv][0][0][nok]2. resume3. send[1][0][1][0]4. rcv[pv][0][1][ok]5. resume6. send[1][0][1][1]7. rcv[pv][0][2][ok]8. resume9. send[1][0][0][2]10. rcv[pv][0][3][ok]11. resume12. send[1][0][0][3]13. rcv[pv][0][4][ok]14. resume15. send[0][0][0][4]16. ...

Test Case Generation

Test Case Generation

Test Run

Test Run

C O

R P

OR

A T

E

T E

C H

N O

L O

G Y


PROFIsafe Project Results

Test cases

Serve as compliance checks for PROFIsafe product suppliers

Identified errors in PROFIsafe reference implementation

Accepted by TÜV Süddeutschland for certification

Formal verification

Helps identify ambiguities in system requirements and the design model

Supports a clean documentation of the design

Requires close communication link between domain experts and verification experts

Still an expensive approach

C O

R P

OR

A T

E

T E

C H

N O

L O

G Y


Model-Based Testing – Evolution

Design and test process

Model-driven development vs.

Test-driven development

Ingredients of MBT

Formal model, e.g.UML + semantics

Test generation algorithm

Coverage criterion

Our focus

MBT approaches based on test models mainly

UML based techniques, U2TP

DesignModel

TestModel

System

Requirementsdesigning

codi

ng

testing

desi

gnin

g

reve

rse

engi

neer

ing

Con

form

?

test

ing

C O

R P

OR

A T

E

T E

C H

N O

L O

G Y


5.

Executing/Verifying Test Scripts

Flow of Events

Flow of Events

Business Rules

Business Rules

Use Case Specification

Use Case Specification

System Specification

3.

Test Scripts

2.

Tester

Annotations

Test Executor

4. Capturing Snipplets

Application Feedback

Problem in the model

Problem in the system

Developers

6.

TDE/UML

UpdateEncounterCancelEncounter

<<include>>

Registrar

PrintArtifacts

<<include>>

UML Models

1.

UML Editor Kit

Capture/

Replay Tool

G

U

ISUT

Successful Validation

Test Management

Test Generation: The TDE/UML Tool – Workflow

C O

R P

OR

A T

E

T E

C H

N O

L O

G Y


Modeling in TDE/UML – The MS Word Example

Modeling of a GUI pop-up window…

… in a UML activity diagram + annotations

C O

R P

OR

A T

E

T E

C H

N O

L O

G Y


TDE/UML – The NYC Subway Project

Project at Siemens Transportation (TS)

Deployment of a MBT approach in system testing of the NYC Subway Railcom project

Contribution

Modeling of about 300 system requirements in UML in Rational Rose

Generate system tests in IEEE 829 format

Creating and running executable test scripts in Rational XDE Tester

Benefits to the customer

Currently about 130 test cases generated (about 200 expected)

Modeling helps uncover incomplete and/or inconsistent specifications

Cost of maintenance is reduced due to a systematic and repeatable test approach

C O

R P

OR

A T

E

T E

C H

N O

L O

G Y


The NYC Railcom Project – TDE/UML Approach

TDE/UML

Test Generator

Coverage: round-trip criteria

Refinement: all refinements

Data variations: all choices

Rational Rose Plug-Infor Modeling

GeneratedTest Script

Statistics abouttest generation

C O

R P

OR

A T

E

T E

C H

N O

L O

G Y


TDE/UML Summary

Supports a model-based approach for testing

Generation of test cases for manual and/or automatic execution

Current projects demonstrate the usefulness of the approach

Decreased effort for test maintenance

Increased notion of requirement coverage during test creation/execution

Decreased overall time for test creation

Model-based testing must be introduced as a service

Models are domain dependent

Transformation from informal requirements to formal models requires experts

Lower entry level required compared to formal verification

C O

R P

OR

A T

E

T E

C H

N O

L O

G Y


How to Improve the Penetration of Formal Methods?

Reward

Effort

[source: J. Rushby, 2004]

“Invisible” formal methods may offer high rewards at low/moderate efforts

C O

R P

OR

A T

E

T E

C H

N O

L O

G Y


Invisible Formal Methods in Practice – Further Work

Some new commercial tools deploy this principle

Support of extended source code analysis

• Polyspace

• Intel Thread Checker

Support of test case generation

• Reactis (for Simulink Stateflow)

Other (similar) projects in this context at Siemens CT SE

Fault diagnosis from communication traces of a distributed system using formal verification (SPIN)

• Model reconstruction from traces

• Library of predefined properties (application dependent, e.g. UTRAN)

Validation of (manually derived) test suites, e.g. quality, coverage

approaches to improve system dependability – from formal...

Documents