static and dynamic analysis at jpl

Static and Dynamic Analysisat JPL

Klaus Havelund

2

Mars Science Laboratory (MSL)

• planned launch 2011

• biggest rover so far to be sent to Mars

• programmer team of 30

• testing team of 10+ people

• programming language is C, 3 M LOC

• highly multi-threaded (over 160 threads)

3

program

input output

specification and programmingLet’s see … a command should always succeed …

4

program

specification

input output

specification and programming

formal

5


program

specification

relationship

input output

formal

6


program

specification

refinement

input output

code generation:-from state machines

-graphical-textual

-from data formats (XML)

formal

7


program

specification

static analysis

input output

formal

abstraction

[](request -> <>response)model checking

8


program

specification

static analysis

input output

Good practice analysisusing commercial staticanalyzers.

formal

9

a checkable

Java coding standard

10

why a standard?two perspectives

• more reliable code

• code that is easier to read, leading to– code that is easier to develop– code that is easier to maintain– code that is easier to share

style

11

the choices

• industry standard

• organizational standard

• project standard

• personal standard

• no standard

good

bad

12

basic questions1. do we need to agree on a standard?1. do we need to agree on a standard?

carefully designed?carefully designed?

lots of rules, which can be turned on/off?lots of rules, which can be turned on/off?or

2. should it be 100% checkable?2. should it be 100% checkable?

3. do we care about naming, style, doc?3. do we care about naming, style, doc?

4. how many rules?4. how many rules?

13

C coding standard

17

free tools

coding standard checkers

coding error checker

21

program

specification

dynamic analysis

input output


formal

model-based testing

23

model-based testing

do:: mkdir dir:: cd dir:: rm file:: ..…od

SPIN

abstract state abstract

realfile system

referencefile system

http://spinroot.com/swarm/index.html

24

program

specification

runtime verification

input output


formal

tool-based log file analysis

26

problems with testing FSW

• flight software engineers work under tight schedules: hard to access.

• system = hardware + software: it is cumbersome to run.

• difficult to determine what events to monitor.

.h.h

.h.h.c.c

A B

27

separation of concerns

loglog

loglog

loglog

loglog

28

architecture

loglog LogMaker

[e1,e2,…,en]

LogScope specspec

violationsviolations

29

command execution

command dispatch success

dispatchfailure

failure

30

...COMMAND 7308 { Args := ['CLEAR_RELAY_PYRO_STATUS'] Time := 51708322925696 Stem := "POWER_HOUSEKEEPING" Number := "4" type := "FSW”}

EVR 7309 { message := "Dispatched immediate command POWER_HOUSEKEEPING: number=4, seconds=789006392, subseconds=1073741824." Dispatch := "POWER_HOUSEKEEPING" Time := 51708322925696 name := "CMD_DISPATCH" level := "COMMAND" Number := "4”}...EVR 7311 { name := "POWER_SEND_REQUEST" Time := 51708322925696 message := "power_queue_card_request- sending request to PAM 0." level := "DIAGNOSTIC”}

EVR 7312 { message := "Successfully completed command POWER_HOUSEKEEPING: number=4." Success := "POWER_HOUSEKEEPING" Time := 51708322944128 name := "CMD_COMPLETED_SUCCESS" level := "COMMAND" Number := "4”}

EVR 7313 { name := "PWR_REQUEST_CALLBACK" Time := 51708322944128 message := "power_card_request - FPGA request successfully sent to RPAM A." level := "DIAGNOSTIC”}

CHANNEL 7314 { channelId := "PWR-3049" DNChange := 67 dnUnsignedValue := 1600 type := "UNSIGNED_INT" Time := 51708323217408 ChannelName := "PWR-BCB1-AMP”}...COMMAND 9626 { Args := ['set_device(1)', 'TRUE'] Time := 51708372934400 Stem := "RUN_COMMAND" Number := "18" type := "FSW”}

EVR 9627 { message := "Validation failed for command RUN_COMMAND: number=18." DispatchFailure := "RUN_COMMAND" Time := 51708372934499 name := "CMD_DISPATCH_VALIDATION_FAILURE" level := "COMMAND" Number := "18”}...

example log?

31







example log?

32







example log

?

33

specification languagesfor trace analysis

• programming languages (Python at JPL)

• state machines

• regular expressions

• temporal logic

• grammars

most commonly used “formal” trace logics

34

the first scripture

look:DRILL_DMP\ evr(CMD_DISPATCH,positive)\ evr(CMD_COMPLETED_SUCCCESS,positive)\ evr(CMD_COMPLETED_FAILURE,negative)\ chan(id:CMD-0004,positive,contains opcode of last immediate command)\ chan(id:CMD-0007,positive)\ chan(id:CMD-0001,negative)\ chan(id:CMD-0009,negative)\ prod(name:DrillAll,1,*)

triggerconsequences

35

property P1

P1: Whenever a flight software command is issued, then eventually an EVR should indicate success of that command

P1: Whenever a flight software command is issued, then eventually an EVR should indicate success of that command

36

recall logCOMMAND 7308 { Args := ['CLEAR_RELAY_PYRO_STATUS'] Time := 51708322925696 Stem := "POWER_HOUSEKEEPING" Number := "4" type := "FSW”}


.

.

.

37

COMMAND 7308 { Args := ['CLEAR_RELAY_PYRO_STATUS'] Time := 51708322925696 Stem := "POWER_HOUSEKEEPING" Number := "4" type := "FSW”}


recall log

.

.

.

P1: Whenever a COMMAND is issued with the Type field having the value "FSW”, the Stem field (command name) having some unknown value x, and the Number field having some unknown value y, then eventually an EVR should occur, with the field Success mapped to x and the Number field mapped to y.

P1: Whenever a COMMAND is issued with the Type field having the value "FSW”, the Stem field (command name) having some unknown value x, and the Number field having some unknown value y, then eventually an EVR should occur, with the field Success mapped to x and the Number field mapped to y.

38

COMMAND 7308 { Args := ['CLEAR_RELAY_PYRO_STATUS'] Time := 51708322925696 Stem := "POWER_HOUSEKEEPING" Number := "4" type := "FSW”}


recall log

.

.

.

pattern P1: COMMAND{Type:"FSW", Stem:x, Number:y} => EVR{Success:x, Number:y}

39

pattern P1: COMMAND{Type:"FSW", Stem:x, Number:y} => EVR{Success:x, Number:y}

40

pattern syntax

pattern ::= 'pattern' NAME ':' event '=>' consequence

consequence ::= event | '!' event | '[' consequence1,...,consequencen ']’ | ‘{' consequence1,...,consequencen ‘}'

41

pattern P2: COMMAND{Type:"FSW", Stem:x, Number:y} => ! EVR{Failure:x, Number:y}

P2: Whenever a COMMAND is issued with the Type field having the value "FSW”, the Stem field (command name) having some unknown value x, and the Number field having some unknown value y, Then an EVR should thereafter not occur, with the field Failure mapped to x and the Number field mapped to y.

P2: Whenever a COMMAND is issued with the Type field having the value "FSW”, the Stem field (command name) having some unknown value x, and the Number field having some unknown value y, Then an EVR should thereafter not occur, with the field Failure mapped to x and the Number field mapped to y.

property P2

42

pattern syntax

pattern ::= 'pattern' NAME ':' event '=>' consequence

consequence ::= event | '!' event | '[' consequence1,...,consequencen ']’ | ‘{' consequence1,...,consequencen ‘}'

43

property P3

P3: Whenever a flight software command is issued, there should follow a dispatch and then exactly one success. No dispatch failure before the dispatch, and no failure between dispatch and success.

P3: Whenever a flight software command is issued, there should follow a dispatch and then exactly one success. No dispatch failure before the dispatch, and no failure between dispatch and success.

44

formalization

pattern P3: COMMAND{Type:"FSW", Stem:x, Number:y} => [ ! EVR{DispatchFailure:x, Number:y}, EVR{Dispatch:x, Number:y}, ! EVR{Failure:x, Number:y}, EVR{Success:x, Number:y}, ! EVR{Success:x, Number:y} ]

45

expressed in first order LTL

46

Python predicate definitions

{:def within(t1,t2,max): return (t2-t1) <= max:}

pattern P6: COMMAND{Type:"FSW",Stem:x,Number:y,Time:t1} where {: x.startswith("PWR_”) :} => EVR{Success:x, Number:y, Time:t2} where within(t1,t2,10000)

47

scoped version of P4

pattern P9: COMMAND{Type:"FSW", Stem:x, Number:y} => { EVR{Dispatch:x, Number:y}, [ EVR{Success:x, Number:y}, ! EVR{Success:x, Number:y} ], ! EVR{DispatchFailure:x, Number:y}, ! EVR{Failure:x, Number:y} } upto COMMAND{Type: "FSW"}

48

from patterns to automata

• temporal patterns are translated into parameterized universal automata

• automaton language more expressive

• user can use both, in practice only temporal patterns have been used for testing MSL

49

recall P3

pattern P3: COMMAND{Type:"FSW", Stem:x, Number:y} => [ ! EVR{DispatchFailure:x, Number:y}, EVR{Dispatch:x, Number:y}, ! EVR{Failure:x, Number:y}, EVR{Success:x, Number:y}, ! EVR{Success:x, Number:y} ]

50

automaton A_P3 { always S1 { COMMAND{Type:"FSW",Stem:x,Number:y} => S2(x,y) }

hot state S2(x,y) { EVR{DispatchFailure:x, Number:y} => error EVR{Dispatch:x, Number:y} => S3(x,y) }

hot state S3(x,y) { EVR{Failure:x, Number:y} => error EVR{Success:x, Number:y} => S4(x,y) }

state S4(x,y) { EVR{Success:x, Number:y} => error }}

52







recall log

53

log = [ { "OBJ_TYPE" : "COMMAND", "Args" : ['CLEAR_RELAY_PYRO_STATUS'], "Time" : 51708322925696, "Stem" : "POWER_HOUSEKEEPING", "Number" : "4", "Type" : "FSW" }, … { "OBJ_TYPE" : "EVR", "name" : "PWR_REQUEST_CALLBACK", "Time" : 51708322944128, "message" : "power_card_request -\ FPGA request successfully sent to\ RPAM A.", "level" : "DIAGNOSTIC" }, … ]

54

running LogScope

import logscope

log = extractLog(…)

obs = logscope.Observer("specs/rv-tutorial")

obs.monitor(log)

56

specification learning

• writing specs is time consuming

• often hard come up with properties

• one approach is to use already generated log files to “get ideas”

• in the extreme case, specifications can be automatically generated from log files

57

architecture

learner monitor

logs

spec

yes

no:……

spec

58

learner APIimport logscope

log1 = … ; log2 = … ;learner = logscope.ConcreteLearner(“P”) learner.learnlog(log1) learner.learnlog(log2) learner.dumpSpec(sfile)

log3 = … ; learner = logscope.ConcreteLearner(“P”,sfile) learner.learnlog(log3) learner.dumpSpec(sfile)

log4 = … ; obs = logscope.Observer(sfile) obs.monitor(log4)

60

using step and success states

61

conclusions

• use of static analysis is now required for flight missions.

• model-based testing is used by LaRS group to test file system – for real.

• log analysis support shows promise.• what’s next?: runtime verification, interaction

between static and dynamic analysis, specification learning, trace visualization.

62

monitor the proof obligations

static analyzer

combining static and dynamic analysis

program + spec

proof obligations

63

RMOR

monitor FileMonitor { symbol open = after call(main.c:openfile); symbol read = after call(main.c:readfile); symbol close = after call(main.c:closefile); symbol send = before call(main.c:senddata); symbol end = before call(main.c:finish);

state closed {when open -> unread;} state unread {when read -> read;} state read {when send => error;} super opened [unread,read] { when end => error; when close -> closed; }}

AspectC likeinstrumentation

state machinespecification

64

thanks for listening!

static and dynamic analysis at jpl

Documents