university of virginia computer science

Automatically Inferring Temporal Properties for Program Evolution

Jinlin Yang and David Evans15th IEEE International Symposium on Software Reliability Engineering 5 November 2004Saint-Malo, FranceUniversity of

VirginiaComputer

Science

www.cs.virginia.edu/ipa 2

Temporal Properties• Constrain ordering of events

– Essential for program correctness (file open before read)

• Reveal important differences between programs: S P not a chef

• But, hard for humans to document correctly [Holzmann’s FSE 2002 keynote]

• Can we infer useful temporal properties automatically?

This talk argues: Yes!


Dynamically Inferring Properties

• Inherently unsound: guessing properties of all executions by looking at a small number

• Value-based invariants– Daikon [Ernst, TSE, Feb 01]

• Temporal properties– Mining specification [Ammons, POPL 02]– Extracting component interfaces [Whaley, ISSTA

02]• We focus on relationship between 2 or 3

events: automation, scalability


System Overview

Inferred Properties 1

DifferentProperties

Diff

erence Analyzer


SharedPropertiesProgram

Version 1

Program Version 2

Dynamic

Inference

… …


Property Inference


DifferentProperties

Diff

erence Analyzer


SharedPropertiesProgram

Version 1

Program Version 2

Dynamic

Inference

… …

Program InstrumentedProgram

Instrumentatio

n

Test Suite

Execution Traces

Testing

Inferred Properties

CandidatePropertyPatterns

Inference


Candidate Property Patterns

• Response pattern [Dwyer, ICSE ‘99]

[-P]* (P [-S]* S [-P]*)*– Filtered version (all non P/S events

removed): S*(P+S+)*SPPSPS SPSP

• Too weak for our purposes


Partial Order of Patterns

MultiEffect

(PS+)*

MultiCause

(P+S)*

EffectFirst

S*(PS)*

Alternating

(PS)*

OneCause

S*(PS+)*

OneEffect

S*(P+S)*

CauseFirst

(P+S+)*

ResponseS*(P+S+)*

Stri

cte

r

ΛΛΛ

Λ

3 Primitive

Patterns4 DerivedPatterns


Inferring Properties• For all pairs of events,

P and S:– Check which (if any) of

OneCause, CauseFirst and OneEffect are satisfied

– Infer the strictest property satisfied if more than one primitive is

MultiEffect(PS+)*

MultiCause

(P+S)*

EffectFirst

S*(PS)*

Alternating

(PS)*

OneCauseS*(PS+)*

OneEffect

S*(P+S)*

CauseFirst(P+S+)*

ΛΛΛ

Λ


Traces Example

Trace 1: PSPS Trace 2: PPS All TracesCauseFirs

t(P+S+)*

OneCauseS*(PS+)*

OneEffectS*(P+S)*

For any two events, determine the strictest pattern satisfied by all traces


Traces Example


t(P+S+)*

OneCauseS*(PS+)* OneEffectS*(P+S)*



Traces Example


t(P+S+)*

OneCauseS*(PS+)* OneEffectS*(P+S)*


CauseFirst OneEffect MultiCause


Implementation

Program InstrumentedProgram

Instrumentatio

n

Test Suite

Execution Traces

Testing

Inferred Properties

CandidatePropertyPatterns

Inference

Automated for Java using JRat Method entry/exit eventsManually for C code or output (today’s examples)

Prototype (described in paper) 900 lines of PerlNew implementation 8K lines of Java Improved performance Analyses, ranking heuristics


Experiments• Hypotheses

– We can automatically extract interesting temporal properties

– Differences in inferred temporal properties among multiple versions can reveal interesting things

• Target programs– Tour bus simulator (8 student submissions)– OpenSSL (0.9.6, 0.9.7-0.9.7d)


Tour Bus Simulator• Bus and each passenger

are a separate thread• Assignment in Fall 2003

graduate-level course (before we started this project)

• 8 submissions from the instructor (all believed to be correct)


Testing

> cville_bus –N 2 –C 1 –T 2Bus waiting for trip 1Passenger 0 boardsBus drives around CharlottesvillePassenger 0 exitsBus waiting for trip 2Passenger 1 boardsBus drives around CharlottesvillePassenger 1 exitsBus stops for the day

N, the number of peopleC, the capacity of the busT, the number of trips

Executed each submission with 100 randomly generated inputs, where

20 < C ≤ 40C+1 ≤ N ≤ 2C1 ≤ T ≤ 10

Event traces extracted fromprogram output (no need to instrument program)


Differences Reveal Problems

Pattern 7 Correct Versions

1 Faulty Version

Alternating waitdrives

MultiEffect

drivesboardswaitexitswaitboards

waitdrives

waitboardsMultiCause boardsdrives boardsdrives

CauseFirst

boardsexits boardsexitsdrivesexitswaitexits

– waitdrives not Alternating• Bus drives around before all passengers exit• Bug in locking code (misplaced synchronization)

– drivesexits, waitexits not MultiEffect• Bus drives around twice before letting passengers exit• Missing synchronization


OpenSSL• Widely used implementation of the

Secure Socket Layer protocol• 6 versions [0.9.6, 0.9.7, 0.9.7a-d]

between Sept 2000 and March 2004• We focus on the handshake

protocol– 38 different event types


SR_CLNT_HELLO SW_SRVR_HELLO

SW_CERT SW_KEY_EXCH SW_CERT_REQ

SW_SRVR_DONE

SR_CERT SR_KEY_EXCH

SR_CERT_VRFY SR_FINISHED

SW_CHANGE SW_FINISHED

BEFORE+ACCEPT

SW_FLUSH OK

SW_FLUSH

Client Server


Testing• Manually instrumented server to

record handshake events• Executed each version of server

with 1000 randomly generated clients– Client modified advance to a

randomly selected state with 5% probability


Inferred Alternating Patterns

0.9.6 0.9.7 0.9.7a

0.9.7b

0.9.7c

0.9.7d

SR_KEY_EXCHSR_CERT_VRFY

SW_CERTSW_KEY_EXCH

SW_SRVR_DONESR_CERT

Documented change:

ignore unrequeste

d client certificates

Fixed bug

causing server

crashes

Race condition

(present in all

versions)7 alternating patterns same for all versions


Partitioning TracesAll Traces

Correct Clients (never jump to random state) Faulty Clients

No Server Error Server Error

Follows SSL specification

Missing Alternating patterns:allows handshake cycle Segmentation fault

(<0.9.7d)


Performance• Prototype implementation (described in paper)

– Up to 3-4 minutes for examples (Pentium 4, 3GHz, 1G RAM)

– Too slow (several days) for larger programs– Scales approximately as trace length events2

• New implementation– Scales approximately as trace length events– Analyze these experiments in < .5s

• Bus: 102 events, 100 traces, 222 events/trace: 0.4s• OpenSSL: 35 events, 1000 traces, 18 events/trace: 0.3s

– Feasible to analyze large systems• Thousands of different events• Traces with millions of events• Preliminary results from experiments with JBoss


Summary of Experiments• Useful in program evolution

– Reveal interesting changes in OpenSSL– Identify unexpected differences

• Revealed bugs– Tour bus: identified faulty implementation

• Multiple implementations are rare, but multiple representations are common (i.e. design, model, code)

– OpenSSL: • Differences between versions revealed• Flaws found by partitioning traces


Future Work/Research Questions

• Can we make this feasible in practice?– Automatically identify interesting events– Heuristics to identify important patterns– Scalability and automation

• Can we make it more useful?– More expressive patterns: more events,

combine with data-flow– Understand impact of different testing

strategies


Conclusion• Automatically inferring temporal

properties is feasible

• Even very simple property patterns reveal interesting program properties


Questions?http://www.cs.virginia.edu/ipa

This work is funded in part by the National Science Foundation.

Thanks: Marty Humphrey for providing the student submissions, Chengdu Huang for help with OpenSSL, Joel Winstead, the anonymous reviewers, and anonymous CS650 students for writing buggy code.

university of virginia computer science

Documents