nsse-2011-abanerjee-automateddebugging

8/7/2019 NSSE-2011-ABanerjee-AutomatedDebugging

1/76

Automated Debugging

Ansuman Banerjee

Indian Statistical Institute


2/76

Outline

Introduction Program Analysis Techniques for Debugging

Program SlicingTesting based Fault Localization

Debugging Change-Induced Bugs

Summary


3/76

What is a bug?

1/19/20113

A software bug (or just "bug") is an error, flaw, mistake, in acomputer program that prevents it from behaving as intended(e.g., producing an incorrect result). Reports detailing bugs in aprogram are commonly known as bug reports, fault reports, change requests, and so forth. --- Wikipedia

Even today, debugging remains very much of an art. Much of the

computer science community has largely ignored the debuggingproblem.. over 50 percent of the problems resulted from the time

and space chasm between symptom and root cause or inadequate

debugging tools. -- IBM Sys Jnl

Debugging is a time consuming activity

Need methods / tools to trace back to root cause of bug from the

manifested error


4/76

Software Controlled Devices Everywhere

Automobiles

Handheld

Medical

Airplanes Military

Entertainm
http://www.gsmworld.com/images/3g/nokia2_lrg.jpghttp://www.gsmworld.com/images/3g/nokia2_lrg.jpg


5/76

Do we live in a Bug-free world? Therac 25: Four dead

The European Ariane 5 rocketexplodes 40 s into its maiden flight

Mars Polar Lander lost

Pentium FDIV bug Numerous in Software Hall ofShame
http://../ESE/ariane.mov


6/76


7/767

Facts on Debugging

Software bugs are costing a lot everyyear

Improvements could reduce cost by30%

Validation (including debugging) caneasily take up to 50-75% of thedevelopment time

When debugging, some people arethree times as efficient than others


8/76

Debugging: Some numbers

ACM CrossTalk article Predictive Quality Control with Software Inspections http://www.stsc.hill.af.mil/crosstalk/1994/06/xt94d06e.asp

Real-life case studies to find actual errors in software via veryslow code inspection

Errors are classified as major errors or not.

Outcome = 13 major errors per 1000 LOC. For Windows Vista (~50 Million LOC), approx. 650000 errors !!

Need automation support

Program Analysis
http://www.stsc.hill.af.mil/crosstalk/1994/06/xt94d06e.asphttp://www.stsc.hill.af.mil/crosstalk/1994/06/xt94d06e.asp


9/76

Software Debugging

Typical Debugging Steps

1. Hypothesizethe cause of

the error

2. Try to confirm it

Observable

error

Error?

Error?

ExecutionError? D

C

B

A


10/7610

int main(int argc, char *argv[]){

int *a;int i;

a = (int *)malloc((argc - 1) * sizeof(int));for (i = 0; i < argc - 1; i++)

a[i] = atoi(argv[i + 1]);

shell_sort(a, argc);

printf("Output: ");

for (i = 0; i < argc - 1; i++)printf("%d ", a[i]);printf("\n");

free(a);

return 0;

A Sample Program


11/7611

A Sample Run

sample 9 8 7$

Output: 7 8 9

sample 11 14$

Output: 0 11


12/7612

How to Debug(Sommerville 2004)

Locateerror

Designerror repair

Repairerror

Re-testprogram


13/7613

The Traffic Principle

T

R

AF

F

I

C

rack the problem

eproduce

utomateind Origins

ocus

solate

orrect


14/76

14


T

R

AF

F

I

C

rack the problem

eproduce

utomateind Origins

ocus

solate

orrect


15/76

15

1. The programmer creates adefect an error in thecode.

2. When executed, the defect

creates an infection anerror in the state.

3. The infection propagates.

4. The infection causes afailure.

From Defect to Failure

Variables

This infection chain must betraced back and broken.

t


16/76

16

Every failurecan betraced back to some

infection, and everyinfectionis caused bysome defect.

Debuggingmeans torelate a given failure tothe defect and toremove the defect.

Debugging

Variables


17/76

17

Search in Space + Time


18/76

18

The Defect


19/76

19

A Program State


20/76

20

A Sample Program

sample 9 8 7$

Output: 7 8 9

sample 11 14$

Output: 0 11


21/76

21

int main(int argc, char *argv[]){

int *a;int i;

a = (int *)malloc((argc - 1) * sizeof(int));for (i = 0; i < argc - 1; i++)


shell_sort(a, argc);

printf("Output: ");for (i = 0; i < argc - 1; i++)

printf("%d ", a[i]);printf("\n");

free(a);

return 0;}


22/76

22

Find Origins

The 0 printed is thevalue of a[0]. Wheredoes it come from?

Basic idea: Track ordeduce value origins

Separates relevantfrom irrelevantvalues

We can trace back a[0]to shell_sort


23/76

23

static void shell_sort(int a[], int size){

int i, j;int h = 1;

do {h = h * 3 + 1;

} while (h = h && a[j - h] > v; j -= h)

a[j] = a[j - h];if (i != j)

a[j] = v;

}} while (h != 1);}


24/76

24

Search in Time

In shell_sort, thestate must havebecome infected.

Basic idea:Observe a

transition fromsane to infected.


25/76

25

Observing a Run


26/76

26

Specific Observationstatic void shell_sort(int a[], int size){

int i, j;int h = 1;...

}

sample 11 14$

a[0] = 11a[1] = 14a[2] = 0size = 3Output: 0 11

fprintf(stderr, At shell_sort);

for (i = 0; i < size; i++)fprintf(stderr, a[%d] = %d\n, i, a[i]);

fprintf(stderr, size = %d\n, size);

The state is infected at the call of shell_sort!


27/76

27

int main(int argc, char *argv[])

{ int *a;

// Input arraya = (int *)malloc((argc - 1) * sizeof(int));for (int i = 0; i < argc - 1; i++)


// Sort arrayshell_sort(a, argc);

// Output arrayprintf("Output: ");

for (int i = 0; i < argc - 1; i++)printf("%d ", a[i]);

printf("\n");

free(a);return 0;

}

Should be argc - 1

Fixing the Program


28/76

28

Sane stateInfected state

Finding Causes

The differencecauses the failure


29/76

29

Concepts

A failure comes to be in three stages:

1. The programmer creates a defect

2. The defect causes an infection

3. The infection causes a failure --anexternally visible error.

Not every defect results in an infection,and not every infection results in afailure.


30/76

30


T

R

AF

F

I

C

rack the problem

eproduce

utomateind Origins

ocus

solate

orrect


31/76

31

Automated Debugging

A variety of tools and techniquesavailable to automate debugging:

Program Analysis Slicing Observing & Watching Runs

Asserting Invariants Detecting Anomalies

Isolating Cause-Effect Chains


32/76

Outline

Introduction Program Analysis Techniques forDebugging

Program Slicing Testing based Fault Localization

Debugging Change-Induced Bugs Summary


33/76

Program Analysis for Debugging


34/76

Program Analysis for Debugging

What to analyze?

the execution run, Dynamic Analysis

the source code, Static Analysis

How to analyze? Program Slicing

Testing based Fault Localization


35/76

Program Slicing

b=1;

y=1;

If (a>1){if (b>1){

x=2;

}}

printf (%d, x);

1

2

34

5

6


36/76

Program Slicing

b=1;

y=1;

If (a>1){if (b>1){

x=2;

}}

printf (%d, x);

1

2

34

5

6 SlicingCriterion

DataDependence

ControlDependence


37/76

Program Slicing

b=1;

y=1;

If (a>1){if (b>1){

x=2;

}}

printf (%d, x);

1

2

34

5

6 SlicingCriterion

DataDependence

ControlDependence


38/76

Dependence Graph

Statement

Control / Data

Dependence

A

B


39/76

Program Slicing

Statement

Control / Data

Dependence

Slicing Criterion


40/76

Program Slicing

Statement

Control / Data

Dependence

Slicing Criterion


41/76

Static vs Dynamic Slicing

Static Slicing source code

statement static dependence

Dynamic Slicing (useful for debugging) a particular execution statement instance

dynamic dependence


42/76

Use of Dynamic Slicing

Program

Input

Exec. Trace

Output

OKUnexpected,

debug it

Dynamic Slice =Bug Reportcriterion

Debugging

Instrument


43/76

Static vs Dynamic

Slicingb=1;

If (a>1)x=1;

else

x=2;printf (%d, x);

1

23

4

56 Slicing Criterion


44/76

Dynamic Slice - Illustration

0. scanf(%d, &A);

1. If (A == 0) {;2. W = X;

3. U = A;4. }

5. printf(%d\n, U);

Criterion: 5,U with input A = 0

Trace: Slice = {0,1,3,5}

Data dependences encountered 5 -> 3, 1 -> 0

Control dependence encountered 3 -> 1


45/76

Forward and Backward

Slicing

a=1;

x=a;

if (x>1)

y=10;

1

2

3

4

Slicing Criterion


46/76

Backward Slicing

a=1;

x=a;

if (x>1)

y=10;

1

2

3

4


47/76

Forward Slicing

a=1;

x=a;

if (x>1)

y=10;

1

2

3

4


48/76

Slicing: Tool Support

WET (Whole Execution Traces)

http://wet.cs.ucr.edu/

BitBlaze

http://bitblaze.cs.berkeley.edu/


49/76

Outline


Program Slicing Testing based Fault Localization

Debugging Change-Induced Bugs

Summary


50/76

Testing and Debugging

Two very close software developmentactivities

Testing Stage

Debugging Stage

Testing

Methods

Debugging

Methods


51/76

Testing Based Fault Localization

Use testing to help debuggingTest Suite (test cases)

Program

Failing Runs Successful Runs


52/76

Fault Localization

Compare Execution

Failing Run Successful Run

Difference As bug report

Choose

Successful Run Pool

Difference Metric

Testing

Change FailingInputGenerate

T i B d F l


53/76

Testing Based Fault

Localization

Successful RunFailing Run

Difference:

Bug?

T i B d F l


54/76

Testing Based Fault

Localization

What to Compare choice of the Execution Run

How to Compare

Ch i f h E i


55/76

Choice of the Execution

Run Failing run

Select one failing run for comparison

Different failing runs may correspond todifferent error causes

Successful Run A Single successful run, VS A Set of successful runs

Ch i f S f l


56/76

Choice of a Successful

Run The difference can be related to:

different inputs the error

The successful run and the failing run should be assimilar as possible Choose a good successful run for comparison

Ch S f l


57/76

Choose a Successful

Run Comparison of Differences Select onefailing run for comparison

diffdiff

Failing Successful Failing Successful

Ch S f l


58/76

Choose a Successful

Run Comparison of Differences Select one

failing run for comparison

diff

diff 0 at the beginning of the method foo()

Check whether these potential invariants hold in thefailing run being investigated.

Past project in an offering of this course, in fact!

Invariant detector --- Daikon

http://pag.csail.mit.edu/daikon/ The potential invariants are inferred, not specified. The tool, of course, needs to fix a language of potential

invariants.
http://pag.csail.mit.edu/daikon/http://pag.csail.mit.edu/daikon/


61/76

Outline


Program Slicing Testing based Fault Localization Debugging Change-Induced Bugs Summary


62/76

Debugging ChangeInduced Bugs

1 74


63/76

Change-Induced Bugs

Changes may induce bugs in stable programs Changes may be due to several factors

Cost of managing software evolution takes large percentof the total cost of software.

Finding root cause of regression bugs is still a majorheadache in large software development projects.

1 75


64/76

Manual review isnt enough!

1 76

A True Story


65/76

A True Story

Release 4.17 of the GNU debugger GDB brought several new features, languages,and platforms, but for some reason, it no longer integrated properly with thegraphical front-end data display debugger DDD

The arguments specified within DDD were not being passed to the debuggedprogram.

Something changed within GDB such that it no longer worked

Something? Between the 4.16 and 4.17 releases, no less than 178,000 lineschanged.

How can I isolate the change that caused the bug and make GDB work again?

The GDB example is an instance of the worked yesterday, not today problem: afterapplying a set of changes, the program no longer works as it should.


66/76

Research Objective

1

Old StableProgram P

Test Input t

New BuggyProgram P

78


67/76

An experiment we tried

1/19/2011 79

Bug-free distribution:Linux (GNU Core-utils, nettools)

Buggy distribution: BusyBox

Busybox distribution is 121 KLOC.Errors to be root-caused in some common

utilities: arp, top, printf

Development

Changes


68/76

Debugging Busybox

1/19/2011 80

The concept Correct program: GNU Coreutils Changed Version: Busybox

De-facto distribution for embedded devices. Aims for low code size Less checks and more errors.

The practice Large bug report produced using trace comparison

A glimpse inside theTh ARP b


69/76

g pse s de t eARP bug

1 81

-Ainet

Shows all computersconnected to hostwith inet addressfamily

Embedded Linux

GNU Coreutils

Crash

Crash identified as NULL pointer access at crash sitehw_typeunexpectedly set as NULL at crash site

The ARP bug

BusyBox ARP


70/76

y

1 82

int arp_main(int argc, char **argv) {....option_mask32 = getopt32(argc, argv, "A:p:H:t:i:adnDsv, &protocol,

&protocol, &hw_type, &hw_type, &device);if (option_mask32 & ARP_OPT_A || option_mask32 & ARP_OPT_p){

ap = get_aftype(protocol);}if (option_mask32 & ARP_OPT_A || option_mask32 & ARP_OPT_p){

hw = get_hwtype(hw_type);

else hw_set = 0;

if (hw_set==0)if ((hw = get_hwtype(DFLT_HW)) == NULL)

// Error check and exit}

....

const struct hwtype *get_hwtype (const char *name) {

const struct hwtype * const *hwp;hwp = hwtypes;while (*hwp != NULL) {

if (!strcmp((*hwp)->name, name))return (*hwp);

hwp++;}

return NULL;Test Input: -Ainet

Crash sincename is NULL

Slicing Criteria

name == NULL

ARP in GNU core-utils


71/76

1 83

int arp_main(int argc, char **argv) {....option_mask32 = getopt32(argc, argv, "A:p:H:t:i:adnDsv, &protocol,

&protocol, &hw_type, &hw_type, &device);

if (option_mask32 & ARP_OPT_A || option_mask32 & ARP_OPT_p){ap = get_aftype(protocol);

}if (option_mask32 & ARP_OPT_H || option_mask32 & ARP_OPT_p)

hw = get_hwtype(hw_type);else hw_set = 0;

if (hw_set==0)if ((hw = get_hwtype(DFLT_HW)) == NULL)

// Error check and exit} const struct hwtype *get_hwtype (const char *name) {

const struct hwtype * const *hwp; hwp = hwtypes;while (*hwp != NULL) {

if (!strcmp((*hwp)->name, name))return (*hwp);

hwp++;}return NULL;}

Test Input: -Ainet

Slicing Criterianame != NULL

Methodology in action


72/76

Methodology in action

1/19/2011 84

Trace Collection withAinet for

buggy and stable ARP

Identify variable responsiblefor crash and map it to stable

ARP

name == NULL name != NULL

Slice busyBox

trace

Slice net-tools

trace solver

comparison

Map back to source to

Generate bug report

Differingterms


73/76

Acknowledgement: Thanks to AndreasZeller

Slides available from Prof. Zellers homepage Yesterday, my program worked. Today, it does

not. Why?Proc. FSE 99

Automated Debugging: Are We Close?IEEE TC Locating Causes of Program bugs, ICSE 2005

Debugging Debugging, FSE 2009 Keynote


74/76

Ongoing Research Projects

Change-driven test-suite augmentation

Problem: Developing test cases / test plan

for changed features is a manual activity Objective: Automatically creating test

cases for the changes made to an oldprogram

Debugging concurrent programs

1 86


75/76

Ongoing Research Projects

Debugging optimized binaries without source

Source code of the buggy program may not beavailable

Stack trace sent from client site

Microsoft Office crashes

Can we decipher this report

based on registry footprint?

1 87


76/76

Thank you !

nsse-2011-abanerjee-automateddebugging

Documents