Presentation at DVClub May 24th 2016

Product Overview

Automatic, Validated Debug

14%

22%

24%

37% 3%

Test Planning

Testbench Development

Creating Test and RunningSimulationDebug

We Can Gain a Lot! Debug is the Largest Verification Task!

2 Source: Wilson Research Group and Harry Foster at Mentor Graphics, 2014 Functional Verification Study

Most time of a project is spent in verification

Verification Tasks

Year Mean Peak Engineers/Project

%Time in Verification

2007 13 46%

2012 17 56%

2014 21 57%

More time spent in debug than any other

verification task

Evolution of Debug

3

Small Designs

Larger Designs

Random Testing

Automated Testing

Automated Debug

1995

2000

2010

PinDown

Man hours spent debugging

Manual Debug

4

1. Analyze 2. Modify 3. Validate

The debug analysis must be validated. The relevant test should pass.

Automatic Debug

5

1. Analyze 2. Modify 3. Validate

The Laws of Automatic Debug The search space must be limited

The debug analysis must be validated

I am stupid. I can only handle a

limited search space

PinDown - Automatic Debugger of Regression Failures

6

Limited Search Space Identified

Validated!

Step 1. Test FailsFails

4 5 6 7 8 9

Latest

Step 2. Run Test on Older RevisionsFails

4 5 6 7 8 9

Latest

Step 3. Remove Commit and Validate

Blame &

Remove

Still Fails

4 5 6 7 8 9

Latest

Step 4. Remove Commit and Validate

Blame &

Remove

Passes

4 5 6 7 8 9

Latest

7

Tests FAIL

Debugging with PinDown

Normal approach, without PinDown

• Manual • Iterative • Uncertain

Validated bug report • Faulty code

lines • Committer • Validation

• Automated • Direct • Validated

With PinDown

Tests FAIL

With PinDown

8

Tests

FAIL

”OK, fixed it”

PinDown Report

PinDown™ - Automatic Test & Debug System

Version Control System

PinDown Results Database

Test Executor (LSF Farm)

10

PinDown TestHub

Email Bug Reports

Automatic debug costs some extra test time on the farm, but reduces retesting of the same tests with the same status multiple times as bugs are fixed faster

Use The Farm Efficiently

10

Debug (10 tests, parallel)

200 tests (200h run time) - 10 failures

200 tests - all pass

Bug 1 after 2h

Bug 2 after 5h

Bug 1 fixed

Bug 2 fixed

11

PinDown Patented Debug Algorithm

Test Results

3. Unroll Changes - 10 selections in parallel

4. Verify

The Rest

1-4 times

* New in PinDown 4

Bug Group 1:

”Error: RX value wrong”

12 test failures

Bug Group 2:

”Address is XXXXXXX”

14 test failures

Fastest Test

Fastest Test

1. Grouping

2. Smart

Selection

Identify

possible bad

commits,

files*, chunks*

or lines*

Revision Control System

(e.g. Perforce, GIT,

Mercurial, Subversion,

Clearcase)

Test passes

No pass

Bug No: 1 (new bug)

Test: test34_vm_seed_0x45679832

Build: dtlb_thdr

Error: Error: RX value wrong

Committer: johndoe

Commit Message:

979301: RX performance improvement

Committed Files:

//depot/bigproject/dv/dtlb/dtlb_rqe/common/interface/dtlb_rqe_tb_if.sv

Log:

/projects/pindown/bigproject/dv/dtlb/tb/ftlb_thdr/sv_sep_dut_comp.log

Next Bug

*

12

The Principle is Simple

...but the reality is difficult

41%

41% incorrect bug reports with simple approach

PinDown supports all scenarios

Customer Case: Bugs Fixed 400% Faster, 5x Less Discussion

13

With PinDown the problems are discussed less and bugs are fixed faster

Customer Case: Large ASIC IP project (40 people, 1 year)

Time saving

per bug

(17.3h/bug)

Time to correct bugs 75% shorter with PinDown...

x

Number of

bugs

11% shorter total

project leadtime!

• Faster time-to-market

• Major cost savings:

4.9 manyears

Quality secured! ...has a direct impact on the total project leadtime

14

Degree of

blocking

(%)

x

Automatic, Validated Debug - Summary

15

• PinDown provides Automatic, Validated Debug

of Regression Failures

• Fewer manual resources spent on debugging

• Faster Bug Churning => Shorter project

The Laws of Automatic Debug The search space must be limited

The debug analysis must be validated