tdd and a new paradigm for hardware verification
TRANSCRIPT
© 2011 XtremeEDA USA Corporation - Version 080721.10
TDD And A New Paradigm For Hardware Verification
Neil Johnson XtremeEDA
[email protected] http://www.AgileSoC.com
1
© 2011 XtremeEDA USA Corporation - Version 080721.10
• Neil Johnson – 13 years of hardware design and verification – Altera, Neterion, Flextronics, Nextwave Wireless
• Principal Consultant XtremeEDA – Consulting services
• Verification experts
– Clients are any size and many applications • Telecom, networking, wireless, computer hardware, etc.
– We work remotely or onsite as part a client’s team
About Me
2
© 2011 XtremeEDA USA Corporation - Version 080721.10
• Neil Johnson – 12 years of hardware design and verification – Altera, Neterion, Flextronics, Nextwave Wireless
• Principal Consultant XtremeEDA – Consulting services
• Verification experts
– Clients are any size and many applications • Telecom, networking, wireless, computer hardware, etc.
– We work remotely or onsite as part a client’s team
About Me
3
Hardware engineer that’s developing an increasingly unhealthy infatuation with Test-Driven Development
© 2011 XtremeEDA USA Corporation - Version 080721.10
TDD And A New Paradigm For Hardware Verification
4
The Old Paradigm
What’s Wrong With This Picture?
A New Paradigm
“So... how’s that working out?”
© 2011 XtremeEDA USA Corporation - Version 080721.10
• ASIC – Application Specific Integrated Circuit – Static structure – Digital or mixed signal – High NRE/Low cost
• FPGA – Field Programmable Gate Array – Reprogrammable structure – Primarily digital – No NRE/High cost
• SoC – Either of the above + embedded processor(s) + software
What Do I Mean By Hardware
5
© 2011 XtremeEDA USA Corporation - Version 080721.10
• Typical SoC design flow – Specification – Design – Verification – Physical design – Fabrication – Validation – Integration
SoC Development Basics – Agile2011
6
Pre-silicon
Documentation
Code
“Stuff”
Chip
Board
System
Production
OS
Drivers
Application
You Guys
© 2011 XtremeEDA USA Corporation - Version 080721.10
• Typical SoC design flow – Specification – Design – Verification – Physical design – Fabrication
Hardware Development Basics
7
Pre-silicon
Production
Documentation
Code
“Stuff”
Chip
© 2011 XtremeEDA USA Corporation - Version 080721.10
1. Verify each Block exhaustively in isolation
Hardware Verification in 2 Easy Steps
8
Block A
Block B
Block C
Block D
Block A
Block I
Block H
Block G
Block F
Block E
© 2011 XtremeEDA USA Corporation - Version 080721.10
1. Verify each Block exhaustively in isolation 2. Integrate and verify at Top level
Hardware Verification in 2 Easy Steps
9
Block A
Block B
Block C
Block D
Block A
Block I
Block H
Block G
Block F
Block E
© 2011 XtremeEDA USA Corporation - Version 080721.10
• Simple testbench with lots of “simple” tests – manually created stimulus/manual checking – upside: focus and early results – downside: inadequate coverage
Directed Testing (aka: Old school testing)
10
DUT Generator Monitor
Pro
gres
s
Time
© 2011 XtremeEDA USA Corporation - Version 080721.10
• Complex testbench with fewer “smart” tests – Random stimulus/model based checking – upside: efficiency and rigor – downside: test harness complexity and long delay in results
Constrained Random Testing (aka: Cutting edge testing)
11
Pro
gres
s
Time
DUT Generator Monitor
Model
© 2011 XtremeEDA USA Corporation - Version 080721.10
• Directed testing – simple test harness – upside: focus and early results – downside: inadequate coverage
• Constrained random testing – smarter test harness – upside: efficiency and rigor – downside: test harness complexity and long delay in results
Just to Repeat...
12
© 2011 XtremeEDA USA Corporation - Version 080721.10
Development Timeline
13
Block Level design
Block Level Testbench
Block Level Testing
Top Level Testbench
Top Level Testing
Block Level
Top Level
DONE START
© 2011 XtremeEDA USA Corporation - Version 080721.10
• Long development cycles – 6-12 months is common is common in block testing – Time for testbench development is increasing
• Used to take weeks... now it takes months
• finish the code, then test it • Unit testing is rare
– smoke testing of design – no testing-of-the-testbench
• progress is based on code written
Observations From The Old Paradigm
14
© 2011 XtremeEDA USA Corporation - Version 080721.10
What Are We Doing Wrong?
• Brainstorm... in 5 minutes or less
© 2011 XtremeEDA USA Corporation - Version 080721.10
Big Bang Progress
16
Way Too Much!!
DONE START
© 2011 XtremeEDA USA Corporation - Version 080721.10
Prioritizing Sub-Systems Over Products
17
Block Level design
Block Level Testbench
Top Level Testbench
Block Level
Top Level
Top Level Testing
Block Level Testing
DONE START
No Feedback
© 2011 XtremeEDA USA Corporation - Version 080721.10
Subjective Definition of Done
18
Block Level design
Block Level Testbench
Block Level Testing
Top Level Testbench
Top Level Testing
Block Level
Top Level
START Oops! NOT DONE
© 2011 XtremeEDA USA Corporation - Version 080721.10
Emphasis on Bug Hunting
© 2011 XtremeEDA USA Corporation - Version 080721.10
What Are We Doing Wrong?
• The Old Paradigm – Big bang progress – Prioritizing block level over product level – Subjective definition of done – Emphasis on Bug hunting
© 2011 XtremeEDA USA Corporation - Version 080721.10
How Do We Change?
• Brainstorm... in 5 minutes or less
© 2011 XtremeEDA USA Corporation - Version 080721.10
Incremental Progress
DONE
Incremental Progress START
© 2011 XtremeEDA USA Corporation - Version 080721.10
Prioritize The Product
Block Level
Top Level
DONE START
Learn From The Product Much Earlier
© 2011 XtremeEDA USA Corporation - Version 080721.10
Redefined Definition of Done
Block Level
Top Level
DONE START
DO
NE
DO
NE
DO
NE
DO
NE
DO
NE
DO
NE
DO
NE
© 2011 XtremeEDA USA Corporation - Version 080721.10
Bug Prevention with TDD
Block Level
Top Level
Design/Testbench Code
Unit Level
DONE
Unit Tests Design
START
© 2011 XtremeEDA USA Corporation - Version 080721.10
How Do We Change?
• The New Paradigm – Incremental progress – Prioritize the top-level (early integration) – Redefinition of the definition of done – Bug prevention with TDD
© 2011 XtremeEDA USA Corporation - Version 080721.10
“So... How’s It Working Out?”
27
© 2011 XtremeEDA USA Corporation - Version 080721.10
New Paradigm Case Study
• Derived from a mid-project assessment of a recent project – what’s up?
• project involved development of a new IP block • I was in charge of verifying the IP block
– I could... • Plan incremental progress • Redefine the definition of done • Prevent bugs in the testbench with TDD
– I couldn’t... • Prioritize top-level • Prevent bugs in the design
© 2011 XtremeEDA USA Corporation - Version 080721.10
• Split the entire test effort into a series of steps – Step 1: Basic sanity of primary behavior – Step 2: Sanity of first major feature set – Step 3: Sanity of second major feature set – Step 4: Sanity of third major feature set – Step 5: Sanity of combined feature sets – Step 6: Functional coverage of combined feature sets – Step 7: Verify misc features – Step 8: Clean-up/release
• Incremental implementation of the testbench to support each step – testbench never includes more than the minimum required – choose techniques that suit each step
• directed tests to start • random tests later
Incremental Progress
29
© 2011 XtremeEDA USA Corporation - Version 080721.10
Incremental Progress
30
© 2011 XtremeEDA USA Corporation - Version 080721.10
Redefine The Definition Of Done
31
© 2011 XtremeEDA USA Corporation - Version 080721.10
Redefine The Definition Of Done
32
© 2011 XtremeEDA USA Corporation - Version 080721.10
• Goal: – a bug free test harness
• How: 1. build component features using TDD 2. integrate component features into the test harness as
they’re completed 3. use the features against the design as soon as they’re
integrated 4. repeat 1-3 until done
Bug Prevention With TDD
© 2011 XtremeEDA USA Corporation - Version 080721.10
SVUnit Testing Framework
34
Testrunner
<ID_a>_testsuite <ID_n>_testsuite
<ID_x>_unit_test <ID_y>_unit_test
test_<a>() test_<b>()
Test Runner Object
Test Suite Objects
Unit Test Objects
Test Methods
Perl scripts and Makefiles
write code here
© 2011 XtremeEDA USA Corporation - Version 080721.10
SVUnit In Practice - Mechanics
35
Write unit test for new test harness
feature
Ensure the new unit test passes
Ensure the new unit test fails
Write model code for new feature
START
Verify the Testbench
(TDD w/SVUnit)
© 2011 XtremeEDA USA Corporation - Version 080721.10
SVUnit In Practice - Mechanics
36
Write unit test for new test harness
feature
Ensure the new unit test passes
Write test
Ensure the new unit test fails
Write model code for new feature
Run test
File BUG
START END
Verify the Testbench
(TDD w/SVUnit)
Verify the Design
(Directed Test)
© 2011 XtremeEDA USA Corporation - Version 080721.10
• Case Study Test harness – 3129 lines of code – 5222 lines of test code – 116 unit tests total – 41 tests against the design – Found 30 design bugs – Found 2 test harness bugs – 9 weeks of effort
SVUnit In Practice – Client Case Study
37
Case Study Model 740 lines of code
2801 lines of test code 62 unit tests
35 tests against the design 19 design bugs
1 model bug 5 weeks effort
© 2011 XtremeEDA USA Corporation - Version 080721.10
New Paradigm Case Study – Summary
• Incremental “step-by-step” milestones make sense – fix the scope of the test harness to the goals of each step
• Easy to convey progress with real results – very little ambiguity
• Good code is easier to work with than garbage code – TDD is great for building a test harness – TDD is great for building reference models – TDD helped keep my bug rate very low
© 2011 XtremeEDA USA Corporation - Version 080721.10
Summary
39
The Old Paradigm
What’s Wrong With This Picture?
A New Paradigm
“So... how’s that working out?”
?
© 2011 XtremeEDA USA Corporation - Version 080721.10
Summary
40
The New Paradigm
What’s Wrong With This Picture?
A New Paradigm
“So... how’s that working out?”
© 2011 XtremeEDA USA Corporation - Version 080721.10
Summary
41
For More Information: • [email protected] • @nosnhojn • www.AgileSoC.com • http://sourceforge.net/projects/svunit