A Multi-Paradigm Verification Flow
Dave Whipp
Copyright © NVIDIA Corporation 2005
Multiple Paradigms
Analog
Mixed Signal
Digital
Formal
Semi-formal
Simulation
Emulation
Directed tests
Directed random
Constrained random
Self checking
Model based
Cycle based
Transaction based
Unit level
System level
Plan driven
Coverage driven
Pure Verilog
SystemVerilog
VHDL
SystemC
E / Vera
HDCaml / Confluence
Copyright © NVIDIA Corporation 2005
A Multi-Paradigm Verification Flow
What do I mean by Multiple Paradigms?
Why would we want one?
Engineering a [ Multiple Paradigm ] Flow
Was it worth the effort?
Copyright © NVIDIA Corporation 2005
Two Philosophies
ConformityEmphasize Standardization
DiversityEmphasize Innovation
Synergy or Conflict?
Copyright © NVIDIA Corporation 2005
Context
Large Project
Multiple Teams
Long Legacy
People move between projectsbenefit of familiar environment
exposure to alternative approaches
Copyright © NVIDIA Corporation 2005
Understanding Variation
To understand what to standardize:you need to understand what not to standardize
Personal Preferences
Technical Aspects of the Designs
Supporting Legacy
Seeking the Next Big Thing
Copyright © NVIDIA Corporation 2005
Personal Preferences
Emacs!
vi
Copyright © NVIDIA Corporation 2005
Personal Preferences
Choice of editor doesn’t affect othersAt least, not much
Choice of scripting language has greater impactBut is encapsulated
A script’s users don’t see the implementation language
Choice of HVL affects whole teamCan’t write “E” tests for a “Vera” testbench!
But a unit testbench isn’t seen by other units
A good flow will allow encapsulation of preferencesI can go to any unit and build & run its tests
Enables rapid localization of infrastructure issues
Copyright © NVIDIA Corporation 2005
Technological Differences
Copyright © NVIDIA Corporation 2005
Technical Characteristics
PCIE Video Frame
Buffer
Off-chip Memory
Graphics Pipe
Copyright © NVIDIA Corporation 2005
Unit Level System Testing
Stage N-1
Stage N+1
Stage N
Graphics Pipeline (Transaction Model)
RTL Unit DIFFDIFF
Copyright © NVIDIA Corporation 2005
Transactions Vs Cycles
Data min_val (Addr a1, Addr a2){ Data d1 = mem_read(a1); Data d2 = mem_read(a2);
if (d1 < d2)return d1;
elsereturn d2;
}
Address
Data
a1
d1
Pipelined Bus
a2
d2
t1 t2 t3 t4 t5 t6
Copyright © NVIDIA Corporation 2005
Reuse Vs Stagnation
Reuse considered GoodAvoid reinventing the wheel
Build on the shoulders of giants
Reuse invites InertiaReuse can propagate dependencies
Dependencies make things harder to change
Resistance to change is known as inertia
Inertia can lead to StagnationImproper reuse accumulates dependencies
Reused code that is not understood will bit-rot
To avoid stagnation, inject agitation
Copyright © NVIDIA Corporation 2005
New Challenges
New Tools
New Platforms
New People
New Ideas
Refactoring
Testability
D.R.Y.
Avoiding Stagnation
Copyright © NVIDIA Corporation 2005
Are Single Paradigm Projects Possible?
Project 1
Unit A
Unit B
Unit C
Project 2 Project 3
Unit D
Unit B
Unit C
Unit B
Paradigm 1
Paradigm 2
Paradigm 3
time
Copyright © NVIDIA Corporation 2005
1999
GeForce 25622 Million Transistors
2002
GeForce463 MillionTransistors
2003
GeForce FX130 Million
Transistors
2004
GeForce 6 Series222 Million
Transistors1993
NV11 Million Transistors
2005
GeForce 7 Series302 Million
Transistors
A Rich Legacy
A project must support at least three approachesLegacy
Mainstream
Future
A Paradigm-Agnostic Verification Flow
Copyright © NVIDIA Corporation 2005
What is a Verification Flow?
The User Interface to VerificationCommand Line Interface
Graphical Interface
Web (browser) Interface
Programmers Interface
Used ByVerification Engineers
Designers
Managers
Robots
Copyright © NVIDIA Corporation 2005
Purpose of a Flow
Conceptual FrameworkWhere do things live?
How do things work?
Keep out of the wayDon’t make life difficult for people
Define MetaphorsSubtly influence future directions
Anchor for VariationGive people something to complain about
Copyright © NVIDIA Corporation 2005
Concepts
A well defined interface behaves as expectedObvious how it works, just by looking (“affordances”)
As consistent as possible, but no more so
Options are decision points
Users don’t read the manual
Copyright © NVIDIA Corporation 2005
Designing a Flow (Interface)
Invent some Users
Figure out the important activities
Create the user model
Sketch a first draft
Iterate
Watch some real users; and iterate again
Copyright © NVIDIA Corporation 2005
Invent some Users
Tim Nalu
Dusk Mike
Copyright © NVIDIA Corporation 2005
Figure out the important Activities
TimburyNew Hire
Needs to get started
Checkout Tree
Run Sanity Regression
Simple Debug
DuskLead Designer
Intensive Debug
Modify RTLQuick Regressions
Analyze Coverage
Uses Emacs
NaluTesting Guru
Creates Testbenches
Hacks the Flow
Intensive Debug
Setup Regressions
Uses vi
MikeManager
View ProgressTests Written
Pass/fail counts
Coverage
Windows® User
Copyright © NVIDIA Corporation 2005
Task Summary
build testbenchesrun quick regression tests (sanity tests)
run specific testrerun specific test with debug options
rerun specific phase(s) of a test
Add a testAdd a testbench
overnight regression testingSummarize results (web page)
Copyright © NVIDIA Corporation 2005
User Model for Running Tests
Build Testbench
Generate Tests
Run Predictor
Run DUT
Compare Behaviors
Copyright © NVIDIA Corporation 2005
The Implementation
“Make” driven flowMakefiles define the build
Automated testbench creation
Compile/generate tests where appropriate
“makepp” is a better “make”
Perl scripting for the gutsEach test is a Perl object
Small number of standard classes
Individual units define additional classes
rdistnvmk run_unit
GEN DIFF
VSIM
CSIM
tbgen
fmtbgen
fmgen
run_testgen
compare
“grok”
randomizetestlist
rand_regress
mod
stgen
ttgen
build_testlist
tjob
LSF
vcs
An Early Sketch
Copyright © NVIDIA Corporation 2005
Command Line Interface
nvmknvmk sanity
nvmk regress
nvmk all
nvmk open_bugs TESTNAME=bug42
nvmk open_bugs TESTNAME=bug42 VSIM=1 DUMP=1
nvmk open_bugs TESTNAME=bug42 CSIM=gdb
Copyright © NVIDIA Corporation 2005
Detailed Triage and Debug
cd rundir/open_bugs/bug42
ls *.loggen.log csim.log
vsim.log diff.log
replay vsim.log +foo +bar
replay –gdb csim.log
Copyright © NVIDIA Corporation 2005
Testing the Flow
Considered a hard problem“If it works, then it works”
Testability is a featureStart testing early
Some tests are better than none
Create a multi purpose “Example Project”A stable testing environment
Acts as documentation
Use the “Invented Users” to create activity scripts
Keep it up-to-date
Copyright © NVIDIA Corporation 2005
Watch some Real Users
Image courtesy of DAZ Productions
Copyright © NVIDIA Corporation 2005
How to “Watch” Users
Meetings and Discussions
Coffee-Break Grousing
Bug Reports
Keep Track of Support Requestscreate FAQs
VNC (Remote Desktop)
Instrumentation
Copyright © NVIDIA Corporation 2005
Biggest Problems
Performancemake-driven user interface was subjectively slow
too much dependency checking
users know what file’s they’ve changed
how to communicate this to the tools?
Too much flexibilityOptions imply decisions
Lack of sufficient understanding
So pervasive cut & paste reuse
Copyright © NVIDIA Corporation 2005
Build Time Distribution: 10,000 per sample
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
> 1 hour
< 1 hour
< 30 min
< 15 min
< 10 min
< 5 min
< 2 min
< 1 min
< 30 sec
< 20 sec
Copyright © NVIDIA Corporation 2005
Build Time Distribution: 1000 per sample
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
> 1 hour
< 1 hour
< 30 min
< 15 min
< 10 min
< 5 min
< 2 min
< 1 min
< 30 sec
< 20 sec
Copyright © NVIDIA Corporation 2005
Summary
Verification Engineers do more than write tests
A flow is a user interfaceEncourage user feedback
Add instrumentation
A flow is an enabler, not disablerencourage innovation
don’t tell users what not to do
Without diversity, a flow stagnates
Without standardization, a flow flounders