verification concepts
TRANSCRIPT
-
7/28/2019 Verification Concepts
1/33
Verification concepts
ASIC DESIGN
The term Asic stands for Application Specific Integrated Circuit. Is an integrated circuit (IC) customized for a
particular use, rather than intended for general-purpose use. Generally an ASIC design will be undertaken for a
product that will have a large production run, and the ASIC may contain a very large part of the electronics
needed on a single integrated circuit. As feature sizes have shrunk and design tools improved over the years, the
complexity in an ASIC has grown from 5,000 over 100 million gates
Mrd
A Marketing Requirements Document (MRD) outlines the requirements a new product. Engineers use an MRD
to create the product. Marketing requirement document covers market needs, the customer value proposition,
and product functionality. It is developed by the Marketing team and upper management.
Architecture Specification
The architect based on the MRD specification, develops the overall architecture of the chip. This is a very high
level plan. Architecture Specification includes functional descriptions of each module, Properties and weights.
Design Specification
The designers and architects sit together to come up with detailed design documents. Design strategies, design
partitions, type of memories to use, etc.
-
7/28/2019 Verification Concepts
2/33
Verification Plan
A Verification specification is called a Verification/test plan. The verification engineer goes through all the above
documents and prepares verification plan to verify the design.
Rtl Design
RTL stands for Register Transfer Level. The designer starts implementing the RTL design in HDL like verilog or
VHDL.
Functional Verification
The verification engineers starts developing TestBench and verifies whether the DUT works according to
specification or not.
Synthesis
Synthesis is the process of taking a design written in a hardware description language, compiling it into a net listof interconnected gates which are selected from a user-provided library of various gates. The design after
synthesis is a gate-level design.
Physical Design
Physical design process includes logic partitioning, floor planning, global routing, detailed routing, compaction,
and performance-driven layout. PD team transforms net list representation of a system into layout
representation.
Timing Analysis
Static timing analysis is an important step in analyzing the performance of a design. In the Timing analysis Setup
time, hold time ,recovery time ,removal time , Clock latency, clock skew, clock uncertainty etc checks are done.
Tapeout
This is the final stage of the design cycle of integrated circuits. Once all the checks are done, the design is ready
to be sent to Foundry.
BOTTLE NECK IN ASIC FLOW
What is Bottle neck in the asic design flow?
Verification consumes 50% to 70% of the effort of design cycle and is on the critical path in the design flow
of multimillion gate ASICs, so verification became the main bottleneck in the design process. The functional
verification bottleneck is an effect of rising the design abstraction level. Majority of ASICs require at least
one re-spin with 71% of re-spins are due to functional bugs.
-
7/28/2019 Verification Concepts
3/33
FUNCTIONAL VERIFICATION NEED
Why we need functional verification?
To build confidence and stay in business.A primary purpose for functional verification is to detect failures so that bugs can be identified and
corrected before it gets shipped to costumer. If RTL designer makes a mistake in designing or coding, this
results as a bug in the Chip. If this bug is executed, in certain situations the system will produce wrong
results, causing a failure. Not all mistakes will necessarily result in failures. The bug in the dead code will
never result in failure. A single mistake may result in a wide range of failure symptoms. Not all bugs are
caused by coding errors. There are possibilities that error may in the specification itself. Sometimes
miscommunications between teams may lead to wrong design.
Example of coding error:
1reg[1:0] state;2
3parameter zero=0, one=1, two=2, three=3;
4
5always@(state)
6begin
7case(State)
8 zero:
9 out =4'b0000;
10 one:
11 out =4'b0001;
12 two:
13 out =4'b0010;14 three:
15 out =4'b0100;
16default:
17 out =4'b0000;
18endcase
19end
There is a coding error in the preceding example. Designer declared "state" in the line-1. Later in the code it
is reference in the line-5. In the line-7 also, the designer intention is to refer "state". But mistakenly he
typed "State". Verilog is a case-sensitive language, and variable "State" and "state" are different and this will
produce wrong results.
TESTBENCH
What is TestBench?
TestBench mimic the environment in which the design will reside. It checks whether the RTL Implementation
meets the design spec or not. This Environment creates invalid and unexpected as well as valid and expected
conditions to test the design.
-
7/28/2019 Verification Concepts
4/33
LINEAR TESTBENCH
Linear TestBench is the simplest, fastest and easiest way of writing testbenchs. This became novice
verification engineer choice. It is also slowest way to execute stimulus. Typically, linear testbenchs are
written in the VHDL or Verilog. In this TestBench, simple linear sequence of test vectors is mentioned.
Stimulus code like this is easy to generate translating a vector file with a Perl script, for example. Smallmodels like simple state machine or adder can be verified with this approach. The following code snippet
shows linear TestBench. The code snippet shows some input combination only. This is also bad for simulator
performance as the simulator must evaluate and schedule a very large number of events. This reduces
simulation performance in proportion to the size of the stimulus process.
Typically, linear testbenchs perform the following tasks:
Instantiate the design under test (DUT)
Stimulate the DUT by applying test vectors.
Output results waveform window or to a terminal for visual inspection manually.
Example: Linear TestBench
module adder(a,b,c);//DUT code start
input[15:0] a;
input[15:0] b;
output[16:0] c;
assign c = a + b;
endmodule//DUT code end
module top();//TestBench code start
reg[15:0] a;
reg[15:0] b;
wire[16:0] c;
adder DUT(a,b,c);//DUT Instantiation
initial
begin
a =16'h45;//apply the stimulus
b =16'h12;
#10$display(" a=%0d,b=%0d,c=%0d",a,b,c);
//send the output to terminal for visual inspection
end
endmodule//TestBench code end
To test all possible scenarios which are known to us, it is not an easy task. Development time increases
exponentially as the number of scenarios increases and maintain them is nightmare. Instead of listing out all
the possible scenarios, pickup some randomly and check the DUT.
-
7/28/2019 Verification Concepts
5/33
NOTETONOVOICEENGINEERS:
Generally tendency of any novice engineer is to see the outputs in the waveform viewer. Waveform viewers
are for debugging designs, not for testbench. Most of the operation in TestBench executes in zero time,
where waveform viewer will not be helpful. All the examples in the book outputs messages to terminal for
analyzing its behavior.
LINEAR RANDOM TESTBENCH
Random TestBench don't use Hardcoded values like linear testbenchs. Input stimulus is generated using
random values. In Verilog, system function $random provides a mechanism for generating random numbers.
The function returns a new 32-bit random number each time it is called. These test cases are not easily
readable and are also not reusable. New tests have to be created when the specification or design changes,
to accommodate the changes. The main disadvantage of this testing is that we never know what random
values are generated and it may waste simulation cycles by generating same values again and again.
EXAMPLE: Linear Random TestBench
module adder(a,b,c);//DUT code startinput[15:0] a,b;
output[16:0] c;
assign c = a + b;
endmodule//DUT code end
module top();//TestBench code start
reg[15:0] a;
reg[15:0] b;
wire[16:0] c;
adder DUT(a,b,c);//DUT Instantiation
initial
repeat(100)begin
a =$random;//apply random stimulus
b =$random;
#10$display(" a=%0d,b=%0d,c=%0d",a,b,c);
end
endmodule//TestBench code end
HOW TO CHECK THE RESULTS
How does a Verification engineer check whether the results obtained from the simulation match the originalspecification of the design? For simple testbenchs like the above, output is displayed in waveform window or
messages are sent to terminal for visual checking. Visually checking is the oldest and most labor intensive
technique. The quality of the verification depends on the determination and dedication of the individual
who is doing the checking. It is not practical to verify a complex model merely by examining the waveform
or text file. Whenever a change is made to the DUT to add a new feature or to fix a bug, same amount of
effort needs to be deployed to check the simulation results.
-
7/28/2019 Verification Concepts
6/33
SELF CHECKING TESTBENCHS
A self-checking TestBench checks expected results against actual results obtained from the simulation.
Although Self-checking testbenchs require considerably more effort during the initial test bench creationphase, this technique can dramatically Reduce the amount of effort needed to re-check a design after a
change has been made to the DUT. Debugging time is significantly shortened by useful error-tracking
information that can be built into the TestBench to show where a design fails.
A self-checking TestBench has two major parts, the input blocks and output blocks.
Input block consist of stimulus and driver to drive the stimulus to DUT. The output block consists of monitor
to collect the DUT outputs and verify them.
All the above approaches require the test writer to create an explicit test for each feature of the design.
Verification approach in which each feature is written in a separate test case file is called directed
verification.
EXAMPLE: adder example
module adder(a,b,c);//DUT code start
input[15:0] a,b;
output[16:0] c;
assign c = a + b;
endmodule//DUT code end
module top();//TestBench code start
reg[15:0] a;
reg[15:0] b;
wire[16:0] c;
adder DUT(a,b,c);//DUT Instantiation
initial
repeat(100)begin
a =$random;//apply random stimulus
b =$random;
#10
$display(" a=%0d,b=%0d,c=%0d",a,b,c);
-
7/28/2019 Verification Concepts
7/33
if( a + b != c)// monitor logic.
$display(" *ERROR* ");
end
endmodule//TestBench code end
HOW TO GET SCENARIOS WHICH WE NEVER THOUGHT
In Directed verification, the Verification Environment has mechanism to send the Stimulus to DUT and
collect the responses and check the responses. The Stimulus is generated in Tests case. Directed testbenchs
may also use a limited amount of randomization, often by creating random data values rather than simply
filling in each data element with a predetermined value. Each test case verifies specific feature of the
design. This becomes tedious when the design complexity increases. As circuit
complexity increases, it becomes more difficult to create patterns that fully exercise the design. Test case
maintenance become harder and time consuming.
In Directed Verification, test writer has to list out each feature. Test writer can't think of all possible
potential bug scenarios and there are chances that Bugs will escape. With these approaches, the bugs lurking
in these corners hide until late in the development cycle, or aren't found at all until product is taped out.
Solution to the above problems is Constraint random verification. Using constraint random verification, the
stimulus required to verify test features are generated automatically. Test writer specifies set of
specification, and the TestBench automatically creates solution space and picks up scenarios from the
solution space.
Constraint random verification also reduces manual effort and code for individual tests. As the scenarios are
generated automatically by the TestBench, the number of test case files gets reduced. In Directed
verification, some of the tests share similar logic, if the engineer has to change the logic which is common to
-
7/28/2019 Verification Concepts
8/33
certain group of tests, then he has to edit all the test case files and it is time consuming. But in Constraint
random verification, the number of tests case files will be very less, so changes will be mostly in
environment and minimal.
Directed verification with a fairly simple TestBench, verification engineers can start finding bugs in
simulation almost immediately even before the TestBench is fully completed. With a constrained-random
verification environment, there is an up-front cost that must be invested before the first test can be run.Constraint-based generators can be easily converted into checkers if required.
Generating fully randomly is meaningless as it may generate invalid scenarios and it may also regenerate the
same scenario again and again wasting potential simulation time. A User must define data structures, whichrepresent stimulus applied to the DUT input. Next, constraints must be defined to guide the random
generator. Using constraint, the solution space is defined, and randomization picks up scenarios randomly
from the solution space. Constraints act as knobs in the TestBench which control the generator's
randomness.
The main disadvantage of constraint random verification is we never know how well the DUT is verified. If
verification engineer can get the information about the logic in DUT which is not verified, he can further
constraint the randomization or write directed testcases to exercise the unverified logic.
HOW TO CHECK WHETHER THE TESTBENCH HAS SATISFACTORILY EXERCISED THE
DESIGN
Code coverage is used to measure the efficiency of verification implementation. It provides a quantitative
measurement of the testing space. It describes the degree to which the source code of a DUT has been
tested. It is also referred as structural coverage.
Code coverage answers the questions like
-
7/28/2019 Verification Concepts
9/33
Have all the branches in " Case ", "if" have been entered?
Have all the conditions in "if","case" statement is simulated?
Have all the variables have been toggles?
Have all the statements of the RTL code have been exercised?
Have all the states in the FSM has been entered and all the legal transitions exercised?
Have all the paths within a block have been exercised?
By applying code coverage analysis techniques to hardware description languages, verification efficiency was
improved by enabling a verification engineer to isolate areas of un-tested HDL code. The verification
engineer examine a coverage report, seeks out the low values and understands why that particular code
hasn't been tested fully and writes more tests or directs randomness to cover the untested areas where
there may be a possibility of bug hiding.
No additional coding is required to get 100 percent code coverage , the tool would automatically show the
item as covered if the required test scenario(s)/combination(s) is(are) exercised.
In unit level verification, a module by module is verified in its own test environment to prove that the logic,control, and data paths are functionally correct. The goal of module level verification is to ensure that the
component/unit being tested conforms to its specifications and is ready to be integrated with other
subcomponents of the product. Code coverage becomes a criterion for finishing unit level testing as it needs
to verify every feature of component/unit. In sub-system level /system level, the goal is to ensure that the
interfaces among the units are correct and the units work together to execute the functionality correctly. In
sub system level /system level testing, code coverage may not be useful as the verification is not targeted at
all the features of the unit.
TYPES OF CODE COVERAGE
There are a number of coverage criteria, they are:
Statement coverage /line coverage
Block/segment coverage
Conditional coverage
Branch coverage
Toggle coverage
Path coverage
Fsm coverage
The following example is considered while explaining code coverage types in further sections.
EXAMPLE
1
2module dut();
3reg a,b,c,d,e,f;
4
5initial
6begin
-
7/28/2019 Verification Concepts
10/33
7#5 a =0;
8#5 a =1;
9end
10
11always@(posedge a)
12begin
13 c = b && a;14if(c && f)
15 b = e;
16else
17 e = b;
18
19case(c)
201:f =1;
210:f =0;
22default: f =0;
23endcase
24
25end26endmodule
STATEMENT COVERAGE
Statement coverage, also known as line coverage is the easiest understandable type of coverage. This is
required to be 100% for every project. From N lines of code and according to the applied stimulus how many
statements (lines) are covered in the simulation is measured by statement coverage. If a DUT is 10 lines long
and 8 lines of them were exercised in a test run, then the DUT has line coverage of 80%. Line coverage
includes continuous assignment statements, Individual procedural statements, Procedural statement blocks,
Procedural statement block types, Conditional statement and Branches for conditional statements. It
considers only it the executable statements and statements which are not executable like module,endmodule, comments, timescale etc are not covered.
Statement coverage report of the above example:
There are total 12 statements at lines 5,7,8,11,13,14,15,17,19,20,21,22
Covered 9 statements. They are at lines
5,7,8,11,13,14,17,19,22
Uncovered 3 statements. They are at line
15,20,21
Coverage percentage: 75.00 (9/12)
BLOCK COVERAGE
The nature of the statement and block coverage looks somewhat same. The difference is that block coverage
considers branched blocks of if/else, case branches, wait, while, for etc. Analysis of block coverage reveals
the dead code in RTL.
-
7/28/2019 Verification Concepts
11/33
Block coverage report of the above example:
There are total 9 blocks at lines
5,7,8,11,15,17,20,21,22
Covered 6 blocks. They are at lines5,7,8,11,17,22
Uncovered 3 blocks. They are at line
15,20,21
Coverage percentage: 66.67 (6/9)
CONDITIONAL COVERAGE
Conditional coverage also called as expression coverage, will reveals how the variables or sub-expressions in
conditional statements are evaluated. Expressions with logical operators are only considered.. The downsideis that the conditional coverage measure doesn't take into consideration how the Boolean value was gotten
from the conditions. Conditional coverage is the ratio of no. of cases checked to the total no. of cases
present. Suppose one expression having Boolean expression like AND or OR, so entries which is given to that
expression to the total possibilities is called expression coverage.
Conditional coverage report of the previous example:
At LINE 13
Combinations of STATEMENT c = (b && a)
B = 0 and a = 0 is Covered
B = 0 and a = 1 is Covered
B = 1 and a = 0 is Not Covered
b = 1 and a = 1 is Not Covered
At LINE 14 combinations of STATEMENT if ((c && f))
C = 0 and f = 0 is Covered
C = 0 and f = 1 is Not Covered
C = 1 and f = 0 is Not Covered
C = 1 and f = 1 is Not Covered
Total possible combinations: 8
Total combinations executed: 3
BRANCH COVERAGE
Branch coverage which is also called as Decision coverage report s the true or false of the conditions like if-
else, case and the ternary operator (? :) statements. For an "if" statement, decision coverage will report
whether the "if" statement is evaluated in both true and false cases, even if "else" statement doesn't exist.
Branch coverage report of the example:
At line 15 branch b = e; not covered
-
7/28/2019 Verification Concepts
12/33
At line 17 branch e = b; covered
At line 20 branch 1: f = 1; not covered
At line 21 branch 0: f = 0; covered
At line 22 branch default: f = 0; not covered
Coverage percentage: 40.00 (2/5)
PATH COVERAGE
Path coverage represents yet another interesting measure. Due to conditional statements like if-else, case in the
design different path is created which diverts the flow of stimulus to the specific path.
Path coverage is considered to be more complete than branch coverage because it can detect the errors related to
the sequence of operations. As mentioned in the above figure path will be decided according to the if-else
statement According to the applied stimulus the condition which is satisfied only under those expressions will
execute, the path will be diverted according to that. Path coverage is possible in always and function blocks . Path
created by more than one block is not covered. Analysis of path coverage report is not so easy task.
Path coverage report of the example:
Path 1 : 15,20 Not Covered
Path 2 : 15,21 Not CoveredPath 3: 15,22 Not Covered
Path 4: 17,20 Not Covered
Path 5 : 17,21 Covered
Path 6 : 17,22 Not Covered
Total possible paths : 6
Total covered path : 1
Path coverage Percentage : 16.67 (1/6)
-
7/28/2019 Verification Concepts
13/33
TOGGLE COVERAGE
It makes assures that how many times variables and nets toggled? Toggle coverage could be as simple as the ratio
of nodes toggled to the total number of nodes.
X or Z --> 1 or H
X or Z --> 0 or L1 or H --> X or Z
0 or L --> X or Z
Above example shows the signal changes from one level to another. All types of transitions mentioned above are
not interested. Only 1->0 and 0->1 are important. Toggle coverage will show which signal did not change the state.
Toggle coverage will not consider zero-delay glitches. This is very useful in gate level simulation.
Toggle coverage report of the example:
Name Toggled 1->0 0->1
a No No Yes
b No No Noc No No No
d No No No
e No No No
f No No No
FSM COVERAGE
It is the most complex type of code coverage, because it works on the behavior of the design. Using Finite state
machine coverage, all bugs related to finite state machine design can be found. In this coverage we look for how
many times states are visited, transited and how many sequence are covered in a Finite state machine.
State Coverage:
It gives the coverage of no. of states visited over the total no. of states. Suppose you have N number of states and
state machines transecting is in between only N-2 states then coverage will give alert that some states are
uncovered. It is advised that all the states must be covered.
Transition Coverage:
It will count the no. of transition from one state to another and it will compare it with other total no. of
transition. Total no. of transition is nothing but all possible no. of transition which is present in the finite state
machine. Possible transition = no. of states * no. of inputs.
EXAMPLE of FSM:
module fsm (clk, reset, in);
input clk, reset, in;
reg[1:0] state;
parameter s1 =2'b00;parameter s2 =2'b01;
-
7/28/2019 Verification Concepts
14/33
parameter s3 =2'b10;parameter s4 =2'b11;
always@(posedge clk orposedge reset)
begin
if(reset) state
-
7/28/2019 Verification Concepts
15/33
MAKE YOUR GOAL 100 PERCENT CODE COVERAGE NOTHING LESS
Never set your goal to anything less than 100% code coverage. Anything less than 100% is a slippery slope. If you
set your goal to 98% , may be the most important feature like reset of the system may be in the untested part
of2%. If the verification engineer sets the code coverage goal to 95% to facilitate the 5% the unused untestable
legacy code, there are chances that the unused legacy code gets executed and the 5% holes may be in theimportant code. 100% code coverage provides advantages not only in reducing the bug count but also in making it
easier to make significant changes to existing code base to remove uncover able areas like the unused legacy
blocks in RTL code.
Dont Be Fooled By The Code Coverage Report
Highly covered code isn't necessarily free of defects, although it's certainly less likely to contain them. By
definition, code coverage is limited to the design code. It doesn't know anything about what design supposed to
do. Even If a feature is not implemented in design, code coverage can report 100% coverage. It is also impossible
to determine whether we tested all possible values of a feature using code coverage. For example, randomization
may not generate packets with all possible lengths, this cannot be reported by code coverage.. Code coverage isunable to tell much about how well you have covered your logic -- only whether you've executed each line/block
etc at least once. Code coverage does not provide information about your test bench randomization quality and it
does not report what caused the line execution/state transition etc. Analysis of code coverage require knowledge
of design to find which features are not verified which is time consuming and out of scope of verification engineer.
If the analysis is done at higher level of abstraction, it would be easier for the test writer to identify the missed
serious which is not possible by code coverage. So if the code coverage is less than 100%, it means there is more
work to do, if it is 100%, it doesn't mean that the verification is complete.
When To Stop Testing?
It's getting harder to figure out when to stop testing as the complexity of the protocol is increasing. In directed
test environment, for each point mentioned in test plan, there will be a separate test case file. So if there are 100
points in test plan, then the engineer has to write 100 test case files. After writing and executing the 100 test case
files, we can say that "all the points in test plan are verified" and we can stop testing.
FUNCTIONAL COVERAGE
In CRV, the each point in test plan is generated automatically. As there points are generated automatically, we
need a mechanism which tells us that all the points in test plan are not exercised. When all the points in testplans
are verified and the code coverage is 100% we can stop verification.
What are the untested features?
In Directed verification, there will be a separate testcase file for each feature to be verified. So to know how
many features are verified, count the testcases. Verification is done when all tests are coded and passing alone
with 100% code coverage. In constraint random verification all the features are generated randomly. Verification
engineer need a mechanism to know the information about the verified features of DUT.
SystemVerilog provides a mechanism to know the untested feature using functional coverage. Functional Coverage
is "instrumentation" that is manually added to the TestBench. This is a better approach then counting testcases.
Functional coverage is better than code coverage where the code coverage reports what was exercised rather that
-
7/28/2019 Verification Concepts
16/33
what was tested.
Functional coverage answers questions like
Have all the packets length between 64 to 1518 are used?
Did the DUT got exercised with alternate packets with good and bad crc?
Did the monitor observe that the result comes with 4 clock cycles after read operation?Did the fifos are filled completely?
Summary of functional coverage advantages:
Functional coverage helps determine how much of your specification was covered.
Functional coverage qualifies the testbenchs.
Considered as stopping criteria for unit level verification.
Gives feedback about the untested features.
Gives the information about the redundant tests which consume valuable cycle.
Guides to reach the goals earlier based on grading.
Introduction To Functional Coverage:
SystemVerilog provides 2 ways to mention coverage. Cover group which is uses information from transactor,
monitor and checker. Assertion coverage which is uses temporal language which can be outside or inside RTL
code.
Covergroup:
There are three types of cover group points:
1. item functional coverage point
2. cross functional coverage point
3. transitional functional coverage point
Item
"Item" is used to capture the information about the sclare value. A range of interested values can also be
observed.
For example, consider a packet protocol. The packet has a address field with possible values of A0,A1 and data
which can be 4 to 10 bytes. At the end of packet parity is also attached for integrity checking.
The following table identifies these coverage points for the above packet:
Item_DL Data length 4 to 10
Item_ADD Address A0,A1
Item_Par Parity Good and Bad
Coverage engine collects item values are during simulation and reports.
Reports consists how many timesPackets with length 4,5,6,7,8,9,10 are generated.
Packets with good parity, bad parity are generated.
Packets with address A0, A1 are generated.
Cross
"cross" is used to examine the cross product of two or more item coverage points.
Example: verify DUT by sending both good parity and bad parity packets with all the address.
-
7/28/2019 Verification Concepts
17/33
Cross_ ADD_Par Item_ADD,Item_Par
Coverage report consists how many timesPackets with Address A0 with good parity are generated
Packets with Address A0 with bad parity are generated
Packets with Address A1 with good parity are generated
Packets with Address A1 with bad parity are generated
Transitional
Transitional functional point is used to examine the legal transitions of a value.
Example:
Verify the DUT with incremental packet length from 4 to 10.
Trans_length ( 4 => 5 => 6 => 7 => 8 =>9 =>10)
Coverage engine reports whether this sequence is exercised or not.
Assertion Coverage:
Assertion coverage looks for the desired behavior in RTL. It uses assertion language which is in temporal nature. It
has direct access to design variables and designer can add many points in RTL which he wants the verification
engineer to cover.
Example: Verify by sending back-to-back packets.
Functional coverage will be discussed in more detail in later chapters.
Verification using functional coverage and randomization is called coverage driven constraint random verification.
COVERAGE DRIVEN CONSTRAINT RANDOM VERIFICATION ARCHITECTURE
Basic functionality of CDRV Environment:
Input side of DUT :
-- Generating traffic streams
-- Driving traffic into the design (stimuli)
Output side of DUT:
-- Checking these data streams
-- Checking protocols and timing
Collecting both the functional coverage and code coverage information.
Writing deterministic tests and random tests to achieve 100% coverage.
-
7/28/2019 Verification Concepts
18/33
Verification Components Required For Cdcrv:
Stimulus
Stimulus generator
Transactor
Driver
Monitor
Assertion monitor
Checker
Scoreboard
Coverage
Utilities
Tests
Stimulus:
When building a verification environment, the verification engineer often starts by modeling the device input
stimulus. In Verilog, the verification engineer is limited in how to model this stimulus because of the lack of high-
level data structures. Typically, the verification engineer will create a array/memory to store the stimuli.
SystemVerilog provides high-level data structures and the notion of dynamic data types for modeling stimulus.
-
7/28/2019 Verification Concepts
19/33
Using SystemVerilog randomization, stimulus is generated automatically. Stimulus is also processed in other
verification components. SystemVerilog high-level data structures helps in storing and processing of stimulus in an
efficient way.
Stimulus Generator
The generator component generates stimulus which are sent to DUT by driver. Stimulus generation is modeled to
generate the stimulus based on the specification. For simple memory stimulus generator generates read, write
operations, address and data to be stored in the address if its write operation. Scenarios like generate alternate
read/write operations are specified in scenario generator. SystemVerilog provided construct to control the random
generation distribution and order. Constraints defined in stimulus are combinatioural in nature where as
constraints defined in stimulus generators are sequential in nature.
Stimulus generation can be directed or directed random or automatic and user should have proper controllability
from test case. It should also consider the generation of stimulus which depends on the state of the DUT for
example, Generating read cycle as soon as interrupt is seen. Error injection is a mechanism in which the DUT is
verified by sending error input stimulus. Generally it is also taken care in this module.
Generally generator should be able to generate every possible scenario and the user should be able to control thegeneration from directed and directed random testcases.
Transactor
Transactor does the high level operations like burst-operations into individual commands, sub-layer protocol in
layered protocol like PciExpress Transaction layer over PciExpress Data Link Layer, TCP/IP over Ethernet etc. It
also handles the DUT configuration operations. This layer also provides necessary information to coverage model
about the stimulus generated. Stimulus generated in generator is high level like Packet is with good crc, length is 5
and da is 8~Rh0. This high level stimulus is converted into low level data using packing. This low level data is just a
array of bits or bytes. Packing is an operation in which the high level stimulus values scalars, strings, array
elements and struct are concatenated in the specified manner.
Driver
The drivers translate the operations produced by the generator into the actual inputs for the design under
verification. Generators create inputs at a high level of abstraction namely, as transactions like read write
operation. The drivers convert this input into actual design inputs, as defined in the specification of the designs
interface. If the generator generates read operation, then read task is called, in that, the DUT input pin
"read_write" is asserted.
Monitor
Monitor reports the protocol violation and identifies all the transactions. Monitors are two types, Passive and
active. Passive monitors do not drive any signals. Active monitors can drive the DUT signals. Sometimes this is also
refered as receiver. Monitor converts the state of the design and its outputs to a transaction abstraction level so it
can be stored in a 'score-boards' database to be checked later on. Monitor converts the pin level activities in to
-
7/28/2019 Verification Concepts
20/33
high level.
Assertion Based Monitor
Assertions are used to check time based protocols, also known as temporal checks.Assertions are a necessary compliment to transaction based testing as they describe the pin level, cycle by cycle,
protocols of the design. Assertions are also used for functional coverage.
Data Checker
The monitor only monitors the interface protocol. It doesn't check the whether the data is same as expected data
or not, as interface has nothing to do with the date. Checker converts the low level data to high level data and
validated the data. This operation of converting low level data to high level data is called Unpacking which is
reverse of packing operation. For example, if data is collected from all the commands of the burst operation and
then the data is converted in to raw data , and all the sub fields information are extracted from the data and
compared against the expected values. The comparison state is sent to scoreboard.
Scoreboard
Scoreboard is sometimes referred as tracker. Scoreboard stores the expected DUT output. Scoreboard in Verilog
tends to be cumbersome, rigid, and may use up much memory due to the lack of dynamic data types and memory
allocation. Dynamic data types and Dynamic memory allocation makes it much easier to write a scoreboard in
SystemVerilog. The stimulus generator generated the random vectors and is sent to the DUT using drivers. These
stimuli are stored in scoreboard until the output comes out of the DUT. When a write operation is done on a
memory with address 101 and data 202, after some cycles, if a read is done at address 101, what should be the
data?.The score board recorded the address and data when write operation is done. Get the data stored at address
of 101 in scoreboard and compare with the output of the DUT in checker module. Scoreboard also has expected
logic if needed. Take a 2 inputs and gate. The expect logic does the "and " operation on the two inputs and stores
the output".
Coverage
This component has all the coverage related to the functional coverage groups.
Utilities
Utilities are set of global tasks which are not related to any protocol. So this module can be reused across projects
without any modification to code. Tasks such as global timeout, printing messages control, seeding control, testpass/fail conditions, error counters etc. The tasks defined in utilities are used by all other components of the
TestBench.
Environment:
Environment contains the instances of all the verification component and Component connectivity is also done.
-
7/28/2019 Verification Concepts
21/33
Steps required for execution of each component is done in this.
Tests
Tests contain the code to control the TestBench features. Tests can communicate with all the TestBench
components. Once the TestBench is in place, the verification engineer now needs to focus on writing tests to
verify that the device behaves according to specification.
PHASES OF VERIFICATION
Verification Plan
In test plan, we prepare a road map for how do achieve the goal, it is a living document. Test plan includes,
introduction, assumptions, list of test cases, list of features to be tested, approach, deliverables, resources, risksand scheduling, entry and exit criteria. Test plan helps verification engineer to understand how the verification
should be done. A test plan could come in many forms, such as a spreadsheet, a document or a simple text file.
Sometimes, test plan simply reside in the engineer's head which is dangerous in which the process cannot be
properly measured and controlled. Test plan also contains the descriptions of TestBench architecture and
description of each component and its functionality.
-
7/28/2019 Verification Concepts
22/33
Building Testbench
In this phase, the verification environment is developed. Each verification component can be developed one by
one or if more than one engineer is working it can be developed parallel. Writing the coverage module can be done
at any time. It is preffered to write down the coverage module first as it gives some idea of the verification
progress.
Writing Tests
After the TestBench is built and integrated to DUT, it's time for validating the DUT. Initially in CDV, the test are
ran randomly till some 70 % of coverage is reached or no improvement in the coverage for 1 day simulation. By
analyzing the coverage reports, new tests are written to cover the holes. In these tests, randomization is directed
to cover the holes. Then finally, the hard to reach scenarios, called as corner cases have to be written in directed
verification fashion. Of course, debugging is done in parallel and DUT fixes are done.
Integrating Code Coverage
Once you have achieved certain level of functional coverage, integrate the code coverage. For doing codecoverage, the code coverage tools have option to switch it on. And then do the simulation, the tool will provide
the report.
Analyze Coverage
Finally analyze both functional coverage and code coverage reports and take necessary steps to achieve coverage
goals. Run simulation again with a different seed, all the while collecting functional coverage information.
ONES COUNTER EXAMPLE
Following example is TestBench for ones counter. It has some verification components which are required, but notall the verification components discussed earlier. All the components implementation can be seen in further
chapters with another protocol. Description of the language construct is discussed in further chapters, so don't pay
attention to them. The intention of showing this example is to make you familiar with some steps required while
building verification environment and to help you to understand the flow discussed above.
Specification:
Ones Counter is a Counter which counts the number of one's coming in serial stream. The Minimum value of the
count is "0" and count starts by incriminating one till "15". After "15" the counter rolls back to "0". Reset is also
provided to reset the counter value to "0". Reset signal is active negedge. Input is 1 bit port for which the serial
stream enters. Out bit is 4 bit port from where the count values can be taken. Reset and clock pins also provided.
-
7/28/2019 Verification Concepts
23/33
The following is the RTL code of onescounter with bugs.
module dff(clk,reset,din,dout);
input clk,reset,din;
output dout;
logic dout;
always@(posedge clk,negedge reset)
if(!reset)
dout
-
7/28/2019 Verification Concepts
24/33
Block Diagram:
Verification Environment Hierarchy
TOP
|-- Clock generator
|-- Dut Instance
|-- Interface
|-- Assertion block instance ( assertion coverage)
|-- Testcase instance
|-- Environment
|-- Driver
| |-- Stimulus
| |-- Covergroup
|-- Monitor
|-- Scoreboard
-
7/28/2019 Verification Concepts
25/33
Testbench Components:
Stimulus
Stimulus is a single bit value.
class stimulus;randbit value;
constraint distribution {value dist{0:=1,1:=1};}
endclass
Driver
This driver consists of reset and drive method. Reset method resets the DUT and drive method generates the
stimulus and sends it to DUT. Driver also calculates the expected DUT output and stores in scoreboard. Coverage is
also sampled in this component. Feature 1 and 2 which are mentioned in Testplan are covered in this cover group.
class driver;
stimulus sti;
Scoreboard sb;
covergroup cov;
Feature_1:coverpoint sb.store ;
Feature_2 :coverpoint sb.store {bins trans =(15=>0);}
-
7/28/2019 Verification Concepts
26/33
endgroup
virtual intf_cnt intf;
functionnew(virtual intf_cnt intf,scoreboard sb);
this.intf = intf;
this.sb = sb;cov =new();
endfunction
task reset();// Reset method
intf.data=0;
@(negedge intf.clk);
intf.reset =1;
@(negedge intf.clk);
intf.reset =0;
@(negedge intf.clk);
intf.reset =1;
endtask
task drive(inputinteger iteration);
repeat(iteration)
begin
sti =new();
@(negedge intf.clk);
if(sti.randomize())// Generate stimulus
intf.data= sti.value;// Drive to DUT
sb.store = sb.store + sti.value;// Cal exp value and store in Scoreboard
if(sti.value)
cov.sample();
end
endtask
endclass
Monitor
The monitor collects the DUT output and then gets the expected value from the score board and compares them.
class monitor;
scoreboard sb;
virtual intf_cnt intf;
functionnew(virtual intf_cnt intf,scoreboard sb);
this.intf = intf;
this.sb = sb;
endfunction
task check();
forever
-
7/28/2019 Verification Concepts
27/33
@(negedge intf.clk)
if(sb.store != intf.count)// Get expected value from scoreboard and compare with DUT output
$display(" * ERROR * DUT count is %b :: SB count is %b ", intf.count,sb.store );
else
$display(" DUT count is %b :: SB count is %b ", intf.count,sb.store );
endtask
endclass
Assertion Coverage
This block contains the assertion coverage related to 3 rd feature mentioned in testplan.
module assertion_cov(intf_cnt intf);
Feature_3 :coverproperty(@(posedge intf.clk)(intf.count !=0)|-> intf.reset==0);
endmodule
Scoreboard
This scoreboard is a simple one which stores one expected value.
class scoreboard;
bit[0:3] store;
endclass
Environment:
Environment contains instances of driver, monitor and scoreboard.
class environment;
driver drvr;
scoreboard sb;
monitor mntr;
virtual intf_cnt intf;
functionnew(virtual intf_cnt intf);
this.intf = intf;
sb =new();
drvr =new(intf,sb);
mntr =new(intf,sb);
fork
mntr.check();
join_none
endfunction
endclass
-
7/28/2019 Verification Concepts
28/33
Top:
The interface is declared and the test bench and DUT instances are taken. Testbench and DUT are connected using
interfaces. Clock is also generated and connects it to DUT and testbench.
interface intf_cnt(input clk);
wire clk;
wire reset;
wiredata;
wire[0:3] count;
endinterface
module top();
reg clk =0;
initial// clock generator
forever#5 clk =~clk;
// DUT/assertion monitor/testcase instances
intf_cnt intf(clk);
ones_counter DUT(clk,intf.reset,intf.data,intf.count);
testcase test(intf);
assertion_cov acov(intf);
endmodule
Tests:
This is a simple test case. It does reset and then send 10 input values.
program testcase(intf_cnt intf);
environment env =new(intf);
initial
begin
env.drvr.reset();
env.drvr.drive(10);
end
endprogram
After simulating with this testcase, the coverage report I got
(S)Total Coverage Summary
-
7/28/2019 Verification Concepts
29/33
SCOREASSERTGROUP
9.380.0018.75
(S)Cover group report
VARIABLEEXPECTEDUNCOVEREDCOVEREDPERCENTGOALWEIGHT
Feature_1 1610637.501001Feature_2 1100.001001
(S)Assertion coverage report:
COVERPROPERTIESCATEGORYSEVERITYATTEMPTSMATCHESINCOMPLETE
Feature_3 001300
This coverage report will be different if you simulate in your tool.
To improve the coverage, than the 1st testcase , I wrote 2nd testcase with more input values and also logic related
to 3 feature in the testplan.
program testcase(intf_cnt intf);
environment env =new(intf);
initial
begin
env.drvr.reset();
env.drvr.drive(100);
env.drvr.reset();
env.drvr.drive(100);
end
endprogram
(S)Download the phase 1 files:
ones_counter.tar
Browse the code in ones_counter.tar
(S)Run the simulation:
your_tool_command -f filelist test_1.sv
your_tool_command -f filelist test_2.sv
(S)Simulation Log Report:
DUT count is 0xxx ::SB count is 0000
DUT count is 0xxx ::SB count is 0000
DUT count is 0000::SB count is 0000
DUT count is 0000::SB count is 0000
*ERROR*DUT count is 1111::SB count is 0001
*ERROR*DUT count is 0111::SB count is 0001
*ERROR*DUT count is 0111::SB count is 0001
*ERROR*DUT count is 0111::SB count is 0001
http://testbench.in/ones_counter.tarhttp://testbench.in/CODE/ones_counter_top.sv.htmlhttp://testbench.in/CODE/ones_counter_top.sv.htmlhttp://testbench.in/ones_counter.tar -
7/28/2019 Verification Concepts
30/33
*ERROR*DUT count is 1011::SB count is 0010
*ERROR*DUT count is 1011::SB count is 0011
DUT count is 0011::SB count is 0011
DUT count is 0011::SB count is 0011
*ERROR*DUT count is 1101::SB count is 0100
VERIFICATION PLAN
The Verification Plan is the focal point for defining exactly what needs to be tested, and drives the coverage
criteria. Success of a verification project relies heavily on the completeness and accurate implementation of a
verification plan. A good plan contains detailed goals using measurable metrics, along with optimal resource usage
and realistic schedule estimates. Verification plan gives an opportunity to present and review the strategy forfunctional verification before the verification engineer have gone into detail to implement it. It also establishes
proper communication. Just imagine how it would be working with a multisite project and you have a query for
which you have to wait till the next day to see the answer in email and they just call you while you are in sleep. It
also gives an idea about the areas that are going to be difficult to verify for taking necessary steps. It is used to
determine the progress and completion of the verification phase of verification. Verification Planning should start
early with system/architecture evaluation phase. Once the functional spec is given to the verification team, they
will start its development.
A verification plan could come in many forms, such as a spreadsheet, a document or a simple text file. Templates
are good if continually used in your company as it makes common interface for information, reviewer know where
to look for certain information even in a huge document that he wants to know at this moment, because different
reviewers want different infomation in different moments.
Generally Verification plan development is divided in two steps: What to verify and How to verify?
Step one: What to Verify?
list of clearly defined features-to-verify. This is called feature extraction phase.
Step two: How to Verify?
After defining what exactly need to be verified, define how to verify them.
Verification Plan Contains The Following:
Overview
Resources, Budget and Schedule
Verification Environment
System Verilog Verification Flow
Feature Extraction
Stimulus Generation Plan
Checker Plan
Coverage Plan
Details of reusable components
-
7/28/2019 Verification Concepts
31/33
Overview
This section contains description of the project, what specification is followed, what languages and methodology
are used. Information related to the HW blocks, SW blocks and HW/SW interaction is outlined.
Feature Extraction
This section contains list of all the features to be verified. Generally each feature is associated with
1. Unique Name Id
2. Description
3. Expected result
4. Pointer to the specification
5. priority.
The "Unique name ID" should be descriptive in nature to understand the nature of the feature and should beunique. For example, ___
Some points on how extraction the features:
Read the MRD, System Specification, Macro and Micro Hardware Specification.
Go through the designer notes/presentations.
Annotate each line/paragraph specifying a functional item e.g., read/write a register
Annotate each line/paragraph specifying a multiple-functional items e.g., steps required to set and cause an
interrupt .
-
7/28/2019 Verification Concepts
32/33
Identify all RTL configurations
Identify interfaces and related protocols across interface.
Identify standards compliance requirements, list corner cases.
Create a list of illegal scenarios to verify.
Create a list of exceptions to verify.
Create a list of operation sequences to verify e.g., interrupt followed by breakpoint etc.Create a list of things to try and break the machine.
Take advantage of existing plans.
Use points from compliance checklist for standard protocols.
Get information about the Common tests for all chips.
Get review by a number of people, usually very experienced engineers. Better if you get reviewed by
Architects, micro Architects, Leads, verification engineers, RTL Designers, software designers, marketing team and
other team members.
Resources, Budget And Schedule
This section contains details of man power required and schedule for each phase of the verification. Informationabout the tools which are used for simulation, debugger and bug tracking are listed.
Verification Environment
A detailed TestBench architecture is essential for a robust verification environment. In this section describe the
topology, about each component of the TestBench, special techniques that are used, IPs, Reused blocks, new
blocks, and guidelines on how to reuse the TestBench components. For example if you think upfront about error
injection, configuration, component communication, callbacks etc, you can provide hooks to do those.
System Verilog Verification Flow
Details about each level (block, sub-system, system) and phases (RTL, gate) of verification are mentioned.
Stimulus Generation Plan
The stimulus generation section contains information about different types of transactions, sequences of
transactions and various scenarios generated as per the specification.
Each point is associated with
1. Unique name ID
2. Stimulus to be generated for driving into the DUT
3. Configuration/constraints Information
-
7/28/2019 Verification Concepts
33/33
Checker Plan
This section will explain the expected result checking's in the TestBench. This can be done in monitor/checker.
Various fields associated with each of the point are:
1. Checker Unique name
2. Unique Feature ID(Defined in Feature plan)
3. Checker Description
Coverage Plan
The coverage section explains the functional coverage of the features. A functional coverage plan should be built
to help implement coverage points in the verification environment. Genarally it would be better if the coverage
block is brocken based on the design logical blocks. It will list the various Coverage groups and assertion.
Various fields of the coverage plan section are:
1. Coverage Group
2. Coverage Description
3. Coverage name
4. Unique name ID (Defined in feature plan)
5. Cover point (Items/cross/transition/assertion)
6. Coverage goal
Details Of Reusable Components
This section contains the details of reusable components and the description about their usage.