system on chip (soc) functional verification · testing using functional verification regression...

Technology Seminar © 1998 TransEDA Ltd 1

TransEDA Technology Seminar

System on Chip (SOC)

Functional Verification

John Perry and David Dempster

Technology Seminar © 1998 TransEDA Ltd. 2

Agenda

■ Introduction■ Overview of Reuse Methodology Manual■ Review of the Design Process■ Testing Strategies■ Testing using Functional Verification

● Unit Testing● Integration and System Testing

● Regression Testing● Test Suite Optimization

■ Users’ Experiences of TransEDA’s tools■ Benefits Summary■ Recent and Current Developments■ Information Resources & Contact Points■ Question and Answer Session


Objectives • Introduction to RMM

(Reuse Methodology Manual)

• To review typical development sequences

• To review typical testing strategies

• Describe improved testing strategies

• Outline the benefits obtainable

• Relate some Users’ Experiences

• Explain recent & current developments

• Question and Answer session


Overview ofReuse

MethodologyManual Presented

by

John Perry


Review ofthe Design

Process

Requirements PhasePlanning Phase

Development PhaseVerification Phase

Rollout Phase


DevelopmentPhasesConcept

Good

GenerateBehavior

GenerateTest bench

Simulate Synthesis

Bad

Place&

Route

Sign Off

Bad

Review of theDesign Process


Source: Tandem Computers in IntegratedSystem Design, January 1997

Time (Effort)

Bug

Rat

e

Period 1 Period 2

Ship ?

Bug

Rat

e


Bug Ratedetection

withoutcode

coverage

Time and Effort



Time (Effort)

Bug

Rat

e

Period 1 Period 2

Ready to Ship

Bug

Rat

e


Bug Ratedetectionwith codecoverage



Time (Effort)

Bug

Rat

e

Period 1 Period 2

With Code Coverage

Without Code CoverageReady to

Ship

Bug

Rat

e


Comparisonof Bug Rate

detection

Ship?



DevelopmentPhases

Time

Cos

t per

Bug

The cost of finding and rectifying abug increases exponentially during

the development cycle



DesignEngineers

VerificationEngineers

FunctionalVerification

Introducingsome control

Functional Verification can help torestore communication and controlbetween engineering staff.

Brings the test engineers back into thedesign process.


Why Verify?

■ Statistics show that coding, done well,has 1 - 3 defects per 100 statements

■ Studies indicate that Inspections canuncover only 60% of Total Defects

■ Studies show testing consumes at least50% of the Product’s Labor Costs

■ Few Engineers like testing!!■ 40% design time■ 60% testing time



Why Verify?

Example

Assume 2 errors per 100 lines of code

2400 errors in 120K lines of code

If a success rate of 90% could be achieved,240 errors

would pass through the verification stage.

Could you afford to accept this???



TestingStrategies

■ Design Reviews

■ Testing using Functional Verification■ Formal Verification■ Requirements Tracing

■ Emulation/ Prototyping


Testingusing


■ Levels of Testing

■ Testing Techniques

We will consider...

...and show how quality and productivitycan be improved


Testing usingFunctional

Verification

Unit/Module testing

Integration orSub-system

testing

Systemtesting

- - - - - - Regression testing - - - - - -

Levels ofTesting


UnitTesting


Verification

FSM

Testing each unit in isolation is normallyquick and easy

ALU+, x, -,/

3:1

MUX


Integrationand Sub-

SystemTesting


Verification

FSM

Integration testing can concentrate onboundary and interface checking

ALU+, x, -,/

3:1

MUX

3:1

MUX

a b

OpCode


FSM 3:1

MUX

select

mc1

mc2

always @ (select[1] or select[2] or select[3])

begin

case (select)

3’b001: mc1=0; mc2=0;

3’b010: mc1=1; mc2=0;

3’b100: mc1=0; mc2=1;

default: “error message”

endcase

end

Glue logic provides Statement and Branch information

Toggle and Variable Trace Coverage on signals/wiresTesting using


Integrationand Sub-

SystemTesting



Verification

SystemTesting

As a system is made up of sub-systems,the sub-system testing strategy can alsobe applied at this level.

System testing can concentrate onboundary and interface checking

Any section could be an IP core.



Verification

RegressionTesting

Identifying which tests contribute to theoverall testing strategy and which testsduplicate other tests can saveconsiderable time and effort during theregression testing phase.

Which tests should I re-run?

Which tests can I remove for myregression runs?


Test_b1 Results_1

Test_b2

Test_b3

Test_b4

Results_2

Results_3

Results_4


Verification

RegressionTesting

usingCoverPlus

CoverPlus identifies tests that areduplicated and tests that are a sub-test of

other tests



Verification

RegressionTesting

usingCoverPlus

Analysis Simulation Results

X X

Time/Effort

Simulation time can be reduced duringregression testing by using CoverPlus



Verification

RegressionTesting

usingCoverPlus

Analysis Simulation Results

Time/Effort

Simulation time, during regressiontesting, can be reduced further usingmultiple workstations with CoverPlus

X X

X X

X

X X


TestingTechniques

■ Functional (or BLACK box testing)

■ Structural (or WHITE box testing)

■ Error-Oriented

■ Stress/Performance


Verification


FunctionalTesting


Verification

Testing restricted to checking that inputsand outputs are correct.


StructuralTesting

Out2:=1

Bus3 := Bus12 + 1

out200 := Sig2

Sig1==0

Sig4 := 0

Sig6 := 0

Bus3 := Bus12 + 2

Sig4 := 1

Sig5 := 0


Verification

Testing involves checking the logicalpaths and data conditions.


StateMachineTesting

Importance of State Machines

■ 30% of an ASIC is a state machines

■ Typical state machines have 28 states

■ Typical state machines have 50 transitions

■ There is increasing demand for good quality IP

■ System on Chip state machine interactions

Critically important to verify statemachines to avoid coding errors and

operational problems


Verification


Users’Experiences

withTransEDA’s

tools

Case Studiespresented

by

John Perry


BenefitsSummary ■ Application to Hardware Design

■ Benefits of HDL Coverage

■ How it fits in the Design Flow

■ Description of Functional Verification



Applicationto Hardware

DesignFront-endSimulation

Regression Testing

What testsdo I need?

Have all mysignals

been toggled?Have I done

enoughsimulation?

What is leftto test?

What are thedifficult

blocks to test?

■ Sign-off

Unit

Sub-System

Integration

SystemBenefitsSummary


■ Reduce Risk by Increasing Quality

■ Measure and Test in a controlled and thorough manner

■ Improve Productivity

■ Reduce Development Time by focusing effort on Untested Code

Benefits ofHDL

functionalverification

BenefitsSummary



fitsconvenientlyin the Design

Flow

ConceptGood

GenerateBehavior

GenerateTest bench

Simulate

Regression TestingCoverPlus

Synthesis

Bad

Bad

Place & Route

Sign off

Bad

BenefitsSummary Functional Verification

HDLCover VeriSure VHDLCoverStateSure


CodeCoverage

Metrics

■ Implementation Focused

● Statement/Line

● Branch● Condition/Expression

● Path

■ Functionality Focused

● Triggering

● Toggle● Signal/Variable Trace

BenefitsSummary


Easy toInstrument

BenefitsSummary

Select the source files

Select the coverage options for each source file

HDLCover

VeriSure

VHDLCover


QuantitativeMeasurement

ofImplementation

Simulation

BenefitsSummary

HDLCover

VeriSure

VHDLCover


StatementCoverageExample

BenefitsSummary

Highlights code-lines not executed

Shows number of times code-line executed

HDLCover

VeriSure

VHDLCover


BranchCoverageExample

BenefitsSummary

Shows number of times each branch has been taken

HDLCover

VeriSure

VHDLCover


Condition/Expression

CoverageExample

BenefitsSummary

Shows data conditions associated with decision paths

HDLCover

VeriSure

VHDLCover


PathCoverageExample

BenefitsSummary

HDLCover

VeriSure

VHDLCover

Identifies paths that have not been visited


TriggeringCoverageExample

BenefitsSummary

Shows activity (expressed as a percentage) ofthe items in the sensitivity list and highlights

signals that have not triggered

33%

HDLCover

VeriSure

VHDLCover


ToggleExample

BenefitsSummary

Shows toggle-count and signals that have no activity

HDLCover

VeriSure

VHDLCover


Signal TraceExample

BenefitsSummary

Displays combinations of user-selected signals

HDLCover

VeriSure

VHDLCover


StateSure

■ Static analysis to highlight coding errors

■ Automatic state diagram generation

■ Verification results annotated on diagram

■ Design hierarchy annotated with

verification results

■ State machines ranked by verification

results

BenefitsSummary


StateSure

BenefitsSummary


StateSure

■ Verify single state machine sequences

■ Verify pairs of state machine sequences

■ Verify paired state machine transitions

■ Verify multiple state machine interactions

■ StateSure supports Verilog, VHDL or both

■ StateSure on UNIX or WindowsNT

BenefitsSummary


StateSure

BenefitsSummary


CoverPlus

■ Sort the most productive test cases to be run first inthe regression suite

■ Simulate the most productive test cases

■ Remove unnecessary test cases

■ Improve bug fixing productivity

■ Plan a short regression suite for the highest coverage

in the shortest possible time

■ Find the difference between test cases, so thatadditions can be made to one test to eliminate a test

case from the suite

■ Find which test cases check specific blocks of code,so increasing productivity after bug fixes or changes

BenefitsSummary


Start CoverPlus and load simulation runs

BenefitsSummary

CoverPlus


Sort the simulation runs for full regression suite

Required Runs

Run has identical twin

CoverPlus

BenefitsSummary


Reduce requirement to statement only for daily regression

CoverPlus

BenefitsSummary


Compare test cases to find code covered by only one of the two tests

Find out what code is covered by List A

CoverPlus

BenefitsSummary


Locate which parts of the design are covered uniquely by one test

CoverPlus



Strategy

■ Adopting an Effective Strategywill Increases Your Productivity

■ Functional Verification Strategy:● Execute at least one Statement in 90% of source files

— add vectors and debug design

● Reach 100% of Statement and Branch coverage


● Reach 90% of Condition Coverage


● Reach 100% Condition Coverage

— optimize Regression Suite

— eliminate unnecessary tests

● Reach 100% coverage on other measurements

(Path, Toggle, Signal Trace, etc.)

BenefitsSummary


Recent

and

Current

Developments

Recent Developments (version 4.2)

• FEC (Focussed Expression Coverage)

• Enhanced command line options/switches

• StateSure

• CoverPlus

Current Developments (version 5.0)

• HDLCover (language neutral)

• NEW improved fast GUI

• Windows NT support

• VHDL-93 and Instance-by-Instance

Current Developments (version 5.1)

• Joint language product (Verilog & VHDL)


FocussedExpression

Coverage

Methodology that concentrates on choosingthe most effective set of test vectors for aboolean expression.

Saves time by identifying the missing orredundant test vectors.

Normally only requires N+1 vectors or 2Nvectors where N is the number of inputs.

Reduces simulation time.

HDLCover

VeriSure

VHDLCover


FocussedExpression

Coverage

FEC methodology is based on checkingthat an input can control the output signaland drive it to ‘0’ and ‘1’

abc

& Out

Although there are 8 combinations of inputpatterns, only the following 4 test vectorsare needed to fully exercise the circuit.

a,b,c = [1,1,1], [0,1,1], [1,0,1], [1,1,0]

This methodology is referred to as N+1 asnormally the number of test vectorsrequired is one more than the number ofinputs.

A detailed Application Note regarding theoperation of FEC can be found at:

www.transeda.com/info-japan/

HDLCover

VeriSure

VHDLCover


FocussedExpression

Coverage

Example of the diagnostic report from FEC

Example of missing test vector needed to check theoperation of input b1

HDLCover

VeriSure

VHDLCover


FocussedExpression

Coverage

The FEC methodology can be used to define the testvectors you should use to fully validate your expression.

Simply apply one test vector, via the test-bench, to themodule under test then use the diagnostic report toidentify the vectors that are actually needed.

Finally update the test-bench with the necessary testvectors and re-run the analysis.

Below is an example of a fully validated expression.HDLCover

VeriSure

VHDLCover


InformationResources

and

ContactPoints

A PowerPoint copy of this seminar together with theCustomer Case Studies, Application Notes on FocussedExpression Coverage and color TransEDA’s Product

Brochures can be found at:

www.transeda.com/info-japan/

TransEDA Ltdweb site: www.transeda.com

Sales and General Enquiries

[email protected]

John Perry [email protected]

David Dempster [email protected]


InformationResources

and

ContactPoints

Kanematsu Electronics Ltd.,Industrial Electronics Systems Division

[email protected]: +81-3-5250-6306

Fax: +81-3-5250-6039

system on chip (soc) functional verification · testing using functional verification regression...

Documents