May 4, 2011 1

Using Formal Tools to Improve the Productivity of Verification at

STMicroelectronics

James Pascoe1000 Aztec West, Almondsbury, Bristol, UK

ChipEx 2013 – Tel Aviv, IsraelMay 1, 2013

May 1, 2013 2

Overview• Who are we:

– The ‘CPU’ part of ‘CPU/GPU’ in TR&D (ST Bristol)– We develop ARM based CPU sub-systems for a range of SoCs

• Organisation:– System-level functional verification (Noida)– Block-level activities (Bristol)– Low-power and DFT verification (Grenoble)

• Formal evaluation:– Sensor Control Block (SCB) – block-level verification– Clock and Reset Manager (CRM) – block-level verification– Documentation driven point-to-point connectivity checking

May 1, 2013 3

Subsystem

May 1, 2013 4

Verification Strategy• Unit testing:

– Performed by Designers– Is this block ready to enter verification?– Designers typically implement HDL based test-benches

• Block-level:– Performed by Verification engineers– Does this block conform to its specification in isolation?– Typically verified using a constrained random approach

• System-level testing:– Point-to-point checks– Functional testing– Performance verification

May 1, 2013 5

Evaluating Formal• We evaluated a formal tool (Jasper) to determine its potential to

enhance our productivity. Our study had three aims:

1. To close verification projects with appreciably less effort than constrained random;

2. To promote greater use of assertions by encouraging designers to develop formal properties for their blocks;

3. To augment or replace legacy in-house flows with mature industry tools that reduce maintenance and other overheads.

May 1, 2013 6

Approach• Projects increase in complexity:

– Sensor Control Block• Digital memory mapped sensor block• Developed in Bristol• Constrained Random test-bench developed in parallel

– Clock and Reset Manager• Provides clock and reset sequencing in subsystem (complex)• Developed in Grenoble• Very good micro-architectural documentation but no functional specification

– Point-to-point connectivity checking• Flow developed to extract assertions from project specifications• Use Jasper to verify connectivity at the subsystem level• Also very useful in the context of low-power (e.g. UPF aware proofs)

May 1, 2013 7

Overview Sensor Control Block

Clock and Reset

Manager

Point-to-point connectivity

checkingConclusions

Sensor Control Block

May 1, 2013 8

Sensor Control Block• Provides a digital front-end to thermal sensors:

– Monitors chip temperature– Selected for its simplicity:

• Connects to thermal sensors• Samples sensors periodically• Generates warning interrupts

– Well specified and understood

• Used Jasper to check:– Registers– APB interface and protocol– Temperature functionality– Sampling periods– Interrupt properties

May 1, 2013 9

Results• Found 3 bugs:

– Found subtle problem with PREADY signal on APB interface• Not detected by Specman

– 1 RTL problem:• Inverted ‘or’ concatenation for sensor overflow / underflow bits

– 1 specification problem:• Registers listed as ‘Write Clear 1’ can be cleared with other values

• Validity:– Didn’t expect to find much wrong

• Designer had performed extensive unit testing prior to verification• Some previous Specman verification performed• Tried and tested components used in the development

May 1, 2013 10

Project Management• We need to answer the following questions:

– How complete is the verification?• Q: are there enough assertions to verify all features?• A: measure ‘out-of-coi’ coverage to find coverage holes

– Is the environment over constrained?• Q: are we masking bugs by constraining the environment too much?• A: measure stimuli coverage and check that all legal scenarios are covered

– How good are the bounded proofs for my block?• Q: do we require more depth in the search space?• A: use bounded coverage analysis

– How does the combination of formal and dynamic cover the design?• Q: how do I interpret Jasper coverage with constrained random coverage?• A: Jasper is working on merging results into other UCDB files

May 1, 2013 11

Coverage • Dead code analysis:

– Dead code due to RTL – 64 cover points• Branches that will not be taken during reset• 1 feature that was later deprecated (but masked with a tie-off in the RTL)• Register implementation uses generic code to minimise flop generation at synthesis• Justified exceptions with designer review

• Out of Cone of Influence:– 219 of 1000 coverage points were uncovered– > 150 uncovered points were due to the register model– Remainder due to P1500 signals:

• Feature added when Jasper work was close to sign-off• Later covered in Specman TB

– Exceptions justified by designer

• Bounded proof coverage – 100% covered

May 1, 2013 12

Overview

Sensor

Control

Block

Clock and

Reset Manager

Point-to-

point connectivity

checking

Conclusions

Clock and Reset Manager

May 1, 2013 13

Clock and Reset Manager• Generates clocks and resets to blocks within subsystem:

– Sequences actions to control transitions between operating points– Enables subsystem to be fully asynchronous– Provides an abstract interface to the SoC– Good micro-architectural documentation but difficult to verify against– Critical block and complex

• Verification approach:– Use Jasper to perform feature extraction

• Not possible with constrained random test-bench

– Prove critical properties using Jasper• Event sequences, specific timings etc.

– Develop Specman test bench in parallel• Compare approaches

May 1, 2013 14

Feature Extraction• Key problem: CRM functionality not well specified:

– Once ‘cmd_in’ rises ‘cmd_ack’ should follow after ‘N’ cycles– Once ‘cmd_in’ falls, ‘cmd_ack’ will follow after ‘M’ cycles– where ‘N’ and ‘M’ are not known …

• Solution – Use Visualise to evaluate the range of cycles the ‘ack’ follows– Write number of properties with different values for ‘N’– Find the lowest ‘N’ for which the property proves– Keep the assertion of the lowest proven ‘N’

assert: $rose(cmd_in) |-> ##[1:80] cmd_ack

– Define cover item for the Min ‘N-1’ for regression

cover: $rose(cmd_in) ##1 !cmd_ack [*80-1]

May 1, 2013 15

Visualize

May 1, 2013 16

Results• No RTL bugs were found. But:

– CRM was verified previously. Jasper was used to boost confidence – Specification is now well understood – solved a big problem– Used to bound overlap conditions on functional stimuli

• Exhaustive proofs on all functional features– 73 assertions – all fully proven– 88 covers – all covered– Not including test-mode

• Used coverage for:– Checking for over-constraints – none found– Ensuring that formal touched all functionality– More details about coverage to follow

May 1, 2013 17

Coverage• Dead code analysis:

– Dead code due to RTL – 4 cover points• Branches that will not be taken during reset or test mode• Due to synchronisers that have reset signals tied low• Designer could have replaced them with synchronisers without reset

– Dead code due to no test mode constraint: 54 cover points (58-4)• Test code that is not active in functional mode – designer reviewed and approved• Wanted to make sure that no functional branches were made dead by mistake

– Dead code due to reset: 63 cover points (121-58)• These are the (!resetn) branches• Reviewed and designer approved

– Dead code due to functional assumptions: 4 cases• Do we constrain away any valid branches?• 4 cases were discovered and approved as part of test mode

May 1, 2013 18

Coverage• Bounded Proof Coverage:

– Initially, not all of the assertions converged• Out of 55 assertions, 28 did not converge• Lowest bound for non-converging assertions is: 774 cycles

– Used bounded proof coverage• Measured coverage within the bounds reached by non-converging properties

– Bounded proof coverage tells us that bound is acceptable because all cover items are covered in less than 268 cycles

– Subsequently, more advanced engines were used and all properties converged

• Out of Cone of Influence:– 46 branches were detected as out of cone of influence– All branches relate to DFT

May 1, 2013 19

Overview

Sensor

Control

Block

Clock and

Reset Manager

Point-to-

point connectivity

checking

Conclusions

Point-to-Point Connectivity Checking

May 1, 2013 20

Point-to-Point Connectivity• Point-to-point connectivity checking:

– Once everything is verified at the unit and block-level …– Point-to-point connectivity checking provides a first check that blocks

have been assembled into the subsystem correctly– Eliminates wiring errors - useful before functional system testing

• Developed a documentation driven point-to-point flow:– 2564 reference connections generated from key project document– Point-to-point checking flow setup in 1 day (with help of Jasper) – All 2564 properties have been proven

• Useful for low-power:– Can check connectivity rules for changing power states– Rule validity can be tied to power states

May 1, 2013 21

Overview

Sensor

Control

Block

Clock and

Reset Manager

Point-to-

point connectivity

checking

Conclusions

Conclusions

May 1, 2013 22

Conclusions• Formal delivers!

• Found bugs quickly:– SCB required 6 m/w of effort to build a Constrained Random test-bench– The formal approach found almost all the bugs within the first week

• Potential for designer involvement is high:– Designers found Jasper easier to learn than other formal tools– Jasper was used to develop assertions and to perform unit-testing

• Unit-testing was then reused in the block-level verification

• Enabled us to verify blocks with incomplete specifications:– Used formal to test implicit assumptions on the CRM– Provided results that could be quickly verified by designers

May 1, 2013 23

Quality Improvements• Insight can be captured as properties when:

– Interpreting specifications– Making assumptions

• Formal provides a good way of stimulating early designs:– No need for HDL test benches that are discarded once verification starts– Formal is a great way of performing unit-testing– Assertions are reused throughout the life-cycle of the IP

• Certain aspects can only be verified formally:– Absence of deadlock– Liveness properties

• Properties automatically validate system level behaviour:– Detects system-level inconsistencies in specifications and assumptions

May 1, 2013 24

The Value of Formal• Unit / block-level:

– Significant time savings– No need to develop complicated Constrained Random test-benches– Potential for property reuse is high– Provides feedback early in the design process– Allows designers to stimulate designs without having to write HDL TBs– Properties add value to subsequent design phases– Feature extraction can be performed

• System-level:– Point-to-point checking is easy to setup and provides good insights– Low level properties assist in verifying system-level assumptions– Absence of dead-lock / liveness

May 1, 2013 25

Overview

Sensor

Control

Block

Clock and

Reset Manager

Point-to-

point connectivity

checking

Conclusions

Questions