Verification: A Crisis of Confidence and How to Resolve It

Part 3

In the previous two articles in this series, I described the verification crisis and explained why it has come to be. First, I defined a verification meta model as a way to describe the key components of this problem and laid out ideas on alleviating this crisis. In the last article, I discussed that we need to find out how to obtain efficient DV flows and what aspects of planning and process affect this. I covered the role of specifications and requirements, the role of a verification strategy and verification plan.

In the final post in this three-part series, I will describe the challenges in optimizing the biggest contributor to verification cost: the Debug, and the most high-valued item, high-quality Sign-off and Review. 

Debug

1. What is the debug process used in the project?

• Does the debug process mandate the use of best-in-class debuggers and expect engineers to know how to drive them without providing additional training?
• Debug is often touted to consume about 70% of the overall design verification time, therefore, it makes sense for us to discuss this as part of the meta model. While the selection of a particular debugger may be a matter of personal choice—and the world will remain divided on the topic of which is the best debugger—what is not a debatable topic is that DV engineers need good hands-on training in debugging.
• Shortening the debug cycle depends not only on being able to use the debugger, but also on the engineer’s knowledge of what is being debugged, familiarity with specifications, and design intent. That said, being able to go from a point of failure to the root cause analysis is most certainly a skill that can be taught, and training programs must be put into place to teach and ascertain the quality of debug. Too often, I hear from management that engineers take a long time to debug. When you ask the engineers in detail about this task, it turns out that they may not be well trained in using the debugger.

2. When a failing debug trace is found by an engineer, what is he/she supposed to do?

• This is something that is often left unspecified in a verification process. When is a failing trace a real design bug? What is the process that a verification engineer needs to follow to establish this? In the good old days, when design and verification engineers were co-located in the same office, this was not an issue, as the designer would either be verifying her own design or could simply look over the shoulder of the verification engineer. The whole debug process is a lot more challenging today: both designs and test benches have become complex and the design and verification teams are often thousands of miles apart, located in different time zones. A good debug process should establish some ground rules. One of them is a process whereby a failure is analyzed locally by the verification engineer in a step-by-step manner, ensuring that the failure is not due to missing constraints. This can then be manually triaged and reviewed by another senior verification lead to bring the discussion to a point where the failing trace is much more likely to be a design bug and can then be assigned to a designer for review. I have often found that just picking a phone and discussing the failure with the designer is healthy, saves time on both sides, and provides valuable insights in adequate constraining of the design, especially when the specifications were not provided up-front or were ambiguous. Moreover, this has another additional value-add: it fosters respect on both sides of the D&V line. By following a process where both design and verification teams can interact freely and work as a single team in root-causing the failure, the focus can remain on solving the problem, rather than playing the blame game.

Signoff and Review

1. Do you have a process of reviewing verification milestones?

• Most people would agree that there are primarily two ways of closing out verification: one is anecdotal, done by bug tracking and another is analytical, done by computing coverage metrics. We will come to both of these later, but it is worth highlighting a third way of assessing verification closure, and that is through a review process. I call it a “process” because one needs it for all aspects of design and verification, not for verification alone. For example, a design review process describes how often the design code is manually reviewed; a verification review process will summarize the key verification milestones and a process of reviewing verification strategy, verification plans, verification code review, review of coverage results, and review of bug tracking reports. Each of these reviews describes when they are carried out, by whom, and by using a templated list of questions that needs to be answered by the verification engineer and needs to be reviewed independently by a third person—not a designer or the verification engineer—who has the knowledge to review the overall status of the verification.

2. Coverage

• There are several ways of calculating different forms of coverage metrics, starting from the easy structural coverage that includes code coverage, toggle coverage, and FSM coverage, all the way to the more complex, but also more meaningful, functional coverage, which is often considered as semantic coverage.

• Coverage means an analysis of the design to understand what features of the design’s functionality have been covered by the verification. While the features take priority, the other forms of coverage that are related to well-formedness of the design are also meaningful. These include the structural coverage metrics that are part of the metric-oriented view. Though these metrics are a fairly good indicator of the quality of verification, the main problem is that these are only as good as the cover groups or cover properties, which are only as good as the functional coverage specification, which in turn is only as good as the specifications and requirements. Moreover, the way this analysis is done is by identifying if it was possible for a design feature to be exercised at least once, or if a certain combination of events was reachable by a stimulus countably many times. This does not establish whether the design is guaranteed to have a certain behavior, as this analysis is done on top of simulation test benches—by itself, it makes no guarantees. Of course, if formal verification has been used, then there are stronger guarantees that can be made on the quality of checks, detection of over-constraints, and even identification of corner case bugs through coverage.

Summary

In this three-part series, I posited how the verification crisis has come to be and explored the factors that exacerbate it. I described the verification meta model as the vehicle by which we can obtain high quality results from verification. I outlined the four main pillars of this model and described some scenarios in which these interact in an unproductive manner. I then went on to highlight how this can be improved by good processes and emphasized that clear plans for training and methodology development can make a difference. The key outcome of this is an efficient DV flow that can be deployed consistently in engineering teams by DV engineers.

This article was first published on the Tech Design Forum. It is reproduced here in its original form for informational and archival purposes, with appropriate acknowledgment to the original publisher.