Verification: A Crisis of Confidence and How to Resolve It

Part 2

How to Optimize the Verification Meta Model

As I outlined in the first part of this three-part series, a good verification meta model defines: the right combination of verification technologies and methodologies to be deployed, in the correct order, by an appropriately trained workforce of skillful, highly motivated DV engineers, with the goal of minimizing the risk of verification by hunting bugs before they reach silicon and/or customers.

The key value-add of the meta model is in outlining a good verification methodology, which describes an efficient design verification (DV) flow that can catch bugs as early as possible in the DV cycle (shift-left) and predict with high certainty that the expected quality levels will be met on schedule. In simple terms: quality won’t compromise productivity, and equally, productivity should not compromise quality.

To make this a reality, we need to focus on the key aspects of a good DV flow:

• Requirements and Specifications
• Verification Strategy and Plan
• Debug
• Signoff and Review

Read on for key questions to address in each of these areas.

 Design Verification Flow

Requirements and Specifications

1. Do we have good specifications? If yes, have they been reviewed for consistency?

• In many high-pressure engineering environments, good specifications do not exist, as implementation often takes priority over documentation. I’m not a big fan of this myself, but the practicalities mean that we need to strive for good specifications while we review the existing ones for consistency.
• A great way of resolving specification ambiguities is by describing block interfaces systematically using formal properties using SVA or PSL. This provides the additional benefit that these properties can be executed both in formal tools and in simulation and emulation. By executing these in formal tools, the user gets an additional benefit: without the need to write input stimuli, the DV engineer can start seeing results through failing properties or exhaustive proofs. If the failures are spurious, meaning that illegal stimuli have been driven into the design, the DV engineer can constrain these. This process opens up a dialog between the verification engineer and the designer as they explore detailed specification attributes on inputs of this block, which, when clarified, become the checks on the output of the driving block. Thus, the process of defining clear interfaces on the outputs of one block naturally helps to define the interfaces for the input driving block. If this process is diligently followed, then one gets not only nicely specified design interfaces for the entire design, but also a much clearer specification. The process of executing these properties finds bugs or even avoids bugs in the first place, as designers get in the habit of thinking more formally through the dialog process. An additional benefit of defining interfaces is that both designers and verification engineers get in the habit of writing formal properties, which then extends naturally to internal whitebox checks on the design block, thereby further exposing bugs in the design.

2. Are specifications being constantly updated?

• When specifications are formally specified on the interfaces, then the process of executing them daily on all design blocks means that if any design changes take place, the regressions on formal properties are likely to fail, highlighting the gap between implementation and specification updates.

3. Are these updates to specifications communicated clearly to the DV engineer?

• There are several ways of establishing communication between designers and verification engineers. This could be handled through emails, code check-ins, face-to-face meetings, whiteboard chats, or special events driven by either designers or verification engineers to review the status of verification. All of these are useful to a varying extent, but the one I like most is face-to-face meetings, or at least dedicated status meetings, which may or may not have to be physically face-to-face. In either case, a process by which both designers and verification engineers become used to writing interface specifications in a disciplined manner would make it much easier to organize such meetings and follow up.

4. If we don’t have time for good specifications, is there an alternate plan?

• When projects cannot afford to devote any time to encouraging a disciplined interface specification process, then we must ask ourselves: do we have an alternate plan? It could take the form of going halfway and describing only part of the specifications, scoping out key verification concerns and focusing on them. The problem with this incomplete effort is that any functional coverage specification needed later for sign-off would require a good specification anyway, and a halfway house is not the best option. Still, it is decidedly better than nothing!

Verification Strategy and Plan

1. Is there a verification strategy document that has been reviewed by architects, designers, and verification leads?

• I often come across verification projects where the strategy is not clearly scoped out. If you dig into this a bit more, you may find that the verification engineer is running a legacy test bench that someone else wrote, and this engineer is responsible for running tests, collecting coverage, and reporting this to the verification lead. The problem with this scenario is that he may not have even been asked to consider scoping out the details of why he is doing what he is doing. Is there even a defined strategy? How do we know that this test bench is even adequate in exposing bugs in what could be a newer version of a design? By following a process where a verification strategy document is a must, even when using a legacy test bench, the engineer would be looking at a prior strategy document and could seek to understand the approach taken in the past for the test bench. In turn, this can open up a dialog with designers and architects on what else would be of significance to watch out for when reusing the old test bench on a new version of the design. To me, this is a no-brainer: reviewing a three- to four-page strategy document is much easier than reviewing several thousand lines of test bench code to understand the scope of the adequacy of the test bench. Having a verification strategy document and a review process in place seems, therefore, to be low-hanging fruit that can add enormous value.

2. Do the verification strategy document and verification plan have sections on Plan B?

• In engineering, as in life, things often don’t go to the original plan. Having a Plan B can only be a good thing. Consider the case when a new block is being verified and the plan is to use constrained random as the main verification technology. Let us say that the block is so complex that it is conceivable that not all corner case bugs would be flushed out by constrained random. Is there a backup plan to find some of these with an alternate technology? For example, formal would be great at finding corner case bugs, especially deadlocks, and it may be part of the strategy and a plan to apply formal to those parts of the design where the risks are highest. The converse can be true as well: when formal is the main technology and it is expected that some of the proofs would be inconclusive, a good Plan B would be to use directed testing to focus verification on those parts of the design for which proofs are inconclusive.

3. Has the DV engineer used all the available technology— especially formal—to ensure that designs are brought up free from simple bugs such as lint errors, FSM issues, and deadlocks?

• This is an important point, as a lot of bugs that are simple to catch with formal are only picked up late in verification using dynamic simulation or even emulation. This leads to long debug times and even longer fixes. It also is the case that some of these bugs are very difficult to catch with simulation or emulation; this includes catching bugs in FSMs, deadlocks, dead code, or even redundant code (e.g., registers that are clocked but never read). Lint errors, often known as autochecks in many formal tools, are able to catch out-of-bounds violations automatically and easily, but these are much more difficult to catch later using simulation. This illustrates the importance of leveraging an appropriate combination of verification technologies.

4. Is the combination of verification technologies suitable for the project?

• I often come across scenarios where, due to prior experience, there is a preference for a particular verification technology, such as UVM for SoC verification. Though UVM is indeed a powerful verification paradigm, it may not be ideal for full SoC testing. Alternatives, such as directed testing, portable stimulus, or even formal, would be natural candidates. At the same time, while the use of directed testing would make sense for carrying out read and write checks from masters to slaves in the SoC, a more well-rounded testing in the presence of multiple masters might want to include some stalling and randomization, which is intrinsic to constrained random (UVM). Further, if you have a complex bus fabric in the SoC, neither directed testing nor constrained random is a good choice; in fact, the use of formal would be ideal to expose corner case bugs. After all, if the bus fabric deadlocks, then so will the entire SoC as a consequence. All of this discussion makes sense in the context of an SoC verification, but the choices would necessarily have to be different for IP-level verification. When it comes to verification technologies, one size most certainly does not fit all—as with clothing, tailored is better, and bespoke is best.

I hope that’s given you something to ruminate on as you consider your team’s current verification situation. In the third and final installment of this series, I’ll continue the discussion on the remaining two aspects of a good DV flow: the debug and the critical sign-off and review.

How does your team’s verification plan measure up to what I’ve begun to outline here? Have you identified any areas that need improvement? Share your thoughts with the comment widget below or on Twitter (@AshishDarbari).

This article was first published on the Tech Design Forum. It is reproduced here in its original form for informational and archival purposes, with appropriate acknowledgment to the original publisher.