Functional coverage plan in order to be effective has to take into consideration two specifications. They are:

• Requirements specifications
• Micro-architecture implementation specifications

In principle verification teams readily agrees to above. But when it comes to defining the coverage plan it does not reflect.

In general functional coverage itself receives less attention. On the top of that among the above two, requirements specifications coverage ends up getting lion share. Micro-architecture implementation coverage gets a very little attention or almost ignored.

For some it may look like an issue out of nowhere. They may argue, as long as requirements specifications are covered through functional coverage, micro-architecture coverage should be taken care by code coverage.

Why do we need functional coverage for micro-architecture?

We need functional coverage for micro-architecture specifications as well because interesting things happen at the intersection of requirements specification variables and micro-architecture implementation variables.

Many of the tough bugs are hidden at this intersection and are caught very late in verification flow or worse in silicon due to above thought process.

How? Let’s look at some examples.

Example#1

Design with pipeline, combinations of states across stages pipelines is an important coverage metric for the quality of stimulus. Just the interface level stimulus of all types of inputs, will not be able to provide idea about whether all interesting states combinations are exercised for the pipeline.

Example#2

Design with series of FIFOs in the data paths, combinations of FIFO back pressures taking place at different points and with different combinations is interesting to cover. Don’t wait for stimulus delays to uncover it.

Example#3

Design implementing scatter gather lists for communication protocol, not only the random size of the packets are important but the packet sizes colliding with the internal buffer allocation sizes is very important.

For example let’s say standard communication protocol allows maximum payload up to 1 KB. Internally design is managing buffers in multiples of 256 bytes then multiple packets of size less than or equal to 256 bytes or multiples of 256 bytes are especially interesting to this implementation.

Note here from protocol point of view this scenario is of same importance as any random combinations of the sizes. If the design changes the buffer management to 512 bytes the interesting packet size combinations changes again. One can argue constrained random will hit it. Sure it may but probabilistic nature can make it miss as well. If it misses its expensive miss.

Covering and bit of tuning constraints based on the internal micro-architecture can go long way in helping find issues faster. Note this does not mean not to exercise other sizes but pay attention to sensitive sizes because there is higher likelihood of hard issues hiding there.

This interesting intersection is mine of bugs but often ignored in the functional coverage plan due to following three myths.

Myth#1: Verification engineers should not look into design

There is this big taboo that verification engineers should not look into the design. Idea behind this age-old myth is to prevent verification engineers from limiting the scope of verification or getting biased from design.

This forces verification engineers to focus only on the requirements specification and ignore the micro-architecture details. But when tests fail as part of debug process they would look in to the design anyway.

Myth#2: Code coverage is sufficient

Many would say code coverage is sufficient for covering micro-architecture specification. It does but it does it only partially.

Remember code coverage is limited as it does not have notion of time. So concurrent occurrence and sequences will not be addressed by the code coverage. If you agree above examples are interesting then think is code coverage addresses them.

Myth#3: Assertion coverage can do the trick

Many designers add assertions into the design for assumptions. Some may argue covering these is sufficient to cover micro-architecture details. But note that if intent assertion is to check for assumption then it’s not same as functional coverage for implementation.

For example let’s say we have simple request and acknowledgement interface between the internal blocks of design. Let’s say acknowledgement is expected within 100 clock cycles after request assertion. Now designer may place assertion capturing this requirement. Sure, assertion will fire error if acknowledgement doesn’t show up within 100 clock cycles.

But does it cover different timings of acknowledgement coming in? No, it just means it has come within 100 cycles. It does not tell if acknowledgement has come immediately after request or in 2-30 clock cycles range or 30-60 clock cycles range or 61-90 clock cycles range or 91-99 clock cycles range and exactly at 100 clock cycles.

However there are always exceptions, there are few designers who do add explicit micro-architecture functional coverage for their RTL code. Even there they restrict scope to their RTL sub-block. The holistic view of complete micro-architecture is still missed here.

Micro-architecture functional coverage: No man’s land

Simple question, who should take care of adding the functional coverage requirements of the micro-architecture to coverage plan?

Verification engineers would argue we lack understanding of the internals of the design. Design engineers lack understanding of functional coverage and test bench.

All this leads to micro-architecture functional coverage falling in no man’s land. Due to lack clarity on responsibility of implementation of micro-architecture functional coverage it’s often missed or implemented to very minimal extent.

This leads to hard to fix bugs only discovered in silicon like Pentium FDIV bug. Companies end up paying heavy price. Risk of this could have minimized significantly with the help of micro-architecture functional coverage.

Both verification and design engineers can easily generate it quickly using library of built-in coverage models from our tool curiosity.

What started off, as “PHY Interface for the PCI Express Architecture” was soon promoted to “PHY interface for the PCI Express, SATA and USB3.1 and SATA”. It was primarily designed to ease the integration of digital MAC Layer with the mixed signal PHY.

Today as of October 2017 latest publicly available PIPE specification is version 4.4.1. All the waveforms and pictures are sourced from this specification. If we go back 10 years in time to 2007, PIPE specification was at version 2.0.

Version 2.0 was for PCI express only. It had only 5 contributors and 38 pages. Whereas version 4.4.1 has 32 contributors and 114 pages. It supports not only PCI express, SATA and USB3.1 as well. Version 4.4.1 has had 5x growth in contributors and 3x growth in number of pages compared to version 2.0.

It’s also indication of rise in complexity of PCI express as well. PCI express has been one of the leading high-speed serial interface technologies. Hence there are multiple IP companies helping with the adoption of the technology. Apart from Big 3 there are also many vendors who provide the PCI express IP solutions. Some provide both the MAC layer and PHY while other provide one of them and partner with complementary vendors to provide the complete solution.

PCI express even at the PIPE interface level has quite a bit of configurability. This comes out in terms of width, rate and PCLK frequencies. Also some companies do make certain level of customization to this interface to support custom features.

To accelerate the simulation speed many verification environments for the PCI express controllers support both the PIPE and serial interface. Parallel PIPE interface runs faster than serial PHY interface. PHY would also require mixed signal simulations as well to verify the analog parts in it.

Considering the complexity there is lot of focus on verifying the MAC layer and PHY layer independently. This creates challenge as to what to verify when they are put together. Obviously it’s not practical to run again all the PHY and MAC layers tests put together on integrated design.

Vendors would make attempt to verify all configurations but do all the configurations get equal attention is something difficult to assess.

When the PCI express IPs are bought from the 3^rd party vendors it’s a challenge as to what to cover for integration verification.

PIPE interface coverage can be one of the very useful metric to decide on what tests to run for integration verification. This becomes even more important if the digital controller and PHY are sourced from different vendors.

One can say simple toggle coverage of all the PIPE signals should be sufficient to ensure the correct integration. Yes this is necessary but not sufficient.

Following are some of the reasons why just toggle coverage will not suffice.

• Toggling in all the speeds

Some of the signals of PIPE interface are used only in specific speeds of operation. So it’s important to check for the toggle in appropriate speed. Simple toggle coverage will not indicate in which speed the signals toggled.

Some examples are link equalization signals of the command interface signals such as RxEqEval, RxEqInProgress or 130b block related signals such TxSyncHeader, RxSyncHeader are only applicable for the Gen3 (8 GT/s ) or Gen4(16 GT/s) speeds etc.

• Transition or sequence of operations

Some of the signals instead of just individual toggle coverage require transition or sequence of events coverage to confirm the correct integration.

Some examples are six legal power state transitions, receiver detection sequence, transmitter beacon sequence etc.

• Variation of timing

Some of the signals can be de-asserted at different points of time. There can be multiple de-assertion or assertion points that are valid. Based on the design it’s important to confirm the possible timing variations are covered.

For example following waveform RxEqEval can de-assert along with PhyStatus de-assertion or later.

• Concurrency of interfaces

PIPE provides 5 concurrent interfaces, transmit data interface, receive data interface, command interface, status interface and message bus interface.

Simple concurrency like transmit and receive taking place at same time cannot be indicated by the toggle coverage.

• Combinations

Some signals, which are vectors, may not have all the values defined to achieve the toggle coverage.

LocalPresetIndex[4:0] for example has only 21/32 valid values. Here toggle can indicate connection but whether both the digital controller and PHY can handle all combinations of values together is not confirmed.

Also behavior of some key signals in different states needs to be confirmed to ensure the correct integration. TxDeemph[17:0] values will have to be covered based on rate of operation as toggle of all bits may not be meaningful.

• Multiple lanes

PIPE interface allows certain signals to be shared across multiple lanes. In multi lane case combinations of the lanes active coverage becomes important.

• Multiple times

Some of events or sequences are not sufficient just covered once. Why?

For example in the following waveforms the InvalidRequest must de-assert at the next assertion of RxEqEval. So multiple RxEqEval are required to complete the sequence of the InvalidRequest

Putting all these together results in functional coverage plan containing 88 cover items.

With our configurable coverage model we can easily customize and integrate all the above covergroups in your verification environment.

We offer service to tell you where your verification stands from PIPE integration coverage point of view.

According to wikipedia Pentium FDIV bug affects the floating point unit(FPU) of the early intel Pentium processors. Because of this bug incorrect floating point results were returned. In December 1994, Intel recalled defective processors. In January 1995, Intel announced “a pre-tax charge of 475 Million against earnings. Total cost associated with the replacements of the flawed processors.

What went wrong here?

According byte.com web page archive, Intel wanted to boost the execution of their floating-point unit. To do that they moved from its traditional shift-and-subtract division algorithm to new method called SRT algorithm. SRT algorithm uses a lookup table. Pentium’s SRT  lookup table implementation is matrix of 2048 cells. Only 1066 of these cells actually contained the valid values. Due to issue in script loading the lookup table 5/1066 entries were not programmed with valid values.

SRT algorithm is recursive. This bug leads to corruption only with certain pairs of divisors and numerators. The “buggy pairs” are identified. They always lead to corruption. At its worst, the error can rise as high as the fourth significant digit of decimal number. Chances of this happening randomly, is about 1 in 360 billion. Usually the error appears around the 9^th or 10^th decimal digit. The chances of this happening randomly, is about 1 in 9 billion.

Isn’t randomization on inputs sufficient?

Here you can see even if we had constrained random generation on divisor and numerators how low is probability of hitting the buggy pair. Even if we create the functional coverage on the inputs how low is probability that we would specifically write coverpoints for the buggy pair.

Of-course now we have formal methods to prove correctness of such computationally intensive blocks.

We are already convinced that constrained random hitting on this buggy pair has very low probability. Hence even if it hits this case it may take long wall clock time. Pure floating-point arithmetic verification approach alone is not sufficient.

For a moment if you think what could have helped maximize the probability of finding this issue in traditional simulation based verification?

Simulation world is always time limited. We cannot do exhaustive verification of everything. Even a simple 32-bit counter exhaustive verification would mean 2 ^ 32 = 4G of state space. That’s where the judgment of the verification engineers plays major role as to what to abstract and what to focus.

Story of curious engineer

Let’s say a curious verification engineer like you looked inside the implementation. He tried to identify the major blocks of design and spotted the lookup table. His mind suddenly flashed insight  floating-point inputs must use all valid entries of the lookup table. This is one more critical dimension for the quality of the input stimulus.

Let’s say he thought of checking that by writing functional coverage on input address of the lookup table. He thought of adding a bin per each valid cell of matrix.

This could have helped maximize the probability of catching this issue. There is a notion that verification engineers should not look into design as it can bias their thinking. While the intent is good but let’s not forget we are verifying this specific implementation of requirements. Some level of balancing act is required. It cannot be applied in purity.

(more…)

Functional coverage complements the code coverage by addressing its inherent limitations. This blog will help you understand what is the key value proposition of the functional coverage. This in turn will help you achieve the maximum impact from it.

Functional verification has to address verification requirements made up of both requirements specifications and implementation.

Functional coverage’s primary purpose is finding out has the functional verification done its job well. The means used for functional verification to achieve the goal doesn’t matter in this context. It can achieve it with all directed tests or use constrained random or use emulation.

When the functional verification relies on the constrained random approach functional coverage helps in finding out the effectiveness of constraints. It helps in finding out the constraints are correct and they aren’t over constraining. But this aspect of the functional coverage has become so popular that the primary purpose of it has become overshadowed.

Functional coverage focuses on following three with possible overlap between each other to figure out whether the functional verification objectives are met:

• Randomization or stimulus coverage
• Requirements coverage
• Implementation coverage

In the following sections we will briefly discuss what each of these mean

Randomization or stimulus functional coverage

Uncertainness of constrained random environments is both boon and bane. Bane part is addressed with the functional coverage. It helps provide the certainty that randomization has indeed hit the important values that we really care for.

At a very basic level it starts with ensuring each of the randomized values are covering the their entire range of values possible. This is not the end of it.

Functional coverage’s primary value for the constrained random environments is to determine its effectiveness in randomization. Functional coverage quantifies the effectiveness of randomization. This does not directly say anything about effectiveness or completeness of the functional verification. This just means we have enabler that can help achieve the functional verification goals.

It’s the requirements and implementation coverage that really measures the effectiveness and completeness of the functional verification.

Requirements functional coverage

Requirements specification view of functional verification is looking at design from the end application point of view.

Broadly it looks at whether test bench is capable of covering the required scope of verification from application scenarios point of view. That includes:

• All key atomic scenarios in all key configurations
• Concurrency or decoupling between features
• Application scenario coverage
  • Software type of interactions:
    • Like polling versus interrupt driven
    • Error recovery sequences
    • Virtualizations and security
  • Various types of traffic patterns
  • Real life interesting scenarios like:
    • Reset in middle of traffic
    • Zero length transfer for achieving something
    • Various types of low power scenarios with different entry and exit conditions
    • Various real life delays(can be scaled proportionately)

Bottom line is requirement specification functional coverage should be able to convince someone who understands requirements that design will work without knowing anything about implementation. This is one the primary value that functional coverage brings over the code coverage.

Implementation functional coverage

Implementation related functional coverage is highly under focused area. But remember it’s equally important one. Many verification engineers will fall in trap that code coverage will take care of it. Which is only partially true.

The implementation means the micro-architecture, clocking, reset, pad or any other analog components interface related aspects.

Micro-architecture details to be covered include internal FIFOs becoming full and empty multiple times, arbiters experiencing various scenarios, concurrency of events, multi-threaded implementation, pipelines experiencing various scenarios etc.

Clocking coverage is whether all the specification defined clock frequency ranges are covered, for multiple clock domains key relations between the two clocks, special requirements such as spread spectrum, clock gating etc.

For resets whether external asynchronous resets at all the key states of any internal state machines. For multi-client or multi-channel where they are expected to operate independently if one is in reset whether other can make progress etc.

Pads and analog blocks interface coverage is very critical. Making sure all the possible interactions is exercised whether effects can be seen or not in the simulation is still important.

Combination of the white box and black box functional coverage addresses both of the above.

We have heard technical debt but what is this functional coverage debt? It’s on the same lines as technical debt but more specific to functional coverage. Functional coverage debt is absence or insufficient functional coverage accumulated over years for silicon proven IPs.

Can we have silicon proven IPs without functional coverage? Yes, of-course. The designs have been taped out successfully without coverage driven constrained random verification as well. Many of the implementations, which are decades old and still going strong, have been born and brought-up in the older test benches.

These were plain Verilog or VHDL test benches. Let’s not underestimate the power of these. It’s not always about swords but also about who is holding them.

Some of these legacy test benches may have migrated to the latest verification methodologies and high-level verification languages(HVL). Sometimes these migrations may not be exploiting the full power the HVLs offer. Functional coverage may be one of them.

Now we have legacy IPs, which have been there for some time and have been taped out successfully multiple times. Now mostly undergoing bug fixes and minor feature updates. Lets call these as silicon proven mature IPs.

For silicon proven mature IPs is there a value in investing for full functional coverage?

This is tough question. The answer will vary case-by-case basis. Silicon proven IPs are special because they have real field experience. It has faced real life scenarios. Bugs in this IPs have been found over years the hard way through the silicon failure and long debugs. They have paid high price for their maturity. They have learnt from their own mistakes.

Bottom-line is they have now become mature. Irrespective of whether proper test cases were written for the bugs found in the design or not after bug discovery, now they are proven and capable of handling various real life scenarios.

So for such mature IPs when the complete functional coverage plan followed by their implementation is done the results can be bit of surprise. Initially there can be many visible coverage holes in verification environment. This is because such IP may have relied on silicon validation and very specific set of directed test scenarios to reproduce and verify the issue in test bench.

Even when many of these coverage holes are filled with the enhancement to test bench or tests, it may not translate to finding any real bugs. Many of these hard to hit bugs have been already found hard way with silicon.

Now the important question, so what is the point of investing time and resources for implementing functional coverage for these?

It can only be justified by calling it paying back technical debt or functional coverage debt in this case.

Does every silicon proven mature IP need to pay back the functional coverage debt?

It depends. If this IP is just going to stay as is without much of change then paying debt can further pushed out. It’s better to time it to discontinuity. The discontinuity can be in the form of major specification updates or feature updates.

When the major feature upgrades takes place the older part of the logic will interact with the newer parts of the logic. This opens up possibility of new issues in both older part of the logic, new logic and cross interaction between them.

So why not just write functional coverage for the new features?

When there isn’t functional coverage et all there is lack of clarity on what has been verified and to what extent. This is like lacing GPS co-ordinates about current position. Without clearly knowing where we are currently it’s very difficult to reach our destination. When it’s done in patchy way the heavy price may have to be paid based on the complexity of the change. For cross feature interaction type of functional coverage the existing feature functional coverage is also required.

Major updates are right opportunity to pay the functional coverage debt and set things right. Build the proper functional coverage plan and implement the functional coverage ground up.

There will be larger benefits for newer features immediately. For older features functional coverage will benefit the future projects on this IP as it will serve as reference about quality of current test suite. Even when the 100 % functional coverage is not hit, there is clarity on what is being risked. Uncovered features would be informed risk rather than gamble.

Conclusion

For silicon proven IPs the value of functional coverage is very low if the IP is not going to change any more. What was taped out just stays, as is then there is no need to invest in functional coverage.

If the IP is going to change then the quality achieved is no longer guaranteed to be same after the change

• With the functional coverage it ensures the functional verification quality carries over even with changes in design
  • Ex: A directed test that was creating scenario for certain FIFO depth may not create the same when FIFO depth is changed
    • Unless this aspect is sensitive in code coverage this can get missed without functional coverage in next iteration
  • Also for new feature updates it may be difficult to write functional coverage unless there is some base layer of functional coverage for existing features to cover the cross feature interaction

For silicon proven IPs the functional coverage implementation is paying of “technical debt” in that project but for new projects

• Provides ability to hold on to same quality
• Provides the base layer to build the functional coverage for new features to utilize its benefits from early on