VerifSudha

Tag: functional coverage

  • Functional coverage debt

    We have heard technical debt but what is this functional coverage debt? It’s on the same lines as technical debt but more specific to functional coverage. Functional coverage debt is absence or insufficient functional coverage accumulated over years for silicon proven IPs.

    Can we have silicon proven IPs without functional coverage? Yes, of-course. The designs have been taped out successfully without coverage driven constrained random verification as well. Many of the implementations, which are decades old and still going strong, have been born and brought-up in the older test benches.

    These were plain Verilog or VHDL test benches. Let’s not underestimate the power of these. It’s not always about swords but also about who is holding them.

    Some of these legacy test benches may have migrated to the latest verification methodologies and high-level verification languages(HVL). Sometimes these migrations may not be exploiting the full power the HVLs offer. Functional coverage may be one of them.

    Now we have legacy IPs, which have been there for some time and have been taped out successfully multiple times. Now mostly undergoing bug fixes and minor feature updates. Lets call these as silicon proven mature IPs.

    Silicon proven IPs
    Silicon proven IPs

    For silicon proven mature IPs is there a value in investing for full functional coverage?

    This is tough question. The answer will vary case-by-case basis. Silicon proven IPs are special because they have real field experience. It has faced real life scenarios. Bugs in this IPs have been found over years the hard way through the silicon failure and long debugs. They have paid high price for their maturity. They have learnt from their own mistakes.

    Bottom-line is they have now become mature. Irrespective of whether proper test cases were written for the bugs found in the design or not after bug discovery, now they are proven and capable of handling various real life scenarios.

    So for such mature IPs when the complete functional coverage plan followed by their implementation is done the results can be bit of surprise. Initially there can be many visible coverage holes in verification environment. This is because such IP may have relied on silicon validation and very specific set of directed test scenarios to reproduce and verify the issue in test bench.

    Even when many of these coverage holes are filled with the enhancement to test bench or tests, it may not translate to finding any real bugs. Many of these hard to hit bugs have been already found hard way with silicon.

    Now the important question, so what is the point of investing time and resources for implementing functional coverage for these?

    It can only be justified by calling it paying back technical debt or functional coverage debt in this case.

    Does every silicon proven mature IP need to pay back the functional coverage debt?

    It depends. If this IP is just going to stay as is without much of change then paying debt can further pushed out. It’s better to time it to discontinuity. The discontinuity can be in the form of major specification updates or feature updates.

    When the major feature upgrades takes place the older part of the logic will interact with the newer parts of the logic. This opens up possibility of new issues in both older part of the logic, new logic and cross interaction between them.

    So why not just write functional coverage for the new features?

    When there isn’t functional coverage et all there is lack of clarity on what has been verified and to what extent. This is like lacing GPS co-ordinates about current position. Without clearly knowing where we are currently it’s very difficult to reach our destination. When it’s done in patchy way the heavy price may have to be paid based on the complexity of the change. For cross feature interaction type of functional coverage the existing feature functional coverage is also required.

    Major updates are right opportunity to pay the functional coverage debt and set things right. Build the proper functional coverage plan and implement the functional coverage ground up.

    There will be larger benefits for newer features immediately. For older features functional coverage will benefit the future projects on this IP as it will serve as reference about quality of current test suite. Even when the 100 % functional coverage is not hit, there is clarity on what is being risked. Uncovered features would be informed risk rather than gamble.

    Conclusion

    For silicon proven IPs the value of functional coverage is very low if the IP is not going to change any more. What was taped out just stays, as is then there is no need to invest in functional coverage.

    If the IP is going to change then the quality achieved is no longer guaranteed to be same after the change

    • With the functional coverage it ensures the functional verification quality carries over even with changes in design
      • Ex: A directed test that was creating scenario for certain FIFO depth may not create the same when FIFO depth is changed
        • Unless this aspect is sensitive in code coverage this can get missed without functional coverage in next iteration
      • Also for new feature updates it may be difficult to write functional coverage unless there is some base layer of functional coverage for existing features to cover the cross feature interaction

    For silicon proven IPs the functional coverage implementation is paying of “technical debt” in that project but for new projects

    • Provides ability to hold on to same quality
    • Provides the base layer to build the functional coverage for new features to utilize its benefits from early on

  • Functional coverage for bug fixes

    What? We don’t even have any functional coverage and you want us to write functional coverage for bug fixes?

    Ah!.. That’s nice try but sorry we don’t have time for that. May be next project, when we have some extra cycles we will certainly revisit it.

    We are very close to meeting our code coverage targets. So I guess we are good here. We don’t have time for wishy-washy functional coverage.

    Fair enough. We hope you are aware of limitations of the code coverage.

    Oh, yeah, we are aware of those. Anything new?

    Let’s see.

    How about bugs? Are you finding any new bugs even after almost closing code coverage goals?

    Awkward pause for few moments, some throat clearing. Yes, we do.

    Well, we can consider the design gold standard till we don’t discover the bugs in it. We can all assume code coverage is good enough and RTL quality is great. But the moment we discover bug even after code coverage closure that assumption breaks down. We can no longer hide behind that.

    There is never a single cockroach

    Lets just presume emulation or SOC verification team reports a bug that should have been ideally caught at unit verification. A bug found in a very specific configuration and specific scenario. Should the unit verification just recreate that scenario in that specific configuration in unit verification and say it’s done?

    We can but first would like to quote my boss from early days of my career. Every time we reported saying we discovered and fixed a bug, he would say “there is never a single cockroach”. Just think back have you ever seen only one? There are always more if you look around. What does this mean in this context?

    Think about it. We can consider, RTL with 100 % code coverage as innocent till a bug is discovered in it. After the bug is discovered it’s guilty and has to face the trial. Thorough investigation must be performed.

    Even if you did not create a comprehensive functional coverage plan as part of initial verification planning now is time to rethink about it. No, no need to rush to create one now. That’s not going to help much. Typically the resource and time at this point are well spent in verifying. We don’t want them to go on systematic hunt for new weak areas with comprehensive functional coverage plan.

    Bugs are already hinting where there is weakness in design. Now the question is how do we use it to reduce the further risk?

    How to verify bug fixes?

    Verification of the bug fixes is very similar to filling the potholes.

    A pothole filling is not effective, if you just pour asphalt right into the pothole and move on. It’s not going to hold up. The right way to do it would be to first cut out the loose areas around the pothole. Clear any debris inside pothole. Now pour asphalt. Roll it nice and clean. If you do this then you have greater chances that it will hold good.

    Similarly while verifying the bug fixes consider widening and deepening the verification scope a bit. Remember there could be more bugs hiding around or behind this current bug. Statistics say every five bug fixes introduce a new bug.

    Functional coverage for bug fixes
    Functional coverage for bug fixes

    Functional coverage is important now as this bug has escaped the radar of code coverage. I hope you are now convinced. Even if you had avoided functional coverage so far now is the time to bring it into play. Set the clear scope for the bug verification using the functional coverage around the area where the bug was discovered. Remember the cost of the bug found increase as time passes.

    Sounds good. But we are really resource and time constrained. That isn’t going away even if we agree.

    If you really want to do it right then our tool curiosity can help. It can help you generate the whitebox functional coverage 3x-5x faster. Generated code requires no integration and compiles out of the box. So you get to coverage results faster. Faster results provide more time for verification. More time for verification of bugs results in better quality of RTL.

    What do you say?

  • Black box functional coverage model – Architecture

    There are two types of functional coverage. Black box functional coverage and white box functional coverage. This blog will focus on the black box functional coverage architecture.

    In order to cater to functional coverage model architecture requirements following architecture is proposed. It’s verification methodology independent. Can be used in verification environments built with or without verification methodology. Please note this is one of the way, not the only way.

    Idea is very simple. We will primarily addresses two questions:

    • How to organize your functional coverage?
    • How to integrate it with test bench?

    A good architecture exhibits the strong cohesion and low coupling. We had applied the same principles to test bench architecture. Now let’s apply the same to functional coverage model architecture as well. This translates to related functional coverage staying together and having very low coupling with the test bench. This model would then satisfy the objectives functional coverage model architecture requirements.
    (more…)

  • Functional coverage model – Architecture requirements

    Functional coverage model architecture requirements are ability to meet following three primary objectives:

    • Portability
    • Maintainability
    • Debuggability

    Portability

    Functional coverage is one of the reusable part of the test bench. It can have both horizontal as well as vertical reuses. Horizontal reuse could come in the form of same design IP used with different parameterizations to meet different use cases. Vertical reuse could come in the form of the reuse of subset of unit level functional coverage at sub-system or SOC level.

    Maintainability

    Functional coverages continues to live as long the IP lives. Add to that, it’s one of the reusable part adds to its higher shelf life as well. Which means it should be maintained for long time.

    Easy to maintain code should be easy to understand, enhance, debug and fix the issues. Easy to understand codes are organized with a theme. After initial effort once code reader recognizes the theme, he sees it consistently reflecting in all parts of code. This makes maintenance process easier.

    Functional coverage model is no exception to this rule. It should follow the same the same principles.

    Debuggability

    Functional coverage implementation is just the first part of the story. Climax of functional coverage story is, it’s closure and meeting the verification objectives successfully. This requires quite bit of debugging and analysis. Let’s understand some of the debugging challenges for functional coverage closure.

    First step of coverage closure involves classifying the functional coverage results generated from regressions into false positive coverage, false negative coverage and coverage that will not be hit. Among these false positive case is the most dangerous one. Review and redundancy is one of the most effective guard against false positive coverage.

    Bulk of these issues are caused by incorrect coverage information tapping points in test bench, incorrect sampling event selection and incorrect definitions of bins.

    Functional coverage model architecture should lend itself well to these debug challenges.

  • Functional coverage types – Black box vs. White box

    Functional coverage is one of the key metrics for measuring functional verification progress and closure. It complements by addressing the limitations of the code coverage. Functional coverage is one of the key factors contributing to quality of the functional verification.

    Code coverage holes are typically closed first followed by the functional coverage. So remember functional coverage is one of those last gates to catch the uncovered areas. After the design is signed off from functional coverage point of view, it’s one step closer to tapeout from functional verification point of view.

    Once the functional coverage is signed off, bugs hiding in the uncovered areas will most likely get discovered in silicon validation. The cost of bug increases significantly when its caught later in the ASIC verificaiton cycle. This emphasizes the importance of functional coverage. Please note functional coverage is not directly catching the bugs. It helps illuminate the various design areas to increase the probability of bugs being found. To make best use of it we need to understand different types of functional coverage.

    Functional coverage is implementation of the coverage plan created in planning phase. Coverage plan is part of, verification plan. It refers primarily to two sources of information for its verification requirements. Requirements specification and micro-architectural specification of the implementation. Functional coverage should address both of them.

    There are two types of functional coverage, black box and white box created to address both the requirements.

    Let’s look at them in more details.

    Yin-Yang of functional coverage
    Functional coverage types

    Black box functional coverage

    Functional coverage addressing the requirements specifications is referred to as black box functional coverage. It is agnostic to the specific implementation of requirements. It will not be dependent on micro-architectural implementation.

    Typically it’s extracted with the help of various test bench components. It also represents the coverage in the form of design’s, final application usage.

    Lets understand this better with simple example. Application usage, in processor world would mean instructions. One of the area for functional coverage would be to cover all the instructions and in all their possible programming modes like registers, direct memory, indirect memory etc. Another example from peripheral interconnect world, one of the coverage item can be to cover all the types of packets exchanged with various legal and illegal values for all the fields of packets.

    One of the best way to write black box functional coverage is to generate it from the specifications. This allows the intent of the coverage items to be preserved allowing the functional coverage to automatically evolve with the specification. Find out how you can take first step in specification to functional coverage.

    White box functional coverage

    Functional coverage covering the micro-architectural implementation is referred to as white box functional coverage.

    White box verification and its functional coverage is one of the under focused area. This is due to reliance on standard code coverage to take care of it. Verification engineers typically leave this space to be addressed by the design engineers. Design engineers do try to take care of this by adding assertions on assumptions and test point to see if the scenario of interest to implementation are covered.

    But for design engineers this is additional work among many others tasks. Thus it ends up not getting the focus desired.  This can sometime lead to very basic issues getting discovered in this area very late in game.

    White box functional coverage will be dependent on the specific design implementation. Typically, it will tap into internal design signals to extract the functional coverage. This tapping can be at design’s interface level or deep inside the design.

    Lets understand this with simple example. One of the white box coverage item in processor world can be instruction pipelines. Covering all possible instruction combinations taking place in instruction pipelines. Note that,  this will not be addressed by code coverage.

    In peripheral interconnect world it can be the FIFO’s in data path. Covering  different levels utilizations, including the full conditions. Transitions from empty to full to empty. Covering errors injected at certain internal RTL state. Covering number of clocks an interface experienced the stall. Covering all possible request combinations active at the critical arbiter interface. These are to name few cases. A simple LUT access coverage could have helped prevent famous pentium FDIV bug.

    White box coverage writing effort depends on complexity and size of the design. White box coverage writing effort can be reduced up to 3x easily by generating them instead of writing. Generation can happen as part of plug-ins in RTL or  using framework like curiosity.

    White box functional coverage and black box functional coverage can have some overlapping areas. There will bit of white box functional coverage in black box functional coverage and vice versa. Right balance of both black box and white box functional coverage provides the desired risk reduction and helps achieve the high functional verification quality.

     

  • Code coverage – Pain points

    Code coverage is composed of line coverage, expression coverage, toggle coverage, FSM coverage and conditional coverage. Code coverage is one of the oldest and reliable forms of coverage metric. It’s a working horse of verification engineers. Code coverage is one of the key metrics used in closure of verification.

    There are two reasons for its popularity. First it’s automatically generated. And second it’s comprehensive.

    Code coverage is automatically generated from the standard simulator tools. It just requires enabling few additional options during compile of the code and during run of test cases. A complete and comprehensive coverage report without any effort from user, that’s what, makes it so attractive.

    While all this true, there are sides of code coverage that are not so attractive. There are some pain points. In this blog we will look in to three such pain points.

    Pain point #1: Code coverage is not useful in early stages of verification cycles

    Code coverage to become useful requires certain level of maturity to RTL code. That’s how it ends up being looked at the later in the verification cycle. What are the downsides of it?

    Code coverage effectiveness in project execution

    While the comprehensive nature of the code coverage is its one of the advantage but it’s also a deterrent. Due to its comprehensive nature, code coverage requires good bit of time and effort to analyze and identify the actionable items for the verification to drive it to closure.
    (more…)

  • Coverage plan

    Coverage is one of the important metrics in measuring verification quality and helping with verification closure. Lack of functional coverage can lead to expensive bugs like pentium fdiv being discovered late in silicon.

    There are two categories of the coverage metrics. First one is code coverage. The simulator tool generates it automatically. Second is functional coverage. Verification engineers develop this by coding.

    Code coverage consists of line coverage, condition coverage, expression coverage, FSM coverage and toggle coverage of DUT. Code coverage is topic by itself. Focus of this blog is on the functional coverage.

    Functional coverage is not built-in and needs to be manually coded. It’s additional effort for the verification teams. The effort has to justify by providing the additional value on the top of code coverage. So it needs to be carefully planned.

    Functional coverage vs. Code coverage

    Code coverage has certain limitations. Some of the limitations of code coverage are:

    • It does not address concurrency
    • Various sequence combinations coverage cannot be determined

    Code coverage as the name says looks at design just as a piece of code. A network switch or processor or fundamentally every RTL design looks same to the code coverage.

    Code coverage does not differentiate them from application use case perspective. This means we cannot completely rely on code coverage. Thus code coverage is necessary but not sufficient.

    This is where the functional coverage comes in. Functional coverage looks at the design from its application use perspective. A network switch is not same as processor to functional coverage.

    In a coverage driven constrained random verification approach the randomness of the stimulus and configuration brings in the uncertainty. Uncertainty of, whether some scenarios really got exercised in certain configuration or not. This is where the functional coverage comes in handy to figure it out with certainty.

    Code coverage is based on the DUT code. Significant part of the functional coverage is based testbench code to check if it created the intended scenarios to ensure the correct functionality of the DUT.

    Coverage plan writing process

    Functional coverage may have certain level of overlap with code coverage. This may have to be done in certain areas for the completeness of the functional coverage. This does not mean, duplicates the entire code coverage in functional coverage.

    Functional coverage is luxury. But you cannot have lot of it. A careful thought has to be given where it can provide the maximum impact. The process for coverage plan also should follow “think theoretically and execute practically” as used in test plan writing process.

    Coverage plan creation takes into consideration 3 inputs:

    • Requirements specifications: This is the same requirement specifications used for designing DUT and creating the verification plan and test bench architecture
    • Design under test (DUT): Various aspects of DUT such as control & state registers, interface and key FSMs etc.
    • Test plan: Test plan listing the various scenarios in the form of stimulus and checks

    Please note DUT’s micro-architecture functional coverage is equally important as requirements specification coverage.

    Using these inputs create the complete functional coverage space using the following 7 steps process described.

    Coverage plan writing process

    Step#1: List all independent random variables

    List all the variables from the specifications that can take multiple values.

    Ideally each variable randomized in test bench should be covered. Random variables of each class have to be covered independently.

    Typically in the current verification environments, which are object oriented programming based, these variables are encapsulated in to classes. These classes fall in to configurations and transactions.

    Configurations can be structure configuration, functional configurations or error configurations. Transactions are encapsulations of request and responses from DUT at various levels of abstractions. All the properties of these classes declared as random should be covered by the functional coverage.

    Step#2: List all applicable combinations of random variables within class

    Look at the usefulness of combinations of two or more variables related to same unit of information. Same unit of information transforms to variables within the same class.

    This could be combination of variables within configurations and transactions may additionally have to be covered in all combinations of values with the other variables.

    For example in a full duplex communication protocol it’s not sufficient to cover all transaction types but also cover it in combination with the direction. This is to ensure all legal transactions types have been both transmitted and received by design. Note that transaction type and directions are properties of the same transaction class.

    Step#3: List all necessary combinations of random variables across class

    Cover the combinations of two or more variables across different units of related information.

    This could be combination of variables across the configuration and transaction classes or across various transactions and across various configurations.

    For example, to cover all applicable error injection types for all transaction types, cover combination of error injection type in error configuration and transaction type in transaction.

    Step#4: List all state variables

    Covering certain scenarios or sequences require covering state variables.

    State variables can be stored in the test bench or the DUT. As a part of this list all independent state variables that needs to be covered.

    Note that to cover certain type of sequences and scenarios new state machines exclusively used for the coverage may have to be developed in the test bench.

    For example in USB host subsystem total ports connected state count coverage for active ports coverage.

    Step#5: List all necessary sequences of same state variable

    Cover the sequences of the same state variables across multiple sampling events.

    For example to cover back-to-back stimulus transaction sequences, store last two stimulus transactions as a state. The combinations of these two transaction types stored in state have to be covered.

    DUT FSMs are covered by default in code coverage. Limitation of code coverage is, it cannot tell if specific sequences of state transitions have taken place. So some of the key DUT FSMs will have to be additionally covered by the functional coverage.

    For example cover sequence of state transitions of a port state machine. USB port going through one of the many state transitions such as: disconnected -> connected -> reset -> enumerated -> traffic -> low power etc.

    Step#6: List all necessary combinations of different state variable

    Cover the combinations of the different state variables across multiple sampling events.

    For example for a N-port USB hub verification environment, need to cover combinations of all port state. This coverage provides insight into, if the hub has experienced various variations of use case scenarios across its multiple ports. This combination can help find out if scenarios such as multiple port connections taking place at same time, one port under rest while data traffic flowing to the other ports, one port under reset while other in low power state etc.

    Another example for link layer is combinations of traffic outstanding state and credit state. This combination can help find out if there was outstanding traffic when credits had run out or insufficient credits for outstanding traffic etc.

    Step#7: List all necessary combinations of different random variables and state variables

    Cover the combinations of random variables and state variables.

    For example communication interface such as MIPI UniPro uses PHY that allows multiple power modes. Idea of power modes is to trade speed for power. It also supports multiple traffic classes for supporting quality of service. Covering combinations of the power mode state and randomized traffic class variable of data traffic transaction tells us if all traffic class were active in all power modes.

    After the coverage space is created, check if some items already have sufficient coverage in the code coverage. For items that have sufficient coverage in code coverage mark it so and focus on the items that are not covered or partially covered by the code coverage.

    Tool’s like curiosity can significantly cut down your functional coverage writing effort with its built-in functional coverage models. Not only its brings time to write it but various built-in coverage models can act as triggers for the functional coverage plan thought process.