Industry leaders from ARM, Intel, Nvidia, AMD are saying there are gaps in verification metrics. ANN STEFFORA MUTSCHLER of Semiconductor engineering compiled panel discussion held during DAC and wrote this great article about “Gaps In Verification Metrics”. Theme of panel discussion was, whether conventional verification metrics are running out of steam?

Following are some key comments from industry leaders.

Alan hunter senior principal engineer from ARM also shared about a new metric called statistical coverage and its benefits.

Maruthy Vedam, senior director of engineering in the System integration and Validation group at Intel, indicated verification metrics play huge role in maintaining quality.

Anshuman Nadkarni who manages the CPU and Tegra SOC verification teams at Nvidia asserted, “Metrics is one area where the EDA industry has fallen short”.

Farhan Rahman chief engineer for augmented reality at AMD says, “The traditional metrics have their place. But what we need to do is augment with big data and machine learning algorithms”.

What do verification teams think about it?

Sounds interesting. Where we are, even getting code and functional coverage to 100 % itself is a challenge, given the schedule constraints. Improvements and additional coverage metrics would be nice, but sounds like lot of overhead on already time-crunched design and verification folks.

Industry leaders do feel there is gap in verification metrics. Addressing them by enhancing and adding new metrics makes it overwhelming for verification teams. How do we solve this paradox? Before we get to that, you might be wondering, how do we know it?

How do we know it?

If you care to know, here is a short context.

How to improve functional verification quality? We started looking for answers to this question two years ago. We started exploring it from three angles: quality by design, quality issue diagnosis and quality boost.

When we shared it with verification community, they said, here is the deal, what’s done is done; we cannot do any major surgery to our test benches. However if you can show us the possible coverage holes with our existing test bench we are open to explore.

We found that most verification teams doing a good job of closing code coverage and requirements driven black box functional coverage. However white box functional coverage was at the mercy of designer. Designers did address some of it but their focus wasn’t much from verification point of view but to cover their own assumptions and fears.

So we started building automation for generating white box functional coverage to quickly discover the coverage holes. As we analyzed further, we found functional coverage alone was not enough to figure out if the stimulus did a good job on microarchitecture coverage. It wasn’t giving clear idea about did it exercise something sufficiently or whether the relative distributions of stimulus across features was well balanced. So we added statistical coverage. Based on the high-level micro-architecture we started generating both functional and statistical coverage.

As it started giving some interesting results, we went back to verification folks to show them and understand what they thought about such capabilities.

Very candid response from them was getting code and functional coverage to 100 % itself is a challenge, given the schedule constraints. This additional coverage metric is nice, but sounds like lot of overhead on already time-crunched design and verification folks. Some of the teams are struggling even to close the code coverage before tape out leave alone functional coverage.

Why are verification teams overloaded?

Today’s verification engineers not only worry about functionality, power and performance but also have to worry about security and safety. Verification planning, building verification environments, developing test suites, handling constant in flux of changing features, changing priorities, debugging the regression failures, keeping the regressions healthy, managing multiple test benches for variations of designs are all challenging and effort intensive.

Result of this complexity has also to lead to most of the constrained random test benches to be either overworking or underworking. Conventional metrics fail to detect it.

We went to verification managers to understand their views. Verification managers are on the clock to help ship the products. They have to face the market dynamics that are best defined as VUCA (Volatile, Uncertain, Complex and Ambiguous). That pushes them to be in highly reactive state for most of the time. Coping with it is definitely stressful.

Under such conditions they feel its wise to choose the low risk path that has worked well in the past and has delivered the results. They don’t desire to burden their verification teams further without guaranteed ROI.

We started to brainstorm to figure out how to make it work and deliver the benefits of white box functional coverage and statistical coverage generation without causing lot of churn.

Its clear there are gaps in the verification metrics. It’s clear the field of data science is advancing and it’s only wise for verification world to embrace and benefit from it. How do we reduce the effort of verification teams to help them gain the benefits from new developments?

Should we cut down on some of existing metrics?

First option to consider is, should we cut down on some existing metrics to adopt new metrics? We still need code coverage. Definitely we need the requirements driven black box coverage. Ignoring the white box functional coverage is same as leaving the bugs on table. Statistical coverage can help you discover bugs that you might end up finding only in emulation or even worse in silicon. We need all of them.

However one mindset change we can bring in is not waiting for 100% code coverage before starting to look at other coverage metrics. You need an act of balancing to choose and prioritize the coverage holes among various metrics that have highest bang per buck at every step of functional verification closure.

Now expecting this analysis and prioritisation from verification team is definitely overkill.

How can we benefit from new metrics without overloading verification teams?

We need to add new role to composition of successful verification engineering teams.

We are calling it as “Verification analyst”.

Verification analyst will engage with verification data to generate insights to help achieve the highest possible verification quality within the given cost and time.

Stop, I can hear you. You are thinking adding this new role is additional cost. Please think about the investment you have made in tool licenses for different technologies, engineering resources around them and compute farms to cater to all of it. When you compare to the benefits of making it all work optimally to deliver best possible verification quality, the additional cost of adding this new role will be insignificant.

Bottom line, we can only optimise for two out of three among quality, time and budget. So choose wisely.

Whether we like it or not the “data” is going to center of our world. Verification world is going to be no exception. Its generating lots of data in the forms coverage, bugs, regressions data, code check-ins etc. Instead of rejecting the data as too much information (TMI) we need to put it to work.

Most of the companies are employing simulation, emulation and formal verification to meet their verification goals. We don’t have metric driven way to prove which approach is best suited for meeting current coverage goals. That’s where the verification analyst will help us do verification smartly driven by metrics.

Figure: Role of verification analyst

Role: Verification Analyst

Objective:

Verification analyst’s primary responsibility is to analyze verification data from various sources using analytics techniques to provide insights for optimizing the verification execution to achieve the highest possible quality within the constraints of cost, resources and time.

Job description:

• Ability to generate and understand the different metrics provided by different tools across simulation, emulation and formal
• Good grasp of data analytics techniques using Excel, Python pandas package, data analytics tools like Tableau or Power BI
• Build black box, white box functional and statistical coverage models to qualify the verification quality
• Collect and analyze various metrics about verification quality
  • Code coverage
  • Requirements driven black box functional and statistical coverage
  • Micro-architecture driven white box functional coverage and statistical coverage
  • Performance metrics
  • Safety metrics
• Work with the verification team to help them understand gaps and draw the plan of action to fill the coverage holes as well as hung the bugs using the right verification technology suitable for the problem by doing some of the following
  • Making sure the constrained random suite is aligned to current project priorities using the statistical coverage.
    • Example: If design can work in N configurations and only M of them are important, make sure the M configurations are getting major chunk of simulation time. This can keep changing so keep adapting
  • Understanding the micro-architecture and making sure the stimulus is doing what matters to the design. Use the known micro-architecture elements to gain such insights.
    • Example: Did FIFO go through sufficient full and empty cycles, Minimum and maximum full durations, Did first request on the arbiter come from all of the requesters, Relative distribution of number of requesters active for arbiter across all the tests, number of clock gating events etc.
  • Identify the critical parts of the design by working with the designer and make sure it’s sufficiently exercised by setting up custom coverage models around it.
    • Example: Behavior after overflow in different states, combination of packet size aligned to internal buffer allocation, Timeouts and events leading to timeout taking place with together or +/- 1 clock cycle etc.
  • Plan and manage regressions as campaigns by rotating the focus around different areas. Idea here is to tune the constraints, seeds and increase or decrease the depth of coverage requirements on specific design or functional areas based on various discoveries made during execution (bugs found/fixed, new features introduced, refactoring etc.) instead of just blindly running the regressions every time. Some of examples of such variations can be:
    • Reverse the distributions defined in the constraints
    • If there are some major low power logic updates then increase seeds for low power and reduce it for other tests with appropriate metrics to qualify that it has had intended effect
    • Regressions with all the clock domain crossings (CDC) in extreme fast to slow and slow to fast combinations
    • Regressions with only packets of maximum or minimum size
    • Regressions for creating different levels of overlaps of combinations of active features for varied inter-feature interaction. Qualify their occurrence with the minimum and maximum overlap metrics
  • Map the coverage metrics to various project milestones so that overall coverage closure does not become nightmare at the end
  • Map the coverage holes to directed tests, constrained random tests, formal verification or emulation or low power simulations
  • Analyze the check-ins, bugs statistics to correlate highest buggy modules or the test bench components and identify the actions to increase focus and coverage for them or identify module ideal for bug hunting with formal
• Promote the reusable coverage models across teams to bring the standardization of the quality assessment methods

Continued from: Verification debate Part II

Few months had already rolled. It was bright sunny day.

Ananta sensed tensed activity at the sheriff’s office. He asked one of the officers. Officer indicated that one of the high profile criminal has escaped during transit. Office was bustling with activities. There was sense of urgency everywhere.

Even among this distressed situation sheriff seemed to be calm and composed. He was messaging and calling all the time while other officers were busy collecting information about incidence and getting it under control.

Just within few hours sheriff called for meeting with his key deputy officers. He provided them with the exact location where to find the escaped convict. He also indicated that this house is located in secluded locality near highway and there is underground passage from house to highway. So we should first set up the team on highway and then attack on the house. They did that and grabbed the convict in matter of few hours. This impressed Ananta further. He thought I am at right place.

Ananta went in and congratulated sheriff on this success. He asked how was he able to pinpoint the location so quick?

Sheriff said it’s my informers network. We have a widespread and deep network of informers. We get a very good intelligence on various happenings in county. This has been one of our key strengths helping us maintain the low crime rate in our county. We do invest, grow and guard our valuable informers network.

Interrogation of the escaped convict captured went on during most of the part of the day.

Sheriff came to Ananta with the expression of melancholy in early evening. Asked him if he knew anything about blocking hacking efforts?

Ananta asked what happened?

Escaped convict had some confidential information, which he has passed on. It was location of our hidden machine guarded data center far away from city limits. Looks like there is already a physical breach and they have got access to serial and usb debug ports of our main machine. By the time we reach there they will hack it. Considering the holiday seasons our key specialists are out. Is there anything you can do help us?

Ananta asked what is stored on these machines?

It’s very confidential. We have information about our informers on that machine we need to protect it at any cost. We cannot afford to lose it.

Ananta said let me see please take me to your team working on it. He joined the team working on blocking the hacking attempt. He got a quick brain dump. The experts working indicated, hacking is going on with the FPGA based custom designed system. They will accelerate by running hacking algorithms on custom hardware quicker.

Ananta asked which port are they trying to break in first? It’s serial port because that’s simplest. When he looked at the schematics of the machine he found a small relay near serial lines. He asked what is it used for? We had plans to allow remote control of machine power but we did not use it. We just have LED attached to it.

Can we control relay remotely?

Yes said the expert. Then let’s start switching it on and off. What do we get by doing that? It might induce some emi noise on the serial lines. They immediately acted and started program to turn relay on and off.

Over here in the data center the mastermind Professor Buggart and his associates were trying to break in with the serial port. The glitches on the lines due to relay switching on and off sent their UART IP’s receive state machines to unknown states resulting in hang. His associate quickly generated the waveforms from embedded analyzer in his FPGA board. Figured out that noise on the line is leading to false start bits. They had not completely covered these cases in verification. We have bug here said assistant.

We don’t have time for these bug fixes shouted professor. You guys had said the code coverage was 100 %. Why are we having this issue? Assistant said that’s something can discuss later. Now let’s try the USB port.

They reconfigured the FPGA with USB IP and started trying out with the USB. The link came up and Professor Buggart was very happy. Now quickly start copying the files.

Here at sheriff’s office hack prevention team had figured out USB port was compromised. Ananta asked what speed is USB port operating? 3.0 replied one of the experts. Lets quickly create a large file with the repeated control symbol values from the training pattern of the USB3x link training. Through a script they quickly created the file of gigabyte in size containing these control symbols and uploaded to machine. Name it as highly_confidential.txt

Here in the data center. When the hackers tried copying the file their USB IP crashed. Professor Buggart was very upset. He shouted, what is happening here? His assistant replied our LTSSM is getting falsely triggered and moving to recovery state during data transfer. Looks like some of our control symbols decode was not validated with LTSSM states. We did randomize the data in our verification but we are not sure how why this is happening. Professor Buggart kept repeating you guys said code coverage was 100 % even on USB IP. Before assistant could answer the sheriff’s team nabbed them.

Sheriff thanked his team and Ananta for his help.

One of the experts on prevention team asked to Ananta how did you think of those cases to prevent hacking? Oh! These are some of the corner cases that verification teams often ignore. I have also figured them out hard way replied Ananta. Also when you guys said they are using FPGA based systems, I guessed FPGA based IP developers generally rush to FPGA rather than verifying it sufficiently in simulation.

Ananta was satisfied with the results and contributions he had made. He was tired with the hustle of the day when he reached back his hostel.

Ananta’s internship was also coming to an end. He reflected deeply on his experiences that night. Find out what insights he gained in the last part of the series.

Conclusion: Part IV.

Continued from: Verification debate Part I

Good morning Sheriff.

I am Ananta. I was selected for citizen internship of 6 months.

Welcome Ananta. Come let’s catch up over a coffee.

Sheriff was an old man in his early fifties. He was tall guy with piercing sharp eyes. His gray hair and scars were proof of experiences he has had. His presence was something that you could not ignore.

At breakfast table, sheriff asked how do you earn your living?

I verify hardware designs for my bread and cheese, replied Ananta.

Well, I am curious what brings you here? I don’t think we do any of those things here.

Ananta said, I have been following your work for quite some time. My interest is to understand how are you able to bring down the crime rate so fast? I want to study your methods.

How is it going to help you young man? Asked perplexed sheriff.

Ananta said there is some similarity in what we both do. You catch criminals we, catch bugs.

What bugs? exclaimed sheriff with surprise

Ananta said oh no! not that bugs. When we say bugs in our professional world, we mean the parts of design violating the requirement specification. It causes malfunction in the operation of the systems using these designs.

Umm! I get it, requirement specifications are like laws then, said sheriff. Since bugs violate laws, you guys treat them as criminals. So you want to bring down the bug rate in your designs by learning my methods.

So that makes you sheriff of the verification land he said jokingly. We enforce laws and you folks enforce requirement specifications.

Exactly! said Ananta. He could not hide the expression of appreciation for the old sheriff who caught up with the concepts so quickly. He was now convinced that his in right place and slight doubts that he had seeing old man in person had vanished.

Sheriff introduced Ananta to all his staff. Asked him to start looking at some of the solved cases as to how his team has solved them.

Alright…. said Ananta half-heartedly. He was excited and eager to see something live and be part of it. He knew experience is the best teacher.

Sheriff went back to his table and immersed himself in the files in front of him. He spoke over the phone to his staff on the status of various cases. Planned out the tasks for all the officers, quickly scanned the newspaper to see any other news of interest. He then went for rounds of his jurisdiction himself.

The Sheriff was busy with some important cases and couldn’t give much time for Ananta.

Ananta was not guy who would sit in corner and wait for things to happen. He had a mission. He started observing and making notes.

Ananta figured out that patrolling is backbone of police operations. It consumed most of the resources. So he first spoke to patrolling head.

Ananta asked what are some of the patrolling strategies used? Officer answered we have few officers doing regular circuits or passes through key areas called a beat.

Hmm Ananta’s verification mind started correlating. This sounds like directed test cases.

Officer said we also have some police cars cruising randomly through city streets supposedly create the feeling that police are everywhere. This method is the most controversial and questionable in terms of its effectiveness.

Ananta chuckled. Officer asked what happened? We also do something similar in our profession. We also have same challenge. Goes without saying constrained random tests passed through his mind.

Suddenly phone started ringing. It was an emergency call. Officer excused himself with his team to attend to their duties. Ananta was reminded of tanked weekly regressions, where all of his team would rush to debug and fix it.

There was curious constable looking at Ananta from a far corner.

Ananta waved his hand and walked to him. He introduced himself. He asked politely what are your responsibilities, sir?

Constable said myself and some of my colleagues are stationed at some key locations. These include airports, railway stations, bus stands and some sensitive spots around the city. We just patrol small area around those key locations on foot. We keep vigil in those sensitive areas everyday. We also keep an eye on arrival and departures of new folks to our county.

Thank you said Ananta. Okay, these folks looks like are doing job of checkers and scoreboards of verification environment.

Days were rolling. Ananta was studying some of the cases. He was also studying various operational plans and tasks lists made in the office by sheriff. He was amazed at the level details and wanted to talk to sheriff about it.

Ananta finally caught up with sheriff on that evening. Ananta shared about his learning’s so far.

Sheriff said yes we do carefully plan our patrols and pick locations where we station our people. Both are very important activities for us and glad you have learnt about it. He showed details on how these plans are laid out with the map of his county. It was very detailed and well thought out.

Ananta appreciated the details.

Sherif said please note detailed does not mean they are lines in stone. We have designed our plans such that they can easily adapt to changes. Which is also equally important as details.  It certainly created great impressions in Ananta’s mind.

It was clear to Ananta these plans looked at like test plan and checks plan part of verification plan. But Ananta was more curious about the worth of time invested in planning.

He asked sheriff what is your general view on planning? Do things go according to plan?

No, said sheriff in his deep voice. Old man quoted Dwight D. Eisenhower saying, in preparing for battle I have always found that plans are useless, but planning is indispensable.

We have learnt from our experiences, no plan survives contact with the reality.

So our plans are really plans for adapting and improvising. It’s only this type of meta-planning that allows us respond quickly and achieve our objectives under dangerous circumstances.

Grave danger? Ananta asked

Sheriff questioned back is there another kind?

Wow, that’s great insight. Yes, importance of planning was now clear to Ananta.

Now it was clear to him why executable, traceable and trackable verification plans were emphasized so much by Achyuta. We must also build our verification plans as meta-plans that have adaptability built into them noted down Ananta.

It was already late and sheriff drove Ananta back to hostel where he was lodged.

Something exciting which Anant was waiting for long is about to happen in next few days.

To be continued in : Part III

Achyuta and Ananta were close friends over decades and practicing functional verification.

Achyuta always takes a carefully planned approach while Ananta believes in jumping to action as soon as possible. Except for this key difference both were well matched in their capabilities. Both had their own share of success and failures.

Ananta had shining career and team always appreciated his brave fire fighting spirit of jumping right into execution. Not only he jumped in but also produced results very quickly. Ananta in past had a good success rate with  ASICs he worked on making to production on first pass. He took pride in it.

However off late Ananta was not feeling the same high. It was not without a reason. There were series of bugs found right after the RTL freeze. This had upset Ananta. He had carefully tracked the bug trends, regression results and code coverage closely. Based on that he had decided they had reached the RTL freeze quality. Now this rise in bugs inspite of all these measure had upset him. He was worried have I lost my midas touch?

Ananta called his old buddy Achyuta for dinner to talk about it. Achyuta in usual methodical way asked Ananta, did you guys write verification plan or just jumped right into execution in your usual style? Ananta said dear Achyuta I have changed my ways a bit by being with you for so long. I have learnt there is thin line between being bold and reckless. We did write the test plan before starting execution. That’s quite a progress for Ananta, Achyuta thought to himself.

Ananta, do you know test plan alone does not complete verification plan? asked Achyuta.

What else? Said Ananta slightly annoyed.

Verification plan is made up of three plans: Test plan, Checks or assertion plan and coverage plan. All three are equally important for successful verification closure.

Oh Achyuta, now don’t start that one again. Cut it short man, said Ananta.

No it’s not just me check out these blogs on verification plan, test plan, checks plan and coverage plan if you want.

I have looked at them. To me it’s evidently the theory of some arm-chair lounger who evolves all these neat little paradoxes in the seclusion of his own study. It’s not practical. Just put him on our current project and I would lay a thousand to one against him if he can rescue it. We are finding bugs after bugs. You may loose your money Achyuta remarked calmly. It’s not about him we are discussing methods here.

Anantha verification requirements have changed a lot in last 10 years, said Achyuta. We cannot dismiss methodical approach any more.

Forget it, said Ananta. All these are easier said than done. All these theories melt in heat of execution.

Achyuta just put brakes on his tongue which was about to slip into debate. He thought this is not a right time for this debate.

Ananta also changed the topic and said anyway reason I called you today was I am thinking of taking sabbatical for next 6 months.

Huh! What do you plan to do?

You see I have laid my eyes on this new sheriff in the town for quite some time. After his arrival the crime rate has gone quite low. I have even heard that the criminals who rarely escape are only talking about their luck for rest of their life.

What does all this has to do with verification and your sabbatical?

I need some fresh thoughts. So I am thinking of taking internship at his office.

Have you gone crazy exclaimed Achyuta.

No, I have given fair bit of thought and I am starting tomorrow. Ananta seemed very determined. Achyuta being good friend understood him and said, call me if you need anything.

Sure, will do said Ananta.

What Ananta learns in sheriff’s office?  Part II

Silicon proven IPs do enjoy some special attention to envy of the one’s aren’t. Do they really deserve it? What is so special about them?

Simulation however much we love and adore is simulation. It’s not real. There are lots of aspects not modeled in the simulation. We may take lot of pride in our latest verification simulation technologies, but still there are lots of details of real world that IPs get shielded in simulation environment. Lets call them simulation blind spots.

Those blind spots can be categorized into into three major areas:

• Electrical characteristics of physical communication channels
• Quirks of analog parts of the design
• Software application interactions

Even beyond all these simulation has speed challenge. For larger designs the highest simulation time is limited by the highest tolerable wall clock times.

Though there are technologies in development to manage these problems independently and in some combinations but there isn’t one clean solution that fits all. Current verification strategy is still work in progress to address these blind spots.

Silicon is real world where all of it comes together. If the design IPs were to be real person, it would be very stressful for them to handle all of these together. That’s why many fail.

First time silicon is like “meet the parents” moment for design IP. The design IP that hasn’t had this moment is still not in the circle of trust. There are many jinx moments for them.

Why are these blind spots such a big deal?

Simple power on reset value for physical pad being incorrect can potentially leave a particular interface dead. Eye of signal can be real eye openers.

Suddenly the clock tolerance of 4 % becomes highly meaningful when the input sampling of the signal starts failing to detect the right data.

That status register which we never cared about becomes hero overnight when software polls that register to figure out why commands are failing.

Crazy things happen in real world. Suddenly those streams of all 0s and all 1s for few milliseconds may seem like cyclone. That line in specification, which almost looked so harmless isn’t harmless any more when you start seeing the wake signals being beaconed at not so good time for your design. That clock gating and wake coming together is not so pleasant.

That’s exactly the reasons why  silicon experience and exposure to real world for design IPs does earn them special place over the ones that aren’t silicon proven.

Everyone related to any form of ASIC design verification has to internalize one fact: the “cost of bug” discovered increases exponentially as it advances in the ASIC verification phase. There is a big cost difference between bug found pre-silicon versus post silicon. Let’s understand what are those phases.

The ASIC verification takes place in various phases planned as a part of verification strategy. Typically they are unit verification, System/SOC verification, FPGA prototyping or/and Emulation and post silicon validation. Hence bugs can be found in any of these phases.

Leaving all complex cost calculations of these phases on the table, let’s just look at the time and effort to debug the bugs found in different stages. This will automatically bring out the cost element and why it’s important to find as many bugs as possible in, as early phase as possible. Let’s first identify what is required for the debug.
(more…)

Testbench is primary vehicle for verifying the functional verification requirements. Quality of testbench affects the quality of verification. Yet, it’s often ignored due to various myths.

This blog will look into 5 such myths about testbench quality and share different perspective to look at them and how it affects the projects.
(more…)