Category: Functional verification – Management

Testbench logging is front end of the debug. These logging guidelines are designed to help make debug effective, not only for selected geniuses in team but also for everyone. Debug activity affects entire team and carried out throughout product’s life cycle. Logging is one of the very important contributors in making debug efficient and simple.

Some of the following guidelines for easing the debug may require additional code implementation beyond normal testbench or test functionality. But this additional effort pays off by saving the valuable engineer’s time during debug throughout product’s life cycle. Let machine’s do bit extra work to reduce the human effort. What do you say?
(more…)

Verification project management gave an overview of the six essential parameters and three phases of the verification project. This article will focus on applying the six essential parameters to planning phase.

Planning is very important phase of the verification project. Verification plan and test bench architecture are created during this phase. It sets the foundation for the development phase and regression phase. Entering execution with weak plan is like taking a route of slippery slope. Once taken there isn’t much chance of recovery and it takes you down for deep fall.

Key objectives of this phase:

• Get the verification plan in place
• Get the test bench architecture in place
• Get the detailed tasks list for the test bench development
• Milestones lists and division of the tasks per milestone

Next let’s look at what each of the key management principles identified in the verification project management translate to in the planning phase.

Planning phase: Clarity

Following four elements should be clear during planning phase. Overall work scope clarity needs to be established during this phase.

• Scope understanding: Scope of the verification is based on the specifications. Key prerequisite for planning phase is specifications. Specifications mean both the standard requirement specifications and implementation specifications. Most of the today’s designs deal with multiple specifications. It’s very important to identify the list of all the specifications applicable.
• Specifications understanding: Clear understanding of the specifications both the requirement specifications and micro-architecture specifications
• Verification strategy understanding: A clear verification strategy needs to be established based on the specification scope and understanding. This forms the basis for the verification plan and test bench architecture creation
• Team understanding:  Clear list of the contact persons in architecture, design and verification team at one place

Planning phase: Metrics

Metrics during planning phase are meant to provide the idea about the completeness of the verification plan, test bench architecture and development task lists creation.

• Total number of features
• Verification plan completeness percentage
  • Number of tests identified
  • Number of checks identified
  • Number of the functional coverpoints identified
• Test bench architecture completeness percentage
  • Overall architecture completeness
  • Major blocks of test bench
  • Total number of major blocks identified
  • Total number of major blocks for which interface and functionality is defined
  • Abstract classes
    • Total number of classes identified
    • Total number of abstract classes coded and compiling
  • Development tasks lists
    • Total number of blocks for which task list is identified
    • Total development tasks

Planning phase: People

• Best of your experienced engineers should be working on verification plan and test bench architecture development. There have been misconceptions to get this done by the relatively inexperienced engineers with perception that all we are producing is a document finally. This is totally a wrong perception to follow
• Technical leader of the project should be involved in this activity very actively. He should be one of the key contributor to verification plan and test bench architecture
• Engineers with natural attraction for the engineering depth are well suited for this activity. Both the verification plan and test bench architecture requires someone to put his heart into activity and go deep in the details. Figure out as many things as possible
• Engineers working on plan should be allowed to think theoretically enumerating all the verification scenarios without worrying about the resource and schedule content. Think theoretical and execute practical approach should be followed

Planning phase: Tracking

• Planning phase involves one of the highest levels of creative work. Out of box thought process may be required to set the verification strategy. Good test bench architecture requires fairly deep and creative ideas. Make sure it’s not rushed through. It’s easy to rush and create half-baked plans as it’s difficult to judge if it’s complete in early stages of the project
• Put your best team in action and trust them to use their best judgment.  Agree we do not want paralysis by analysis situation but at the same time high pressure to close can result in premature closure. Premature closing of planning phase will result in costly reworks later. Its pay now or later with huge interest rates
• Put your team working on planning and architecture in an island. Talk to them only when needed. Allow the team members to get into flow state for best work to emerge. Instead of weekly consider biweekly or monthly status tracking of the metrics identified.

Planning phase: Review

• Verification strategy should be carefully reviewed. It sets the basis for the verification plan and test bench architecture
• Verification plan structure review has to be conducted periodically. This ensures no big ticket items are missed
• Review of the verification strategy, verification plan and test bench architecture should be conducted with the architecture and design teams
• Verification plan should be executable. It should not be limited to a very high level plan. A junior verification engineer should be able to pick up the test plan and should be able to write a test, implement a check from checks plan and write cover points from functional coverage plan with very little interaction with the verification plan owner
• Verification managers should pick a feature and question it for its consistency in thought process across all three test plan, checks plan and coverage plan
• Architecture should be questioned for flow from normal operation, error injection, control event generation and processing, various checks implementation and ease of functional coverage collection point of view
• Alignment of the verification plan and test bench architecture to the verification strategy should be reviewed
• Review actions should be tracked and closed. It’s possible that certain areas are going to remain open for a while till the further understanding is developed. This is fine as long as they are clearly documented and tracked

Planning phase: Closure

• All the specifications applicable are identified and made accessible in a global shared location for verification team
• Code repositories for development, email aliases for the communication and discussions, Task tracking system for project and bug database are setup
• All three plans test plan, checks plan and coverage plan constituting the verification plan in executable form should be checked-in under code repository
• Detailed test bench architecture in place and checked-in under code repository
• Reviews of the verification strategy, verification plans and test bench architecture completed.  Major review items addressed and open items documented and put in a task tracking system for closure
• Tasks list for all the development activities captured in the task management system. Tasks grouped under the functional milestones
• Schedule forecasts made created based on tasks lists. Based on the forecasted schedule, deadlines and current team availability hiring or external contractors required for the next phases should be identified
• Decisions regarding the exploration and evaluations of third party Bus functional model(BFM )s, verification solutions or tools should be short listed
• Simulator licenses and compute resource requirements should be checked for sufficiency based on the forecasted verification plan’s requirements. If additional upgrades are needed the procurement process should be started
• It’s possible verification plan and test bench architecture is not 100% complete. It’s hard to achieve that unless its repetition. Focus during planning phase should be on completeness of major sections in verification plan and major blocks in test bench. Second level of sub-sections in verification plan and details of blocks in test bench architecture can be complete any where between the 60-80 %. This provides good visibility and rest can be figured out as part of development phase. This achieves balance between waiting too long vs. starting too early

Verification project management is a set of process and principles to achieve high quality verification results.  These process and principles are anchored around six essential generic parameters. Six essential generic parameters are Clarity, People, Metrics, Tracking, Review and Closure. Verification project consists of primarily three phases. They are planning, development and regressions. All six essential generic parameters are applicable to all three phases of verification.

This article is organized as:

• Brief introduction to six essential generic parameters
• Brief overview of three phases
• Brief introduction on people management
• Brief introduction on role of Verification manager
• Applying these managmement concepts in all three phases
  • Planning phase management
  • Development phase management
  • Regression phase management

Six essential generic parameters :

1. Clarity: Identifying the aspects that should be clear. It can be clarity of goals, tasks, timelines and process. Clarity is not absence of ambiguity but a constant battle against the ambiguity.  This is a single most important factor that keeps team motivation levels up and translates to results.
2. People: Right person for right job is half the job done. Bad engineers are rare. Most are just mismatches in assignments.
3. Metrics: Management guru Peter Drucker quoted “If you can’t measure it, you can’t manage it”. Metrics provide indication of how the project execution is progressing. Right metrics provide insights to improve the execution.
4. Tracking: Tracking is a process of collecting metrics and using metrics to tune the processes to achieve the desired results.  Tracking provides the push required for closure.
5. Review: One of the popular Russian proverb says “Trust, but verify”. Review is a process to guard the quality in each of the tasks to achieve overall quality goals.
6. Closure: ABC of verification project management is “Always be closing”. Functional Verification is never ending process. Unless you close it, it will go on.

Three phases of the verification project:

The three major phases of the verification project are planning, development and regressions.

1. Planning phase primarily focuses on creating verification plan and test bench architecture. Verification plan consists of test plan, checks plan and coverage plan. Based on verification plan and test bench architecture detailed task lists are created.
2. Development phase focuses on building the bus functional model(BFM), test benches and tests. HVLs and Verification methodologies play a major role in building these.
3. Regression phase is about executing all tests and its variants to catch the issues and meet the coverage goals.

These phases do not have very precise boundaries. Only the focus moves from one phase to another during project cycle. The phase under focus gets more attention and resources.

Planning phase gets highest attention during the initial part of the project. Planning activity mostly completes about 70-80 % during planning phase. Remaining part of planning activity completes within first 50 % of the development cycle.

After the initial planning phase completes, the verification project execution is divided into multiple milestones for making feature verification time bound. Every milestone starts with brief planning activity mainly consisting of scheduling of tasks subsequently it’s dominated by development and regression phases. Milestones early in verification project are development dominated and towards end it’s dominated by the regression phase.

People management

People management aspects are not covered in detail as part of this article. This is the only paragraph on people management aspects. Quoting some of points from Dan Pink’s TED talk on puzzle of motivation. Extrinsic motivation works great for left-brain tasks made up of clear set of rules and have a single solution to problem. Functional verification does not fall in that category. It does not mean there aren’t any tasks of this nature. But it’s dominated by the tasks that require some level of creativity and ability to deal with ambiguity.

Extrinsic motivators of the type carrot and stick alone cannot provide results. Management is for compliance. Self-direction is for engagement. New age mantra for driving teams is to create environment of autonomy, mastery and purpose.

Ending this topic, by quoting Sahil Gupta, aspire to be a leader whose legacy is not just the products delivered but also the teams created. These form principles for people management.

Verification manager

Verification manager plays a key role in the verification project management. Look at the composition of the successful verification teams to understand the roles and responsibilities of the verification manager further.

One of the key responsibilities of the verification manager is to define, refine and improvise process to achieve sustained high quality verification results. We have developed a framework to help you do a quick audit of your constrained random stimulus. This will help you make your stimulus work hard for you and helping you meet your verification quality goals.

Take care of process and results will take care of themselves. – MS Dhoni (Captain of Indian cricket team)

Project manager is an owner. An owner who needs to ensure the quality of the verification meets the requirements and results are delivered in time. Project should not be organised around project manager. Project manager should organise project around sound principles and processes.

There are multiple phases of the verification project. Each phase has it’s own challenges. Each phase has it’s own of best practices suitable for it. Each of the generic essential parameters identified should be viewed in the light of requirement of verification project phases. Be it tracking, metrics used, reviews conducted, follow up styles should be adapted to meet the objective of respective phase.

Inspite of all the efforts some times its challenging to meet the verification quality goals. Our services can help you quickly spot the coverage gaps in your constrained random verification.

Let’s now look at how to apply six essential generic parameters in each of the planning, development and regression phases.