Verification plan review is still a manual process. Verification plan is made up of three plans Test plan, Checks plan and Coverage plan. As of today’s technology, I am not aware of any tools to do automatic verification plan review based on machine learning.

First challenge in the verification plan review is to have a verification plan. Second challenge is to make it a traceable verification plan so that it can be trusted.

What  home work is required from verification plan reviewer?

Verification plan building requires specification expertise and verification expertise. Verification plan writer gets time to do a detailed scan of the requirement specification to write the verification plan.

The challenge for the verification plan reviewer will not be able to get same amount of time. Verification plan reviewer will still have to do the home work to make the review effective.
(more…)

Traceable verification plan is the one where following are all stitched together.

Test plan <-> Tests <-> Test variants <-> Regression status 

There is accountability for every test yet to be written, passing and failing test in the test plan. A test added to test plan is guaranteed to be executed. A verification plan that can be trusted. A traceable verification plan can be trusted. Its anchor for the functional verification quality.

Right not the concept is called out in the context of test plan which is the one of the three plans of verification plans where traceability matters most. Ideally it should be implemented to all three plans.

Why we need traceable verification plan?

Three phase of functional verification are: Planning phase -> Development phase <-> Regression phase. Requirements specifications are not really written for the functional verification.
(more…)

Review of verification plans can be very challenging process. First challenge is presence of the verification plan. If present, matching it with the latest status. Verification plans are created initially in many cases but they are not kept up-to-date.

Three possible scenarios good, bad and ugly based on status of verification plan are captured below.

• Verification plans itself does not exist – Yes this can happen in certain cases. Verification has just evolved. This is ugly
• Verification plans were created initially but not updated. Initial verification plans were never updated and current state of execution is far different from what is present in the plans. But it’s agreed that plans are not up to-date. This is good
• Verification plans are present and unclear update status. Initial verification plans were updated some times but thought to be completely updated. Yet they cannot correlate 1:1 to tests and regression results. This is bad

The ugly is just plain ugly. It’s raw and hence its clean to fix.
(more…)

Poor quality in regression phase is like spoiling the climax of the functional verification. If the right things have been done in planning phase and development phase, good news is it is easy recover back from quality lapses in the regression phase.

If the planning phase and development phase also had poor execution regression phase can become hell. Effects of poor planning phase and poor development phase impacts are not mixed. Focus here is effects of poor regression phase on functional verification quality.  
(more…)

Poor quality in the development phase results in poor quality test bench and tests. Poor quality in development phase has second biggest impact on functional verification quality after the poor planning phase.

One of the early symptoms of poor quality test benches are schedule slips in writing tests and getting them to passing state. Poorly architected test benches make the test writers job difficult. They do not provide adequate hooks and abstractions. This causes test writers to write lot of additional test specific code.  Coupled with this are the test bring up delays due to struggle with the debugging the test bench code and test bench bugs are early symptoms.

Certain feature update in test bench especially during the middle of the development, showing more than average bring up delays and causing significant instability regression causing large number of existing passing tests to fail frequently is also clear sign of potential bad architecture and implementation around that feature area.
(more…)

Poor quality in planning phase leads to poor verification plans and test bench architecture. This is a seed. This seed grows into tree as the project progresses and yields bitter fruits later. Poor planning phase has big impact on the functional verification quality.

Bulk of the planning activities about 60-80% completes during the planning phase of the project and remaining is completed during the development phase. This provides opportunity to recover from some of the mistakes of the planning phase during development phase.

If there are quality problems found with the planning phase deliverables during development phase, it’s best to fix them at that point rather than ignoring or putting it off to later. It’s well known fact that cost of fixing the mistakes increases exponentially as the project progresses.
(more…)

Functional verification quality has big impact on the quality of the design.

Quality and functional verification are very closely related. The whole objective of the functional verification is to build the high quality design. In order to do that all aspects related to functional verification itself should be of high quality.

As a part of ZVM: Verification methodology for quality we have already seen science, art and religion of verification needs to applied together for achieving high quality verification, which in turn increases the chances of achieving high quality design. This is all good but what if, for some reasons process was not followed and now we have quality issues with the functional verification. What to do now?

Here are the series of articles to help restore the quality of the functional verification. There could be varied level of quality issues and each needs varied application of these generic principles to restore quality. These are generic guidelines to help detect the causes and series of steps to address them. A firm commitment to invest time and resources is required to restore the quality. Quality always comes with its own price tag.
(more…)