Track issues to ensure that you don’t have to debug them twice

Article By : Ronak Dham and Sourabh Goyal

Here is a technique to track issues that engineers commonly face to ensure that they don’t have to debug the same problem twice.

How often do you have a déjà vu? Like you have seen an already-debugged issue before, and when you can’t recollect even after digging through e-mails for hours, you end up debugging the same issue, thereby re-investing the same amount of time again.

Here are some facts to consider:

  1. Fixing a bug takes 30 times longer than writing a line of code.
  2. 75 percent of a developer’s time is spent on debugging; that amounts to 1,500 hours a year.

Known issues and solutions or “KIAS” is a term coined by the authors as a short form of a notebook that comprises all the issues faced, ranging from problems in projects to editors like gvim to programming languages to Linux and Windows-related OS problems.

As time passed, the notebook became a word file and moved to the cloud. Then, it became increasingly difficult to involve others in the system as no one really felt like documenting the stuff. The additional step to go search for issues in this pre-registered page/doc was just too cumbersome, so people gave up on the idea.

Even if people logged information on issues every now and then, no one really went on to search once they found an issue during a debug cycle, rendering all the logged information of no use. So, we came up with the idea of an executable KIAS.

However, before we delve deeper into that, what’s the primary motivation? The answer to this is a rhetorical question. Do we really want to add more time to what we have already spent on debugging issues? Therefore, the solution aims to remove two dependencies:

  1. Dependency on “who” by eliminating the dependency on the person who first debugged the issue.
  2. Dependency on “when” and “how” by eliminating the dependency on self-memory to remember when and how one debugged an issue.

The solution aims to add two facets to errors previously faced by developers:

  1. Provide root cause analysis (RCA) information to the user.
  2. Provide a list of possible solutions to the user.

Here, it’s worth mentioning that JIRA and other bug-tracking tools have limitations. These tools don’t proactively tell users that this issue has arisen before, and they may not be logged for common issues that aren’t really bugs.

Executing KIAS

Here is how the idea of executable KIAS is achieved:

  • By logging KIAS information in a YAML file—could be virtually any format, including XML, JSON, TXT—and a script to parse for pre-registered errors and append information to errors seen in the compile/simulation log files.
  • By using system Verilog callbacks to intercept the UVM_ERROR/UVM_FATAL message and analyze and append information before it’s printed to the log.

Now let’s explore both options.

  1. Using system Verilog callback

For users who only wish to log KIAS for runtime issues and uvm errors, it’s an ideal solution. It uses the catch and throw mechanism in the uvm_report_catcher class. It’s like implementing exception handling when we want to demote some errors in our TB for certain tests.

However, in this case, what we are really doing is modifying the error message by appending KIAS information to the error string itself, thereby automating the printing of past logged debug information of the error/issue in question.

The uvm_report_catcher is used to catch messages issued by the uvm report server. Catchers are uvm_callbacks#(uvm_report_object,uvm_report_catcher) objects, so all facilities in the uvm_callback and uvm_callbacks#(T,CB) classes are available for registering catchers and controlling catcher state.

The uvm_callbacks#(uvm_report_object,uvm_report_catcher) class is aliased to uvm_report_cb to make it easier to use. Multiple report catchers can be registered with a report object. The catchers can be registered as default catchers, which catch all reports on all uvm_report_object reporters, or catchers can be attached to specific report objects (i.e. components).

User extensions of uvm_report_catcher must implement the catch method in which the action to be taken on catching the report is specified. The catch method can return CAUGHT, in which case further processing of the report is immediately stopped, or return THROW in which case the (possibly modified) report is passed on to other registered catchers. The catchers are processed in the order in which they are registered.

On catching a report, the catch method can modify the severity, ID, action, verbosity, or the report string itself before the report is finally issued by the report server. What we intend in this case is to change the report string.

screenshot of macros defined to help provide string-based messagesFigure 1 Some macros that we have defined to help us provide string-based messages, which we would like to see whenever the error is logged.

Here’s the callback implementation.

screenshot of the system callback implementation codeFigure 2 The implementation uses system callback.

2. KIAS Python script

The KIAS Python script can capture any form of error string—compilation or simulation—and search for it in the given log files to provide KIAS information about that error string. The script can parse log files independent of the language, EDA tool, or process, thereby helping verification engineers and design engineers alike.

It has a look-up list of KIAS stored in a YAML file format. Users can use the GUI mode to save information in the YAML file, which would be used in all subsequent KIAS script execution of parsing log files for error strings and providing logged probable cause and possible solution for the same.

The KIAS script has mainly three functions. The first one is to provide a means for the user to easily add new KIAS to the YAML file using a GUI platform (entry form shown in Figure 3) or an interactive mode. It helps the user to create a database of KIAS that can be used by the next function.

The second function loads the YAML file and extracts it contents in a Python dictionary (dict) datatype. Part of this function is also to distinguish between compilation error and simulation error KIAS so that both can be separated into two different arrays or lists.

The third function uses the lists created by the second function to search and process the log files for the error message and append the KIAS information extracted from the YAML file.

screenshot of the KIAS entry formFigure 3 Here’s how the KIAS entry form looks; users can add new KIAS to YAML and can also use regular expressions in their error string to perform string matches accordingly.

Here is how the sample usage and YAML format look:

Sample usage

./ –infile kias.yaml –log debug –comp compile.log –sim sim.log

YAML format

—<Start of Yaml Document identifier>

KID:001<KIAS ID number>

OWN:Ronak<Name of contact person who identified/worked on this issue>

TYP:COM<Type of error, COM for compile time, SIM for simulation time>

ERR:RO period mismatch<Err msg which script should look for in the log file, more precise, the better>

RCA:RO period mismatches because clk period provided in testcase is not correct<Root Cause Analysis, why the error showed up, reason behind the issue>

SOL:|<Use when need to write multi-line info>

Provide proper clk period in test as per RTL/Waveform.

Check RTL for proper clock frequency. <Multi-line/single line message of what is the possible way to resolve this kind of issue or the exact solution found in this case>—

This article was originally published on EDN.

Ronak Dham and Sourabh Goyal work on the verification of FPGAs at Xilinx.

Related articles:

Leave a comment