Many people erroneously credit the term "bug" to Rear Admiral Grace Murray Hopper. In fact, however, no one knows where the term originated, and it may go back to Thomas A Edison or even earlier. Nevertheless, US Naval Reservist Hopper found a moth that had short-circuited Relay #70, Panel F, of the Mark II Aiken Relay Calculator while it was being tested at Harvard University on Sept 9, 1945. By removing the moth—that is, "debugging" the computer, she solved the glitch that had temporarily shut down the machine (Reference 1). Although that early example literally removed a bug from a system's hardware, "debugging" today refers to the process of understanding a program failure and removing the defective code. A failure includes any small deviation from the original intention, and removing the defective code is much better than adding correction code. In an ideal situation, all characteristics, such as bus and register values of a system, would be accessible at any time for monitoring and modification. But with ICs moving toward SOCs (systems on chips), accessibilitybecomes more difficult.
Debugging hardware is about providing as much access as possible to the internal resources of an embedded system, either through observing characteristics of the system, such as CPU states and PC values, or by changing system parameters. You can debug embedded systems in the early stages of design with simple techniques, such as logging and monitors, or you can use more recently developed methods for multicore SOCs, such as tracing, cache debugging, and crosstriggering. This article focuses not on how to write or debug the code, but on which embedded hardware IP (intellectual property) is available and which aspects of debugging this IP addresses.
LOGGING AND MONITORS
The oldest and most common method of debugging is to add print statements in the code to show information about which pieces of the software execute and provide insight into the actual values of registers and variables. This exercise may be a legacy of the firstyear- student exercise "hello world," in which the appearance of the two words on the screen prove that the program was alive and has executed up to a certain point. The print statement, or printf, is just one variant of logging, the process of using the processor to write important information to a "pipe" for external tracing. The pipe you use depends on the system; in the printf case, the pipe is the standard output (screen), but it can also be a UART, a USB, or even generalpurposeI/O.
Logging can be valuable when you need to structure component information in a way that is meaningful to the programmer, such as providing sensor information or state-machine transitions. You can use logging tools to analyze logging information and generate a postprocessing database. To achieve efficiency, you must carefully employ logging functions. For example, logging messages should start with keywords, such as "warning," "error," or "debug" and should identify the message initiator. You should group logging functions in a small set of files for easier maintenance and provide timestamp information if available. Unfortunately, logging is intrusive and modifies the software's realtime behavior to be unlike that ofthe final application.
The debug monitor, another popular debugging tool, works withcode running on the target (references 2 and 3) in the CPUmemory. A debugger running on ahost communicates with themonitor through a dedicated portto send commands and receiveresponses. You can consider thegdbserver program for Linux,although more complex than earlyROM monitors, as a debuggingmonitor (Figure 1). When a userwants to set a breakpoint on aninstruction, gdbserver saves theinstruction and replaces it with asystem call. Gdbserver then relieson the ptrace program from Linuxto get information on allapplications doing system calls.Subsequently, the gdbserver cantake control of the application youare debugging when the system calloccurs to invoke the breakpoint.The debugger runs on a hostmachine and connects to the targetthrough a serial or Ethernetconnection (Reference 4).Monitors are cheap and practicalbut have several disadvantages,such as the need to load the codebefore any debugging and possibleinteraction with applicationsoftware. You cannot use monitorsoftware if code resides in flashbecause it needs to patch theapplication to insert softwarebreakpoints.
IN-CIRCUIT EMULATION
The first hardware-based debugging technology, the ICE (incircuit emulator), is a version of the processor you are debugging. ICEs usually employ an FPGA (fieldprogrammable gate array). FPGAs bond out their internal buses and state signals and make them available to the user (Figure 2). ICEs provide more debugging capabilities than do ROM monitors. To use ICEs, you must replace the processor you are debugging on a board by a connection to an ICE box. A host running and emulating the functions of the debugger controls this box. One of the primary limitations of the ICE is its high cost. Additionally, although the method suits simple processors, modern SOCs with increasing complexity, integration, and frequencies make it difficult for IC vendors to provide ICE versionsfor the modern processors.
In 1985, a group of European companies formed the JTAG (Joint Test Action Group), a consortium to overcome the problems of testing semiconductor ICs. They created the IEEE 1149.1 standard for boundary-scan testing of ICs, which they released in 1990 (Reference 5 and Figure 3). The JTAG standard defines a limited I/O JTAG port with as many as five signals to perform testing and analysis of circuits through serial communication: TCK (test clock), TMS (test-mode select), optional TRST (test reset), TDI (test-data in),and TDO (test-data out).
The IEEE based JTAG hardware on a 16-state finite-state machine, which the TMS signal controls. The TCK's rising- edge clock captures this TMS signal. The data information shifts in on the TDI pad and shifts out on the TDO pad. You eventually use TRST to reset the design. You add scan registers for each pad of the IC and connect them internally to form a boundaryscan chain. You can shift this chain in and out through TDI/TDO and JTAG commands to test the external connections on a board, test the logic connections inside the IC, capture values of the pads, and put the JTAG chain in bypass mode.
TEST CAPABILITIES
JTAG provides low-cost manufacturing-test capabilities, and it has become the most common method for testing. But, due to its ease of use, high availability, and low-cost implementation, designers also frequently use JTAG as a debugging port to access on-chip debugging resources (Reference 6). JTAG is the transport layer of debugging communication between a debugger running on a host and the embedded processor's debugging resources. Instructions shift into the instruction register to access the debugging hardware IP. Due to the increasing cost of ICEs, many semiconductor vendors integrate more debugging hardware on-chip to cope with debugging limitations and provide similar functions to the ICE. One popular implementation of such on-chip debugging hardware is the BDM (background debugger mode) from Freescale Semiconductor Inc, formerly, Motorola) on 68-kbit Coldfire embedded processors and PowerPC (now, Power Architecture) processors. Other vendors use proprietary names forsimilar functions.
On-chip debugging hardware adds some functions, such as hardware breakpoints, internal register access, read/write to memory, and watchpoints, which you could previously access only through an ICE. In a multiprocessor SOC, you can connect each piece of chip-debugging hardware to the main JTAG controller. The connection may vary depending on the vendor, but the typical implementation creates a TDI-TDO JTAG chain between the on-chip debug JTAG state machines and thehost debugger (Figure 4).
TRACING
One of the biggest issues in real-time-system debugging is theHeisenberg bug, or probe effect: Any software or hardware you addfor debugging or monitoring islikely to change the real-timesystem's behavior. This situationcan happen when you add softwareto collect information for profiling,debugging, or monitoring. You cansee a similar impact when usingdebugging hardware. For example,on-chip debugging hardware maymodify the processor executionf low to insert breakpoints orprofiling hardware may "steal" partof the processor bandwidth tomemory to store profilinginformation.
Debugging hardware may also use the UART connection to log information when the production software is supposedly using this interface. ICE usually provides nonintrusive trace capabilities in which the developer can access the program counter of the processor at any time. More SOC vendors are integrating trace hardware on-chip to provide similar functions (Figure 5). Trace hardware achieves nonintrusiveness through the use of dedicated trace hardware, a dedicated trace port, separate data buses for trace data and processor data, and a JTAGinterface.
You can capture trace information using a trace port. In this method, you connect a trace box or a logic analyzer on the trace port to reconstruct the messages and correlate them with source code. Another method of capturing trace information is to use a virtual trace buffer, in which the processor's memory stores trace information and the host retrieves it—through the JTAG port, for instance—once the test ends. A third alternative is to use a dedicated trace buffer in which a dedicated memory stores the trace information, which the host retrieves once the test ends.
GENERAL-PURPOSE INTERFACE The IEEE-ISTO 5001TM 2003 Nexus 5001TM Forum Standard for a Global Embedded Processor Debug Interface provides an open, general-purpose interface for the software development and debugging of embedded processors (Reference 7). The Nexus Forum started its work in 1998 and issued its first release of the Nexus standard in 1999 with updates in 2003. The goal was to build on several vendors' experience in embedded-system debugging and tools to standardize on-chipdebugging features and interfaces. Because several of the vendors in the forum already offered proprietary products for on-chip debugging, the requirements for basic on-chip debugging are general enough to ease Nexus compliance. The benefit is the standardization of the Nexus trace interface, such as trace features, signals, messagingprotocol, and APIs (application-programming interfaces).Meanwhile, standardization leavesenough room for vendor-definedcustomizations. Initially targetingautomotive applications, the Nexusstandard has rapidly spread in thewireless and networking markets.
MULTICORE DEBUG
Caches provide high performance but are difficult to debug because they hide the CPU's execution from the external memory buses, and the coherency between cores and DMA or acceleration hardware is hard to understand. Embedded trace hardware is helping to solve this problem because the bus under trace is usually virtual (before the caches), not physical. It may also be very helpful to trace buses before and after caches to better understand the cache behavior (Figure 6).
Comparison of the resulting two traces provides good indication of the cache misses, when the cache generates accesses on the physical bus. This approach helps to reduce cache misses and enhance the performance of the software. Another option is to add embedded cache-debugging hardware to read the cache content in debugging mode or write into cache. This option is usually in the form of cache-debugging registers accessible through software or JTAG ports. With a debugger, a user can halt the program execution and check the cache's content. The user can employ this information to debug cache-cleaning issues, such as invalidation, synchronization, orflushing.
Modern SOCs often integrate several processors on one chip, and it becomes difficult to debug the interaction of the cores with traditional debugging hardware. Cross-triggering, a recently emerging debugging technique, has become popular for debugging complex multicore SOCs (Reference 8). The principle is to translate events from one core domain to generate triggers in another core's domain or to thesame core domain.
Typical events are entry into debugging mode, interrupt occurrence, watchpoint occurrence, and breakpoint occurrence. Input triggers are typically debugging requests. Triggers generate a debug request, an interrupt, or a glitch on one SOC pad. They can also start or stop thetrace on the processor.
The combination of triggers leaves all flexibility to the end user to create complex debugging sequences. You can use crosstriggers to start a trace on Core A when Core B reaches a specific program address or to stop Core B's activity when Core A entersdebugging, for instance.
With increasing pressure on SOC sizes, low-cost debugging may be the Holy Grail for embeddedsystem architects. But they must keep one thing in mind as their priority: Never compromise the debug capabilities of the system. If they couldn't anticipate all the bugs of their systems, they should at least increase their chances ofcapturing them in the future.
The cost savings of reduced embedded debugging hardware may be at the expense of higher software-debugging costs later in the project. They should also remember that debugging and security have contradictory requirements. Many manufacturers now ship products and simply disable the debugging features to protect themselves from hackers. This practice is unwise. You can never anticipate the type of issues you will encounter in the field. Protecting debugging access with security methods, such as keys or fuses, is a better option that will not jeopardize your debuggingcapabilities.
REFERENCES
1. "Rear Admiral Grace Murray Hopper, USNR".
2. O'Keeffe, Hugh, "Embedded Debugging: A White Paper," Ashling Microsystems Ltd.
3. Engblom, Jakob, "Debugging real-time multiprocessor systems: Parts 1 and 2," Virtutech, March 23, 2006, www.embedded.com/.
4. Best, Steve, "Mastering Linux debugging techniques," IBM, www.ibm.com.
5. JTAG IEEE Standard 1149.1-2001, Revision of IEEE Standard 1149.1- 1990, IEEE Standard Test Access Port and Boundary-Scan Architecture.
6. Mittag, Larry, "Software Debug Options on ASIC Cores," www.embedded.com.
7. The Nexus 5001 Forum Standard for a Global Embedded Processor Debug Interface Version 2.0, www.nexus5001.org.
8. Dannenberg, Andreas, "Hunting for bugs in real-world applications … more than a child's game," ww.embedded.com.
