Random interrupts in multicore scenarios

Article By : Aaron Bauch

How do you catch random interrupts or have control of the interrupt handling when applications are very complex?

The demand for complex applications is increasing and it’s a clear trend that multicore devices are becoming more and more popular. And as we all know “If we develop it, debugging will come”. But how do you catch random interrupts or have control of the interrupt handling when applications are very complex? In this article, I will discuss how to deal with these random interrupts and provide some guidance on tools that might be useful when developing Arm-based applications.

Interrupt basics

In embedded systems, using interrupts is a method for handling external events which are by nature, not synchronized to your software running on the system.  For example, detecting that a button was pressed. In general, when an interrupt event occurs, the core immediately stops executing the code it was running, and starts executing an Interrupt Service Routine, or ISR instead.

When the interrupt service code finishes doing what is needed to react to the external event, the processor should resume from the instruction following the one which was executing when the ISR started running.  If done properly, the main application code should not even be “aware” that anything changed.  It was simply paused for the time while the Interrupt Service Routine ran.

It is important that the state of the machine is restored after the interrupt is handled—this includes the values of processor registers and the processor status register. This makes it possible to continue the execution of the original code after the code that handled the interrupt was executed.

A professional compiler supports the syntax of writing interrupts, software interrupts, and fast interrupts in C/C++. For each interrupt type, an interrupt routine can be written but the syntax and handling will depend on the MCU implementation. For example, when Arm defined the Cortex variants of Cortex A, R and M, they also defined the interrupt handling hardware and mechanisms for each of these families. In previous chips of the Arm 7, Arm 9 flavors, etc., the interrupt controller design and operation varied by silicon vendor making portability difficult.

However, a modern commercial toolchain will take these differences into account and make implementing interrupt support in your code as transparent as possible regardless of the underlying interrupt hardware and mechanisms for interrupt management.

A well understood and frequently discussed practice is that interrupt code should be as short as possible. This assures that the CPU can return to the main task in a timely manner. The interrupt service routine should only execute the critical code and the rest of the task can be relegated to the main process by setting a flag variable. If you call a normal function from an Interrupt Service Routine, you may get unexpected code bloat preserving unused registers.

One useful technique for managing this process is through the use of deferred processing using software interrupts.

In this method, the hardware interrupt, such as the button press mentioned above, triggers a high priority interrupt service routine.  In this routine, you will handle that which must be handled at once such as resetting the switch so that it can fire again.  However, the ISR may post a lower priority software interrupt which will run when it is the highest priority event in the system.  In this way, if the priority is set lower than some important time-critical application functions, it will be deferred while these functions run.  But as soon as its priority is higher than the other code that might run, it will execute and finish the less time-critical functions needed for management of the interrupt event.

If you have good reasons for doing function calls inside an ISR, the key thing you can do to is to provide information to the compiler so that it may be able to optimize the code as best it can. In general, you will need to check what happens in your particular situation, but you could try defining the called function a) static and b) in the same file (compilation unit) as the ISR. By doing this, the compiler can know exactly what registers are used by the called function when compiling the ISR, and you indicate to the compiler that this function is only called from inside the same source file (compilation unit). If there is a pragma or option to encourage the compiler to generate inline code, you could try this as well. With this information, it is possible that the compiler will inline the called function, allowing for further optimizations similar to if the called function had actually been written inline (e.g., removing not needed push/pop pairs). You should turn on appropriate optimization and look at the generated code to see that it does what you want as efficiently as possible.

Interrupts in multicore systems

By using multicore devices, it’s possible to load balance and take advantage of the different strength of the different cores. It will also possibly lower the product cost to have one multicore device rather than multiple devices. Different multicore systems may handle interrupts differently.  In more complex, SMP systems where code is arbitrarily scheduled to run on one of several identical cores, interrupt handling may also be randomly assigned to available cores based on some algorithm.

However, the more interesting model for embedded systems are using an asymmetric multi processing approach where each core has its own program which it executes, occasionally communicating and synchronizing with one or more other cores in the system.  This is the model we will focus on in this discussion.

In this kind of system, each interrupt is generally directed to one (and only one) CPU. The programmable interrupt controller (PIC) of the chip controls how this happens. When you initialize the PICs in your system’s startup, you can program them to provide interrupts to any CPU you like.

Multicore Arm chips often use the Advanced Programmable Interrupt Controller (APIC) and also integrated interrupt controller that implements the Generic Interrupt Controller (GIC).This can be configured to deliver I/O interrupts to particular cores or groups of cores. In addition, it provides interprocessor interrupts (IPI) which are used by the operating system to coordinate the activities of multiple cores.

We will not explore the specific MCU/MPU multicore configurations further in this article, we will rather delve into how to monitor and debug interrupts.

The benefits of multicore debugging

Interrupts are notoriously difficult to debug in single core systems, because stopping to inspect the execution of an interrupt handler in the debugger doesn’t go very well with the typical time sensitive aspect of interrupt handling. It becomes even more complex in multicore scenarios with the possibility to balance interrupts and set or redirect a different core in case one is completely occupied by other interrupts. Since interrupts are often triggered by peripherals or external events, certain bugs may be triggered only rarely and seemingly at random or by having the interrupts being connected to a wrong core or busy core.

A multicore debugger can stand out and facilitate the task by showing the programs running on multiple cores. Particularly it has the ability to selectively stop and start cores based on breakpoints inside an Interrupt Service Routine (ISR) on another core (Figure 1).

click for full size image

Figure 1: Multicore debug control with independent start/stop options. (Source: IAR Systems)

It is possible to run multicore debugging in two ways.

  • Symmetric multicore debugging (SMP), which means debugging two or more identical cores.
  • Asymmetric multicore debugging (AMP), which means debugging two or more cores based on different architectures. It could be two different Arm cores, for example a Cortex–A9 and a Cortex–M0.

Full control over all cores is essential for the best results.

The Arm Cortex family of microcontrollers adds several advanced debugging features that were not available in previous Arm microcontrollers. The serial wire debug, or SWD, via the Serial Wire Output (SWO) interface provides access to features of the CoreSight debug infrastructure such as timestamping and tracing of interrupt events.

Notice that most of the software and hardware debuggers support SWO in only one of the cores at a time in multicore scenarios. This can best be combined to monitor one of the cores and make use of breakpoints and the Cross Trigger Interface (CTI) interface to selectively stop and start adjacent cores.

Interrupt logging provides you with comprehensive information about the interrupt events. This might be useful, for example, to help you locate which interrupts you can fine-tune to become faster. You can log entrances and exits to and from interrupts. The interrupt log, shown in Figure 2, allows you to see a log of the execution of interrupt service routines and how long each one takes to execute.

click for full size image

Figure 2: Example of interrupt logging. (Source: IAR Systems)

The timestamp information can be available either as a real-time figure or in CPU clock cycles.

We can see the number of times that each interrupt has been triggered, as well as statistics on the length of time taken to execute the ISR (Figure 3). It is also possible to see the interrupt log info graphically in Figure 4.

click for full size image

Figure 3: Example of interrupt log summary. (Source: IAR Systems)

click for full size image

Figure 4: Timeline with graphical interrupt being displayed. (Source: IAR Systems)

This allows us to see a graph of when each ISR was active and compare it to other ISRs, as well as correlate to other timed activities such as data log breakpoints.

Another useful and powerful feature when working and debugging interrupts is the use of trace. Real-time trace is a continuously collected sequence of every executed instruction for a selected portion of the execution. By using trace, you can inspect the program flow up to a specific state, for instance an application crash, and use the trace data to locate the origin of the problem. Trace data can be useful for locating programming errors that have irregular symptoms and occur sporadically like random interrupts.

Trace is also mostly available to be viewed one core at a time in multi-core scenarios. There are devices with trace funnels, which combine the trace from each source into a single flow with a more complex chip to control. Having the trace information on one core and taking advantage of the CTI interface to stop and start all other cores will give you precise picture on how your application is behaving.

A professional trace debugger can provide information for various aspects of your application, collected when the application is running. The information can be represented in a timeline, as a graph that displays the sequence of function calls and returns collected by the trace system. You can also get timing information between the function invocations. This can also help you to analyze the application’s behavior, as well as allow you to quickly to navigate to the next interrupt entry or to the closest previous interrupt entry aside on having a clear picture on what routines have been interrupted. The graphical call stack can display the sequence of function calls and returns collected by the trace system, and it also includes the interrupts and ISR that have been triggered and handled with the precise time and cycle counts (Figure 5).

click for full size image

Figure 5: Timeline with graphical call stack with functions and ISR. (Source: IAR Systems)

This capability provides comprehensive information about exceptions and interrupts in the system. It is useful, for example, to locate which interrupt can be fine-tuned to execute faster, or analyze problems with nested interrupts.

Conclusion

It is very common that developers mainly disable all interrupts in order to avoid concurrent interrupts or random interrupts. This might however make the systems react or slower and hide some critical issues that might be triggered in the future and make things worse. Having full control and understanding when and why interrupts are triggered is the most important step to ensure good performance and having the application behaving exactly as expected. A debugger with the capability of logging all interrupt handling is crucial for catching random interrupts and having control of the interrupts handing.

This article was originally published on Embedded.

Aaron Bauch is a Senior Field Application Engineer at IAR Systems working with customers in the Eastern United States and Canada. Aaron has worked with embedded systems and software for companies including Intel, Analog Devices and Digital Equipment Corporation. His designs cover a broad range of applications including medical instrumentation, navigation and banking systems. Aaron has also taught a number of college level courses including Embedded System Design as a professor at Southern NH University. Mr. Bauch Holds a Bachelor’s degree in Electrical Engineering from The Cooper Union and a Masters in Electrical Engineering from Columbia University, both in New York, NY.

 

Leave a comment