Debugging Fundamentals

Embedded systems are microprocessor or microcontroller-based systems with a dedicated operational role. Rather than being made of separate components, like desktop computers, laptops or, gaming consoles, embedded systems integrate all the hardware and software necessary for a particular purpose. Nowadays, embedded systems are everywhere: automobiles, cameras, household appliances, and mobile devices are just some examples.

Embedded systems design can be challenging since it combines hardware design, firmware, and software development, all in one particular device or product. In order to produce high-quality embedded firmware and software for a particular device or product, debugging is a necessary step in their development process. Debugging is the process of confirming that, one by one, many things that we believe to be true and functional in our code are true. We find a "bug" in our code when one our more assumptions are not valid.

People worldwide have been talking about "bugs" for a long time; even Thomas Alva Edison used the word back in his days. The word bug has been used as an old term for "monster"; like gremlins in machinery, bugs are malicious.

The following article will discuss different debugging tools and techniques used to find bugs in microcontroller-based systems, especially those based on Arduino® hardware.

Debugging tools and techniques

There are some basic debugging tools and techniques that we can use and implement to validate our code:

• The compiler and syntax errors.
• Traditional techniques: trace code and GPIOs.
• Remote debuggers.
• Simulators.
• In-circuit emulators and in-circuit debuggers.
• Hardware tools: multimeters, logic analyzers, oscilloscopes, and software-defined radios.

Let us take a look into each one of the debugging tools and techniques.

The Compiler and Syntax Errors

Compiling is transforming high-level code into machine language that can be understood by a processor, for example a microcontroller. In this process, the compiler also helps to identify syntax errors. Syntax errors indicate something wrong with the program's syntax; for example, when a semi-colon is omitted at the end of a statement in a program, the compiler generates a syntax error.

Using the compiler for debugging syntax errors can be sometimes tricky; let us analyze two commonly encountered situations:

• The compiler shows 100 errors: This usually does not mean there are 100 errors; it often gets thrown off track for a while when the compiler finds an error. The compiler tries to recover and pick up again after the first error, but sometimes it reports false errors. Only the first error message is genuinely reliable; try to fix one error at a time and then recompile the program.
• Getting weird compiler messages: Compiler errors are shown in terse jargon, sometimes hard to read and understand, but with hidden and valuable information. Read the error message carefully; it will always tell where, inside the code, the error occurred.

Traditional Techniques: Trace Code and GPIO's

Adding trace code is probably the simplest and most basic debugging technique used in embedded systems software design. This method adds trace code to the program to print out messages or values of variables (using the

Serial.print()

Copy
1// Print a message if the execution gets here
2Serial.println("Code got here");
3
4// Try to execute myFunction1()
5myFunction1(); 
6
7// Print a message if the execution gets here
8Serial.println("Code got here, myFunction1 executed"); 
9
10// Try to execute myFunction2()
11myFunction2(); 
12
13// Print a message if the execution gets here
14Serial.println("Code got here, myFunction2 executed");

Using trace code for debugging is usually applicable only during the early stages of an embedded system code development. Adding trace code to our programs takes a significant amount of processing time and resources. Therefore, it can easily disrupt critical timing tasks in our programs. Additionally, if we use the UART for other tasks, we can have problems displaying trace code.

You can pass flash-memory based Strings to

Serial.print()

F()

Serial.println(F("Code got here"))

Another trace code technique consists of dumping strategic information into an array at run time, we can then observe the contents of the array at a later time (for example, when the program terminates); this technique is also known as dump into array. Assume

good

bad

Copy
1#define DUMP_BUFFER_SIZE 32
2unsigned char goodBuffer[DUMP_BUFFER_SIZE];
3unsigned char badBuffer[DUMP_BUFFER_SIZE];
4unsigned long count = 0;

The variable

count

good

bad

Copy
1void Save_Debug_Buffer(void) {
2  if (count < DUMP_BUFFER_SIZE) {
3    goodBuffer[count] = good;
4    badBuffer[count] = bad;
5    count++;
6  }
7}

General Purpose Input/Output (GPIO) pins can help debug purposes when the UART is in use or adding trace code is not particularly helpful. For example, we can turn on or off the built-in LED of an Arduino® board by inserting a

digitalWrite(LED_BUILTIN, HIGH)

Copy
1// Print a message if the execution gets here
2Serial.println("Code got here");
3
4// Try to execute myFunction1()
5myFunction1(); 
6
7// Turn on the built-in LED for one second to indicate that myFunction1 was executed
8digitalWrite(LED_BUILTIN, HIGH); 
9delay(1000);
10digitalWrite(LED_BUILTIN, LOW); 
11
12// Try to execute myFunction2()
13myFunction2(); 
14
15// Turn on the built-in LED for one second to indicate that myFunction2 was executed
16digitalWrite(LED_BUILTIN, HIGH); 
17delay(1000);
18digitalWrite(LED_BUILTIN, LOW);

Remote Debuggers

Remote debugging is another common approach used to debug embedded systems. Remote debuggers work by connecting a particular embedded system to a host computer and then using software in the host computer to interact with the embedded system hardware. Remote debuggers are helpful when the development environment is on a different architecture rather than the target system. For example, think about developing code on a Windows-based computer for an ARM-based microcontroller system.

Remote debuggers usually have two essential parts: a front-end debugger and a back-end debugger.

• The front-end debugger contains the user interface (can be graphical or command-line-based) and offers the programmer choices about the execution of the code in the embedded system hardware.
• The back-end debugger, also known as the "debug monitor," is specific for a particular processor architecture or family an usually work with an external hardware tool, like an in-circuit emulator or an in-circuit debugger. It starts when the processor resets and handles the runtime instruction between the front-end debugger and the embedded system hardware.

The debugger tool is a newly introduced yet less famous feature of Arduino IDE 2. Check out this tutorial that shows how to use the Arduino® IDE 2 debugger with supported boards.

Simulators

Simulators are tools used to simulate the functionality and the instruction set of the target processor. These tools, usually, can only simulate the target processor functionalities but not its environment and external parts and components. Simulators are handy in the early stages of the development process, where we only have the software but have not implemented any hardware yet.

Tinkercad Circuits is an excellent simulator for beginners in the Arduino® ecosystem. This tool can simulate the instruction set of an Arduino® UNO board and the functionalities of several electronic components such as resistors, LEDs, motors, LCDs, and some sensors.

In-Circuit Emulators and In-Circuit Debuggers

An in-circuit emulator (or ICE) is a specialized tool that allows developers to examine the state of the processor while a particular program is running. ICEs are considered embedded systems by themselves; they are a copy of the target processor and its memory (RAM and ROM); this is why they provide an unintrusive way to debug code at the target processor. Historically, ICEs were the tool of choice of embedded systems developers, but as processor complexity and clock speed increased, ICEs became more expensive, and their availability declined considerably.

Binarysequence, CC BY-SA 3.0, via Wikimedia Commons.

An in-circuit debugger (or ICD) is also a specialized tool connected between a host computer and a processor for debugging real-time applications faster and easier; this tool uses some memory and GPIO pins of the target microcontroller during the debugging operations. With an ICD, developers access an on-chip debug module which is integrated into the CPU over an interface (for example, JTAG). This debug module allows developers to load, run, halt and step the processor.

SEGGER Microcontroller GmbH & Co, CC BY-SA 3.0, via Wikimedia Commons.

The fundamental difference between an ICE and an ICD relies on the resources used to control the debug target. In ICEs, resources are provided by the emulation hardware; in ICDs, resources are provided by the target processor.

Arduino® ICD Support

The Arduino® boards with a SAMD microcontroller support ICD debugging; these boards are the following:

• Zero.
• Nano 33 IoT.
• MKR Zero.
• MKR WiFi 1010.
• MKR WAN 1300.
• MKR WAN 1310.
• MKR FOX 1200.
• MKR NB 1500.
• MKR GSM 1400.
• MKR Vidor 4000.

The Arduino® Zero board features an on-board debugger, the Atmel® Embedded Debugger (EDGB). In addition to programming and debugging support, the EDGB also offers data streaming capabilities between the host computer and the target processor. Check out this tutorial to learn how to use the Arduino® Zero board debugging capabilities with the Arduino IDE 2.

Arduino® boards with a SAMD microcontroller feature native on-chip debug capabilities; these debugging capabilities can be used with an external ICD tool over JTAG or SWD interfaces. CMSIS-DAP compliant debug probes can be used with the Arduino IDE 2 out of the box without any configuration file; non-standard debug probes require a special configuration. Check out these tutorials to learn how to use an external ICD tool with SAMD based Arduino boards and the Arduino IDE 2:

• Debugging with the SEGGER J-Link.
• Debugging with the Atmel-ICE.

The Arduino® Portenta H7, H7 Lite, and H7 Lite Connected boards from the Pro family also support ICD debugging; these boards use the TRACE32 debugger from Lauterbach. The TRACE32 debugger allows testing embedded hardware and software by using the in-circuit debug interface of processors. Check out this tutorial to learn how to use the TRACE32 debugger with the Portenta family boards:

• Lauterbach TRACE32 GDB Front-End Debugger for Portenta H7.

Hardware Tools

Embedded systems developers and typical software developers differ on a key aspect: their "closeness" to hardware; embedded system developers are usually closer to hardware than typical software developers. There are several tools that embedded systems developers use to find out what is going on with the hardware, which is very helpful for low-level software debugging. These tools are multimeters, logic analyzers, oscilloscopes, and software-defined radios (SDRs).

Let us take a look at each one of the hardware debugging tools. A basic understanding of these tools can significantly improve debugging skills, especially while developing embedded systems.

Multimeters, logic analyzers, oscilloscopes and, software-defined radios help debug interactions between the processor and other electronic parts on an embedded system. These tools do not control the flow of execution of the code of an embedded system.

Multimeters

A digital multimeter (DMM) is a hardware tool that can be used to measure two or more electrical values, usually voltage (in volts), current (in amps), and resistance (in ohms). DMMs are great tools and one of the most fundamental pieces of test equipment that can be used to debug electrical problems within an embedded system.

oomlout, CC BY-SA 2.0, via Wikimedia Commons.

Logic Analyzers

A logic analyzer is a hardware tool designed specifically for capturing, displaying, and measuring electrical signals in a digital circuit. This tool consists of several digital inputs pins capable of detecting whether an electric signal is at a specific logic level (1 or 0). Logic analyzers are also capable of showing the relationship and timing of different electrical signals in a digital circuit and often capable also of analyzing digital communication protocols (for example, SPI communication protocol).

SparkFun Electronics, CC BY-SA 2.0, via Wikimedia Commons.

Oscilloscopes

An oscilloscope is a hardware tool that graphically displays electrical signals and shows how those signals change over time. Signals are measured in an oscilloscope with the help of a sensor.

Dave Jones from Australia, CC BY-SA 2.0, via Wikimedia Commons.

Software-Defined Radios

A software-defined radio (SDR) is a radio communication system that uses software for the modulation and demodulation of radio signals. Traditional radio communications systems processing relies on hardware components; this limits their reprogramming to a very low level. SDRs are much more flexible since they can be reconfigured by software.

Alexander Neumann, Public domain, via Wikimedia Commons.

Debugging with Hardware Tools

While there may be several debugging techniques, using a LED as a pass mark for the debugging process is the simplest yet fastest method one can utilize. The indicator will be set in different points of interest to observe the correct execution of tasks visually. For instance, there can be multiple points located simultaneously to turn it on or off the LED on a single point of interest at a time for step-by-step verification. This will provide just enough information to construct an additional layer of the code or proceed to the following structure sector to debug its behavior. It will not give precise or in-depth information about registry or data exchange, so it has to be used as a tool for code structures that are not complex in their architectural nature and behave mainly in a linear trend execution. It is handy when a debugger is not available and quickly understand how the code behaves.

Sometimes, LEDs might not be present or might not be available in a particular system; there is no way to make a visual inspection in the system. However, we can use an oscilloscope directly to monitor the state of GPIO pins of the system in this case. The oscilloscope, in this case, can be used to monitor specific GPIO pins and see if the code gives specific feedback by driving the GPIO pin to the desired logic state. A DMM can also be handy for the same task.

To get the most out of an oscilloscope and GPIO pins is by measuring its performance, this means to determine a signal's electrical and timing properties. For example, an unnecessary delay in the code can be identified with this information:

Copy
1void myFunction() {
2  digitalWrite(LED_BUILTIN, HIGH); 
3  Serial.println("Code got here");
4  count++;
5  digitalWrite(LED_BUILTIN, LOW); 
6}

myFunction()

myFunction()

Now let us talk about wireless communications. Wireless communications are a key feature in developing new Internet of Things (IoT) devices with different requirements or specifications and for different purposes. Wireless communication is present on many embedded systems, and Arduino® hardware is no exception for this feature. The question now is: how do we debug wireless communications between devices?

A simple and common technique used to debug wireless communications between two or more devices consists of using acknowledge flags. Using acknowledge flags aids in understanding the device's behavior when communication is established between them by providing their current status. This process is also found on physical communication protocols, such as I2C or SPI. Because of the different protocol types in wireless communications, using acknowledge flags as a debugging method can be applied but differ in their own rules. The easiest way to confirm that the data exchange was successful is to check the data log on each end device. Hardware tools mentioned earlier (DMMs, oscilloscopes, and logic analyzers) can be used to provide more in-depth details and add more value to this type of debugging technique.

However, not everything is connected on a physical layer, but on an abstract layer. This abstract layer in wireless communications is where electromagnetic waves propagate through space. Why would we need to debug electromagnetic waves propagating through space? Sometimes, we need to verify that the wireless transceiver configuration of a particular embedded system is correct, for example, its transmission power. SDRs can be helpful in this situation: SDRs can be used as cheap spectrum analyzers. A spectrum analyzer is a hardware tool that measures the magnitude of a signal versus the frequency; this tool is mainly used to measure a signal's power spectrum. Using SDRs as spectrum analyzers is usually an optional debugging method in the development process, ensuring a more robust system design.

Several programs can be used with SDRs; GQRX is one of the most popular software; GQRX is open-source and with cross-platform support (Linux and macOS). AirSpy and CubicSDR are other popular software used with SDRs and cross-platform support (Windows, Linux, and macOS).

Shown visual representation of the signal via SDR software can now be used to verify the transmission power outputted by the device and the amount of data transmitted. This will help visualize the device's wireless communication configuration properties. It will be possible to verify the transmission and reception power, the number of bytes transmitted, and the frequency it is supposed to transmit. These properties can be debugged through the frequency spectrum and refined to provide edge wireless communication performance on embedded systems.

Debugging Techniques Example

A simple example will demonstrate the implementation of some of the debugging techniques we discussed before and how they can be handy for the development process of code in the Arduino® ecosystem. We will use the Arduino® Nano 33 BLE Sense board, and its onboard inertial measurement unit (IMU) features to show the debugging process's importance. The example code uses accelerometer, gyroscope, and magnetometer data simultaneously, having the tasks be executed to obtain every value:

Copy
1/*
2  Program:
3  - Debugging_techniques_example.ino
4
5  Description:
6  - This example combines data from the LSM9DS1 accelerometer, gyroscope, and magnetometer into a single code. On top of it,
7  it helps to understand different methods and techniques of debugging when the code structure combines multiple tasks.
8
9  The circuit:
10  - Arduino Nano 33 BLE Sense.
11
12  Code based on examples created by Riccardo Rizzo, Jose García, and Benjamin Dannegård.
13  Modified by Taddy Ho Chung and José Bagur (16/02/22).
14*/
15
16#include <Arduino_LSM9DS1.h>
17
18#define DUMP_BUFFER_SIZE 32
19unsigned char GoodBuffer[DUMP_BUFFER_SIZE];
20unsigned char BadBuffer[DUMP_BUFFER_SIZE];
21unsigned long count = 0;
22uint8_t good, bad = 0;
23
24float x, y, z, ledvalue;
25int degreesX = 0, degreesY = 0;
26int plusThreshold = 30, minusThreshold = -30;
27
28void setup() {
29  Serial.begin(9600);
30  while (!Serial);
31  Serial.println("- Started");
32
33  if (!IMU.begin()) {
34    Serial.println("- Failed to initialize IMU!");
35    
36    bad++;
37    save_bebug_buffer();
38    disp_debug_buffer();
39    debug_stop();
40  }
41
42  accelermeter_setup();
43  gyroscope_setup();
44  
45}
46
47void loop() {
48  for (int i = 0; i < 5; i++) {
49    accelerometer_task();
50    gyroscope_task();
51    magnetometer_task();
52  }
53  
54  save_debug_buffer();
55  debug_stop();
56}
57
58// Accelerometer setup
59void accelermeter_setup() {
60  Serial.print(F("- Accelerometer sample rate: "));
61  Serial.print(IMU.accelerationSampleRate());
62  Serial.println(F(" Hz"));
63}
64
65// Read accelerometer data in all three directions task
66void accelerometer_task() {
67  if (IMU.accelerationAvailable()) {
68    Serial.println(F("- Accelerometer data ready"));
69    IMU.readAcceleration(x, y, z);
70    good++;
71  } else {
72    Serial.println(F("- Accelerometer data not ready"));
73    bad++;
74  }
75
76  if (x > 0.1) {
77    x = 100 * x;
78    degreesX = map(x, 0, 97, 0, 90);
79    Serial.print(F("- Tilting up "));
80    Serial.print(degreesX);
81    Serial.println(F("  degrees"));
82  }
83
84  if (x < -0.1) {
85    x = 100 * x;
86    degreesX = map(x, 0, -100, 0, 90);
87    Serial.print(F("- Tilting down "));
88    Serial.print(degreesX);
89    Serial.println(F("  degrees"));
90  }
91
92  if (y > 0.1) {
93    y = 100 * y;
94    degreesY = map(y, 0, 97, 0, 90);
95    Serial.print(F("- Tilting left "));
96    Serial.print(degreesY);
97    Serial.println(F("  degrees"));
98  }
99
100  if (y < -0.1) {
101    y = 100 * y;
102    degreesY = map(y, 0, -100, 0, 90);
103    Serial.print(F("- Tilting right "));
104    Serial.print(degreesY);
105    Serial.println(F("  degrees"));
106  }
107  
108  delay(1000);
109}
110
111// Gyroscope setup
112void gyroscope_setup() {
113  Serial.print(F("- Gyroscope sample rate = "));
114  Serial.print(IMU.gyroscopeSampleRate());
115  Serial.println(F(" Hz"));
116  Serial.println();
117  Serial.println(F("- Gyroscope in degrees/second"));
118}
119
120// Read gyroscope data in all three directions task
121void gyroscope_task( ) {
122  if (IMU.gyroscopeAvailable()) {
123    IMU.readGyroscope(x, y, z);
124    Serial.println(F("- Gyroscope data ready"));
125    good++;
126  } else {
127    Serial.println(F("- Gyroscope data not ready"));
128    bad++;
129  }
130
131  if(y > plusThreshold) {
132    Serial.println(F("- Collision front"));
133    delay(500);
134  }
135
136  if(y < minusThreshold) {
137    Serial.println(F("- Collision back"));
138    delay(500);
139  }
140
141  if(x < minusThreshold) {
142    Serial.println(F("- Collision right"));
143    delay(500);
144  }
145
146  if(x > plusThreshold) {
147    Serial.println(F("- Collision left"));
148    delay(500);
149  }
150}
151
152// Read magnetometer data in all three directions task
153void magnetometer_task(){
154  IMU.readMagneticField(x, y, z);
155  
156  if(x < 0) {
157    ledvalue = -(x);
158  }
159  else {
160    ledvalue = x;
161  }
162  
163  analogWrite(LED_BUILTIN, ledvalue);
164  delay(500);
165}
166
167// For debugging purposes
168void save_debug_buffer(void) {
169  if (count < DUMP_BUFFER_SIZE) {
170    GoodBuffer[count] = good;
171    BadBuffer[count] = bad;
172    disp_debug_buffer();
173    count++;
174  }
175}
176
177// Debugging array buffer
178void disp_debug_buffer() {
179  Serial.println(F("\n Debugging array buffer result >>"));
180  Serial.print(F("- Good marks: "));
181  Serial.println(GoodBuffer[count]);
182  
183  Serial.print(F("- Bad marks: "));
184  Serial.println(BadBuffer[count]);
185}
186
187void debug_stop() {
188  Serial.flush();
189  exit(1);
190}

As shown, the complete code unifies the accelerometer, gyroscope, and magnetometer into a single code structure. As it involves tasks from different modules, it is separated into different functions and executed in a more identifiable manner. The code includes a trace code technique for debugging (dump into an array) to understand precisely how the code operates. The

good

bad

It is crucial to know when to stop the code from debugging. While the code shown above can be debugged at runtime, it is much easier to know when to stop or how to stop the code operation. For instance, stopping within the first run gives the following result:

In the Arduino Serial Monitor, we can observe that it has a

1

1

The accelerometer performed its task without any issue except the first runtime instance, resulting in

9

1

Serial.println(F())

Additionally, it is possible to modify the loop code by simply adding a

digitalWrite(12, HIGH)

digitalWrite(12, LOW)

Copy
1void loop() {
2  for (int i = 0; i < 5; i++) {
3    digitalWrite(12, LOW);
4    accelerometer_task();
5    gyroscope_task();
6    magnetometer_task();
7    digitalWrite(12, HIGH); 
8  }
9  
10  save_debug_buffer();
11  debug_stop();
12}

Final Thoughts about Debugging

Debugging is a necessary step for developing robust and reliable embedded systems software. We can end this article by mentioning the four most essential phases of debugging stated by Robin Knoke in this article about debugging embedded C that was published in the Embedded Systems Programming magazine:

• Testing: this phase exercises the capability of the embedded software by stimulating it with a wide range of input values and in different environments.
• Stabilization: this phase attempt to control the conditions that generate a specific bug.
• Localization: this phase involves narrowing the range of possibilities until the bug can be isolated to a specific code segment in the embedded software.
• Correction: this phase involves eradicating the bug from the software.

Knowing the potential causes of bugs allows us to adopt strategies that minimize their occurrence. Many different debugging techniques and external devices are present to aid this process. Maybe some software designs do not require the usage of external debuggers, for example. However, when the software involves different requirements, especially scalability, things change drastically for the development process. The debugging techniques and the external debuggers will support this development process, thus granting sophisticated software. In most cases, we will know how the device will behave with the software, its computational performance, and even achieve non-power hungry devices due to clean memory management.

Debugging may be an overlooked aspect in developing embedded systems, but it is its most serious yet crucial tool. If we desire to develop robust and reliable embedded systems, the debugging process should consistently be implemented to achieve these goals.

Further Reading and Resources

Debugging is an exciting topic to study; if you want to learn more about debugging tools and techniques, check out the following links:

• Do you want to improve your debugging and engineering skills? A highly recommended reading is Debugging: The 9 Indispensable Rules for Finding Even the Most Elusive Software and Hardware Problems by David J. Agans.
• Do you want to learn more about digital multimeters? Learn more about them in this article from Fluke®.
• Do you want to learn more about oscilloscopes? Learn more about them in this article from Tektronix®.
• Do you want to learn more about logic analyzers? Learn more about them in this article from Saleae®.
• Do you want to learn more about spectrum analyzers? Learn more about them in this article from Tektronix®.
• Do you want to learn more about SDRs? Check out the Great Scott Gadgets video series about SDRs. The video series from Great Scott Gadgets is a complete course about SDRs. You will learn the fundamentals of Digital Signal Processing (DSP) and build flexible SDR applications using GNU Radio.

References

[1] P. Koopman, Better Embedded System Software. S.L.: Drumnadrochit Press, 2010.
[2] D. J. Agans, Debugging: The Nine Indispensable Rules for Finding Even the Most Elusive Software and Hardware Problems. New York: Amacom, 2002.
[3] M. Barr and A. Massa, Programming Embedded Systems: with C and GNU Development Tools. Johanneshov: MTM, 2013.
[4] J. W. Valvano, Embedded Systems: Introduction to ARM® Cortex™-M Microcontrollers. United States: Self-published, 2015.