How to visualize response times in FreeRTOS

Efficient development of FreeRTOS-based firmware requires good understanding of the timing and interactions between tasks, interrupts and the kernel. Percepio Tracealyzer is the premier solution for analysis and visualization of FreeRTOS-based embedded software. It provides more than 25 graphical views showing different aspects of the software’s real-time behavior, interconnected in clever ways and very intuitive to use.

We collect and reproduce examples of how Tracealyzer has been applied on real-world issues by our customers. In this case, the customer developed a networked system running a TCP/IP stack, a flash file system, and an RTOS running on an ARM Cortex-M4 microcontroller. The system contained several RTOS tasks, including a server-style task that responds to network requests, and a log file spooler task. The response time on network requests had often been an issue and when testing their latest build, the system responded even slower than before. So now they really wanted to figure this out!

Obviously, they started by comparing the code between the two versions. But they could not see any obvious reason for the longer response time. There were many small changes due to refactoring, but no new functions had been added. They therefore decided to use Tracealyzer to compare the runtime behaviors of the earlier version and the new version.

Traces were recorded from both versions in similar conditions and they began at the comparison in the Statistics Report (Figure 1A and Figure 1B), that includes high-level timing statistics like CPU usage, number of executions, scheduling priorities, and response times.

Figure 1A: Earlier version Figure 1B: New version

As expected, the Statistics Report revealed that response times for Server Task are higher in the new version, here about 50 % higher on average. The execution times of the Server task (i.e., the cycles actually used) were however quite similar, only about 7 % higher in the new version. So the main reason of the higher response time must be other tasks that interfere.

To find out what is causing this difference, we simply click on the extreme values in the Statistics Report. This focuses the main trace view on the corresponding locations, so we can see the details, as illustrated below. And by opening two parallel instances of Tracealyzer, one for each trace, we can easily compare and see the differences.

Figure 2A: Earlier version Figure 2B: New version

Since the application server task performed several services, two User Events have been added to mark the points where the specific request are received and answered, labeled “ServerLog” in Figure 2A and 2B. The zoom levels are identical, so we can clearly see the higher response time in the new version. We can also see that the Logger task preempts the Server task 11 times, compared to only 6 times in the earlier version, which is a significant difference. Moreover, we see that the Logger task is running on higher priority than Server task, so every logging call preempts the Server task. So there seems to be new logging calls added in the new version, causing the Logger task to interfere more with the Server task.

To see what is logged, let’s add a User Event in the Logger task to show the messages in the trace view. Perhaps some can be removed to improve performance?

Figure 3: Added a “user event” in Logger task.

Now we see that also other tasks generate logging messages, for instance the ADC_0 task (in yellow). To see all tasks sending messages to the Logger task, we can use the Communication Flow graph, illustrated in Figure 4.

Figure 4: Communication Flow graph

The Communication Flow graph shows a summary of all operations on message queues, semaphores and other kernel objects, performed by tasks and interrupts in the trace. This visualizes the high-level design and runtime dependencies in the recorded situations.

In this case, the Communication Flow reveals five tasks that sends logging messages. By double-clicking on the “LoggerQueue” node in the graph, the Kernel Object History view is opened and shows all operations on this message queue (Figure 5). As expected, the can see that Logger task receives messages frequently, one at a time, and is blocked after each message, as indicated by the “red light”.

But is this a really good design? It is probably not necessary to write the logging messages to file one-by-one. If we would increase the scheduling priority of Server task above that of the Logger task, the Server task would not be preempted as frequently and would thereby be able to respond faster. The logging messages would be buffered in LoggerQueue until the Server task (and other high priority tasks) has completed. Only then would the Logger task be resumed and process all buffered messages in a batch.

Figure 5: Object history view, showing operations on a particular message queue.

Let’s try that. Figure 6 shows the result. The highest response time is now just 5.4 ms instead of 7.5 ms, which is even faster than in the earlier version (5.7 ms) despite more logging. This since the Logger task is no longer preempting the Server task, but instead processes all pending messages in a batch after Server is finished.

Here we also see “event labels” for the message queue operations. As expected there are several “xQueueSend” calls in sequence, without blocking (= red labels) or task preemptions. There are still preemptions by the ADC tasks, but this no longer causes extra activations of the Logger task.

Problem solved!

Figure 6: Highest response time of Server Task after changing the priorities.

How does it work? Tracealyzer uses software-defined trace, which is very flexible and works on any processor. To record a trace, you only need to include Percepio’s recorder library in your build, configure it and start the tracing.

The performance overhead is only a few microseconds per event, and you can stream the trace continuously to the host computer via a debug probe, TCP/IP or other channels. The trace can also be kept in a target-side RAM buffer and uploaded on demand.

To learn more and get started, please refer to the following on-line resources:

- Product page: http://percepio.com/tz/freertostrace

- User guide: http://percepio.com/docs/FreeRTOS/manual

- Getting started: http://percepio.com/gettingstarted

If you have any questions, please contact support@percepio.com or your local distributor.