© 2010 inhand electronics, inc. – troubleshooting real-time software issues using a logic...

© 2010 InHand Electronics, Inc. – www.inhand.com

Troubleshooting Real-Time

Software Issues

Using a Logic Analyzer

Dave Stewart

Director of Software Engineering

InHand Electronics

[email protected]

www.inhand.com

Embedded Systems Conference Silicon Valley 2010 Class ESC 364/366

Dave Stewart, ESC Silicon Valley – April 2010

Troubleshooting Real-Time Issues using a Logic Analyzer © 2010 InHand Electronics, Inc. – www.inhand.com 2

Troubleshooting Real Time Software Issues using a Logic Analyzer

Introduction Print Debug Macros Logic Analyzer Debug Macros Tricks and Techniques for

Real-Time Debugging Measuring Execution Time and

Real-Time Performance Summary



What are The Tough Bugsto Debug in Real-Time Systems?

Glitches Timing and Synchronization Problems Driver Errors Misbehaving Interrupts Memory Corruption Priority Inversion Performance Issues Hardware Errors



Limitations of Traditional Debugging

Print Statements There is no console, so Print Statements don’t work Print Statements are too slow thus provide insufficient information Print function significantly affects real-time performance Writing debug output to a serial port changes the timing too much Adding print statements changes program behavior Can’t measure performance at a fine granularity Max 50 to 100 print statements per second The code crashes, but there is insufficient feedback as to where

Symbolic Debuggers (e.g. IDEs, via JTAG, ActiveSync, or other comm link) A symbolic debugger or emulator is not available Stepping through the code makes the program behave differently Breakpoints will “break” real-time performance There is real I/O, it doesn’t work Debugger doesn’t deal with interrupts properly There might be a race condition or other synchronization problem



Solution:Use a Logic Analyzer for Real-Time Issues

Most detailed view of your code Microsecond view of software Easily 50,000 debug data points per second Info is time-stamped for timing assessment

Real-Time Can use it for interrupts and I/O drivers Impact on real-time execution is negligible Identify temporal relationships among tasks Monitor interrupts and how they may affect execution “Watch” variables without changing execution time

Anomalies Spot anomalies and different patterns of execution Obtain sufficient proof that a problem is hardware, not software Fine-grain timing measurements to identify performance culprits



Logic Analyzer for DebuggingShould I use it all the time?

Using a Logic Analyzer is NOT easy Use Print Statements and Symbolic Debuggers to solve

easy and non-real-time problems first Add to your repertoire of available tools to solve hard

problemsSolve functional problems using print statements.

If necessary, run the functions on the desktop, and debug them there. Only move to embedded environment when it is working well.

Use Symbolic Debuggers mainly for Tracing through code that fails in a consistent manner Post-Mortem debugging of crashes, to view all variables



Logic Analyzer Features

Preferably at least 8 channels 16 to 24 channels is very nice to have 2 to 7 channels still useable, but it prevents use of some techniques

Large memory depth in M-Samples, not K-Samples

Multiple Views Timing Diagrams State Listings Decoding of Serial Protocols Useful features: Search, Filtering, and Triggering

High-Speed Interface to a PC Built-in PC, USB Memory Stick, Ethernet all OK

Common logic analyzer features that are NOT needed: High Speed. Don’t need GHz. Very slow analyzers, e.g. 10MHz, are OK. Many channels. Don’t need 64 or 128 channel analyzers.

Low cost USB Logic Analyzer Pods are OK A $25,000 logic analyzer is nice to have and will have many extras that could be

useful• Examples in this presentation use the Tektronix TLA700

What is discussed in this talk can be accomplished with a $400 USB pod.• Examples in this presentation use the Intronix LogicPort



Sample Logic Analyzer OutputTiming Diagram

This output format will be used for most examples in this class.

More details interpreting this diagram to follow.



Sample Logic Analyzer OutputState Listing

count trig mode16 mode data time-relative time-absolute



Target Setup: Digital OutputsWhy so many bits?

8 bits Can output full 8-bit number

9 to 15 bits: Output 8 bits + additional bits to specify mode, or module For software triggers

16 bits: Can output full 16-bit number

17 or more bits: Output 16 bits + additional bits to specify mode, or module For software triggers



Additional Signals Access to communication signals, such as UART, I2C, SSP,

SPI, I2S, etc. Access to trigger keys or keypad signals

Specify debug ports when a system is designed, to ensure they are available when it is time to debug and test the system. If a software engineer doesn’t ask for it,

don’t expect a hardware engineer to include it.

Easy to hook-up connectors for the logic analyzer probes

Target Setup: Desirable



Target Setup: Interconnections

Logic

Analyzer

(Assuming

8-bit

channels)

System

Under Test

Px.0-Px.7Ch.1

Px.8-Px.15Ch.2

RX/TX

Ch.3I2C

SPI

Keys

This is an example setup given 24 bits.

With fewer bits, connect whatever you can.



An AsidePrint Statement Debugging

Forms the basis for logic analyzer methods



An AsidePrint Statement Debugging

myfunc() {code hereprintf(“I got here\n”);more code hereprintf(“Going to call yourfunc()\n”);result = yourfunc();printf(“My result is %d\n”,result);etc

}

I got here

Going to call yourfunc()

My result is 384



Print Statement Debugging: Problems

A Lot of typing Minimal information per statement Hard to separate between debug and normal print

statements Prone to errors Cannot easily disable them Ultimately very inefficient



Debug Macros instead of Print Statements DEBUG_WHERE()

#define DEBUG_PRINT fprintf(stderr,

#define DEBUG_WHERE() \DEBUG_PRINT \“[%s:%u-%s]\n” , \__FILE__,__LINE__,__FUNCTION__)

Complete .h file for all macros given in this class are in the paper

that accompanies this presentation on the conference website or CD.

Note: Not all compilers support __FUNCTION__.



Print Statement Debugging

myfunc() {code hereDEBUG_WHERE();more code hereDEBUG_WHERE();result = yourfunc();DEBUG_INT(result);etc

}

[myfile.c:3-myfunc]

[myfile.c:5-myfunc]

[myfile.c:7-myfunc] result=384



Debug Macros instead of Print StatementsDEBUG_INT()

#define DEBUG_INT(_var) \DEBUG_PRINT \“[%s:%u-%s] %s=%d\n” , \

__FILE__,__LINE__,__FUNCTION__ \#_var,_var)

result = 384

DEBUG_INT(result)

[myfile.c:7-myfunc] result=384



Debug Macros instead of Print StatementsDEBUG_STRING()

#define DEBUG_STRING(_var) \DEBUG_PRINT \ “[%s:%u-%s] %s=%s\n” , \


char hello[] = “Hello World”;

DEBUG_STRING(hello);

[myfile.c:7-myfunc] hello=Hello World



Debug Macros instead of Print StatementsDEBUG_MESSAGE()

#define DEBUG_MESSAGE(_var) \ DEBUG_PRINT \ “[%s:%u-%s] %s\n” , \ __FILE__,__LINE__,__FUNCTION__,\_var)

DEBUG_MESSAGE(“Start of Data Dump”);

[myfile.c:7-myfunc] Start of Data Dump



Debug Macros instead of Print StatementsDEBUG_HEX8()

#define DEBUG_HEX8(_var) \DEBUG_PRINT \“[%s:%u-%s] %s=0x%02X\n” , \ __FILE__,__LINE__,__FUNCTION__ \#_var,_var)

result = 0xA4

DEBUG_HEX8(result)

[myfile.c:7-myfunc] result=0xA4




#define DEBUG_HEX16(_var) \DEBUG_PRINT \“[%s:%u-%s] %s=0x%04X\n” , \


result = 0x52A4

DEBUG_HEX16(result)

[myfile.c:7-myfunc] result=0x52A4




#define DEBUG_HEX32(_var) \DEBUG_PRINT \“[%s:%u-%s] %s=0x%08X\n” , \


result = 0x52A4419C

DEBUG_HEX32(result)

[myfile.c:7-myfunc] result=0x52A4419C



Debug Macros instead of Print StatementsOptionally add Time Stamp

#define DEBUG_HEX32(_var) \DEBUG_PRINT \“[%s:%u-%s@%06d] %s=0x%08X\n” , \ __FILE__,__LINE__,__FUNCTION__,clock() \#_var,_var)

result = 0x52A4419C

DEBUG_HEX32(result)

[myfile.c:7-myfunc@002765] result=0x52A4419C



Debug Macros instead of Print StatementsOptionally add Process ID

#define DEBUG_HEX32(_var) \DEBUG_PRINT \“[%05d#%s:%u-%s@%06d] %s=0x%08X\n” ,\ getpid(), \__FILE__,__LINE__,__FUNCTION__,clock() \#_var,_var)

result = 0x52A4419C

DEBUG_HEX32(result)

[41785#myfile.c:7-myfunc@002765] result=0x52A4419C



Debug Macros forLogic Analyzers

Similar to debugging with Print Statements, Except: (disadvantage) Much more difficult to use

• Learning curve could be days, not minutes

(but its worth it for the “hard” bugs that could otherwise take weeks to debug)

(advantage) 1000 times more information• Print Statement: 50 Lines/Data Points per second• Logic Analyzer: 50000 Data Points per second is easy

(advantage) Real-time view of the system• Data points are time-stamped with microsecond resolution



Basics of Logic Analyzer Debugging

bart = 0x0063;

lisa = 0x1018;

homer= 0xF04E;

marge= 0x00E5;

PDEBUG_HEX8(0x40);

PDEBUG_HEX16(bart);

PDEBUG_HEX8(0x41);

PDEBUG_HEX16(lisa);

PDEBUG_HEX8(0x42);

PDEBUG_HEX16(homer);

PDEBUG_HEX8(0x42);

PDEBUG_HEX16(marge);

See next pages for zoom of

this timing diagram.

This example assumes only a single

8-bit port is available.




Rectangle shows portion of image shown in next slide.




PDEBUG_HEX8(0x40); bart = 0x0063;

PDEBUG_HEX16(bart);

Timing Diagram of each bit.

We’ll see later the value of

these for creating code

patterns.




Each code represents a different variable. Correlate with source code.

PDEBUG_HEX8(0x40);

PDEBUG_HEX16(bart);

PDEBUG_HEX8(0x41);

PDEBUG_HEX16(lisa);

PDEBUG_HEX8(0x42);


PDEBUG_HEX8(0x42);





Variable data displayed in sequence.

bart lisa homer marge

PDEBUG_HEX8(0x40);

PDEBUG_HEX16(bart);

PDEBUG_HEX8(0x41);

PDEBUG_HEX16(lisa);

PDEBUG_HEX8(0x42);


PDEBUG_HEX8(0x42);





Timescale is Microseconds.

Easily add debug statements with minimal intrusion on real-time code.

In contrast, each print statement takes 2+ milliseconds.

1 usec



PDEBUG_HEX8() MacroInitialization (e.g. MSP430)

// Example, init MSP430 Port 4// Simple memory-mapped port

uint8_t *pdebug_dout8 = (uint8_t *) 0x001D;uint8_t *pdebug_dir8 = (uint8_t *) 0x001E;*pdebug_dir8 = 0xFF;// Initialize to output



PDEBUG_HEX8() Macro(E.g. MSP430)

#define PDEBUG_HEX8(_val) \*pdebug_dout8 = (uint8_t)(_val)

More details and more macros are provided in the paper

that accompanies this presentation on the conference website or CD.

bart = 0x0063;

PDEBUG_HEX8(0x40);

PDEBUG_HEX16(bart);



PDEBUG_HEX16() MacroIf only 8 bits available

#define PDEBUG_HEX16(_val) \PDEBUG_HEX8( (_val) >> 8); \PDEBUG_HEX8( (_val) )

bart = 0x0063;

PDEBUG_HEX8(0x40);

PDEBUG_HEX16(bart);



PDEBUG_VAR16() Shortcut

bart = 0x0063;

PDEBUG_HEX8(0x40);

PDEBUG_HEX16(bart);

bart = 0x0063;

PDEBUG_VAR16(0x40,bart);

#define PDEBUG_VAR16(_code,_val) \

PDEBUG_HEX8( _code); \

PDEBUG_HEX16(_val )



Test your MacrosExample to test PDEBUG_HEX8()

Always test your macros before using them.

PDEBUG_HEX8(0xFF);

PDEBUG_HEX8(0x01);

PDEBUG_HEX8(0x02);

PDEBUG_HEX8(0x04);

PDEBUG_HEX8(0x08);

PDEBUG_HEX8(0x10);

PDEBUG_HEX8(0x20);

PDEBUG_HEX8(0x40);

PDEBUG_HEX8(0x80);

PDEBUG_HEX8(0x55);

PDEBUG_HEX8(0xAA);



Test your MacrosExample to test PDEBUG_HEX8()

Allows you to test logic analyzer setup too.

If color analyzer, adjust colors to your liking too.




#define PDEBUG_HEX32(_val) \PDEBUG_HEX8( (_val) >> 24); \PDEBUG_HEX8( (_val) >> 16); \PDEBUG_HEX8( (_val) >> 8); \PDEBUG_HEX8( (_val) )




mickey = 0x0009B900;

donald = 0x0000CFB8;

PDEBUG_HEX8(0x44);

PDEBUG_HEX32(mickey);

PDEBUG_HEX8(0x45);

PDEBUG_HEX32(donald);



PDEBUG_HEX8() Macro(PXA270)

// Example, PXA270, use LCD data bits LDD6..LDD13// 32-bit architecture, uses GPIO2// Separate registers to set/clear bits

uint32_t *pdebug_set8 = (uint32_t *) 0x40E00020;uint32_t *pdebug_clr8 = (uint32_t *) 0x40E0002C;uint32_t *pdebug_dir8 = (uint32_t *) 0x40E00014;*pdebug_dir8 |= 0x000000FF; // Init to output



PDEBUG_HEX8() MacroInitialization (PXA270)

#define PDEBUG_HEX8(_hex8) \{ uint32_t h32 = (uint32_t) hex8; \

uint32_t sg2,cg2; \\

sg2= ( h32 & 0x000000FF); // set bits \

cg2= ((~h32) & 0x000000FF); // clear bits\ *_gpsr2 = sg2; \

*_gpcr2 = cg2; }



Logic Analyzer Debug Macros

There are many possibilities for defining macros

Use macros to enable a common interface across platforms

For each new platform, define the macros in a hardware-dependent manner, and place in a .h file

Simply include a different .h file for each platformAdditional macro examples

Paper that accompanies this presentation includes additional macro examples not covered here.



Tricks and Techniques for Real-Time Debugging

Tracing Drivers, Low-Level Code, and Real-time Code

Conditional Debugging Timing and Synchronization Fine-Grained Code Optimization Performance Anomalies Memory and Data Corruption Interrupts and OS Overhead Less-than-ideal Setup Setup Extras




Recall prior example using Print Statements

myfunc() {

code here

DEBUG_WHERE();

more code here

DEBUG_WHERE();

result = yourfunc();

DEBUG_INT(result);

etc

}

[myfile.c:3-myfunc]

[myfile.c:5-myfunc]

[myfile.c:7-myfunc] result = 384




#define PDEBUG_WHERE() \PDEBUG_HEX8(0x11); \PDEBUG_HEX8( (__LINE__) >> 8); \PDEBUG_HEX8( (__LINE__) )

#define PDEBUG_INT(code,_val) \PDEBUG_WHERE(); \PDEBUG_HEX16(_val);




myfunc() {

code here

PDEBUG_WHERE();

more code here

PDEBUG_WHERE();

result = yourfunc();

PDEBUG_INT(result);

etc

}

<11><00><03><11><00><05><11><00><07><01><80>

384 = 0x180

Line Numbers in Hex

<11> just a code to spot line numbers

The output is cryptic compared to using print statements. Hence,

continue using print statements if you can.

But use the logic analyzer to trace code

when print statements don’t work.

(Some analyzers will print decimal!)



Memory and Data CorruptionWatch a memory location that is changing unexpectedly

uint8_t *pdebug_memptr; // Globaluint8_t pdebug_memval; // Global

#define PDEBUG_WATCH(addr) \*pdebug_memptr = (uint8_t *)(addr); \pdebug_memval = *(pdebug_memptr)

#define PDEBUG_CHECK() \if (*pdebug_memptr != pdebug_memval) { \

PDEBUG_STOP(); }



Memory and Data CorruptionWatch a memory location that is changing unexpectedly

Scatter PDEBUG_CHECK throughout code. First, at beginning and end of function to narrow

down which function is changing it. Then between each line of code or function call to

determine which line is changing it. If the line is in another function call, repeat. This method will pinpoint the line of code causing

the memory corruption.



PDEBUG_HALT() Macro

Use to trigger the logic analyzer.If 9th bit is available, use that.Define PDEBUG_TRIG() similar to PDEBUG_HEX8().

Otherwise, setup logic analyzer to trigger after a specific sequence, like <EE><EF>

#define PDEBUG_HALT(_val) \PDEBUG_WHERE(); \PDEBUG_HEX8( 0xEE); \PDEBUG_HEX8( 0xEF); \while (1); // clearly show on analyzer the end

Set logic analyzer to trigger at 100%.This means the trigger is the end-point of data collection.Data up to the end-point is saved.



Fine-Grained Code Optimization

ALWAYS measure execution time of a code segment before and after optimization. If an optimization breaks a pattern that the compiler

recognizes, it could in fact make things take longer. Not all optimizations are obvious.

• What is faster, pointers or index arrays?• What is faster, case statements or if-then else?• What is faster, for loops or while loops?• Measure to find out.

Use PDEBUG Macros to measure code segments PDEBUG_START() before the code segment PDEBUG_STOP() after the code segment



Performance Anomalies

Quite often, performance is not as expected Why Not?

Scatter debug statements throughout code Start and stop of each function and thread is a good

starting point. Often, just keep all the prior debug statements in place, they

may provide the data needed.

Run the code, and monitor on the logic analyzer. Look for patterns of behavior. Identify patterns that use too much CPU. Verify what is happening when analyzer shows periods of

inactivity. Monitor execution time of interrupts.

Some examples on next few slides.




Green is commmunication.

We can clearly see communication stopped for a period of time.

Why? Is that normal?




See the Pattern?

Four short communication packets, one long one, then none for what seems like two or three system cycles.




What line of code executes immediately after PDEBUG_HEX8(0x9F)?

Whatever it is, it takes a LONG time.

If there is a performance issue, focus on that code first.




This is an example of code executed with PDEBUG_WHERE() plus a few other PDEBUG statements between every two

lines of code.

What is happening on Line 51 (0x33)?



Interrupts and OS Overhead

Even without source code for the OS, these can be measuredCreate a high-priority task that simply does the following:

while (1) {PDEBUG_HEX8(0x00);PDEBUG_HEX8(0x01);

}This will toggle one bit continuously.

Whenever this task gets interrupted, the toggling will stop.



Interrupts and OS OverheadDevice Driver Calls

If OS source code or device driver is available(e.g. stream driver, aka open/read/write/close): Add debug macros inside these wrappers. E.g.int read(int fd, void *buffer, int nbytes) {

PDEBUG_HEX8(0x30 | fd);dev_driver->read(xxx)PDEBUG_HEX8(0x40 | fd);

} Use codes 0x10 and 0x20 for open Use codes 0x50 and 0x60 for write etc.



Interrupts and OS Overhead

If OS source code is NOT available: Create your own functions for any OS function. E.g.:

int myread(int fd, void *buf, int nbytes) {PDEBUG_HEX8(0x30 | fd);rv = read(fd,buf,nbytes);PDEBUG_HEX8(0x40 | fd);return rv;

} Replace read with myread for any driver you want to measure.



Less-than Ideal SetupScrounging bits from anywhere

PXA270, we had following bits available, by leveraging the Camera Interface bits: GPIO Port2: 84, 85, 93, 94 GPIO Port3: 116,106,107,108 Remember PXA270 has separate set/clear registers. Code became too complex for macro, hence:#define PDEBUG_HEX8(val) \

pdebugHex8(val) Code takes much longer to execute

• 5 usec instead of 0.5 usec• Still very useable with minimal intrusion on system



PXA270 Bit Scrounging

pdebugHex8(uint8_t hex8) {uint32_t h32 = (uint32_t) hex8;

uint32_t gr2,gr3,ngr2,ngr3;

// don't write to registers yet; we want to write to registers // as quickly as possible, to minimize race conditions. gr2=(((h32 & 0x0C)<<(29-2)|((h32 & 0x03)<<(20-0)))); // D3.D2 | D1.D0 gr3=(((h32 & 0xE0)<<(10-5)|((h32 & 0x10)<<(20-4)))); // D7.D6.D5 | D4 h32 = ~h32; ngr2=(((h32 & 0x0C)<<(29-2)|((h32 & 0x03)<<(20-0)))); // D3.D2 | D1.D0 ngr3=(((h32 & 0xE0)<<(10-5)|((h32 & 0x10)<<(20-4)))); // D7.D6.D5 | D4

*_gpcr2 = ngr2; *_gpsr2 = gr2; *_gpcr3 = ngr3; *_gpsr3 = gr3;}



pdebugHex8 Initialization

static volatile uint32_t *_gpdr2 = (uint32_t *)0x40E00014;// GPIO 2 Directionstatic volatile uint32_t *_gpsr2 = (uint32_t *)0x40E00020;// GPIO 2 Setstatic volatile uint32_t *_gpcr2 = (uint32_t *)0x40E0002C;// GPIO 2 Clear

static volatile uint32_t *_gpdr3 = (uint32_t *)0x40E0010C;// GPIO 3 Directionstatic volatile uint32_t *_gpsr3 = (uint32_t *)0x40E00118;// GPIO 3 Setstatic volatile uint32_t *_gpcr3 = (uint32_t *)0x40E00124;// GPIO 3 Clear

// set directions#define _MASK2 0x60300000#define _MASK3 0x00101C00*_gpdr2 |= _MASK2;*_gpdr3 |= _MASK3;



Setup Extras ExampleMore than 16 Bits

PDEBUG_HEX16() macro becomes a single operation No need to split as two 8-bit signals.

Create two separate PDEBUG_HEX8 E.g. call second one PDEBUG_MODULE() Upon entry or exit to any function in a module, call this one. Use PDEBUG_HEX8 to trace and print variable data. Makes it easier to correlate logic analyzer output to source

code

Reserve some bits as triggers A lot easier to setup logic analyzer to trigger on an

individual bit, rather than a sequence of bytes.



Setup Extras Example Keys and Scope

Monitor key presses They often serve as trigger points Ensure that if debouncing is needed, it is properly

handled

Oscilloscope Analog measurements of power consumption Correlate power usage with specific code

segments Aids in software design for low power systems



Setup Extras Example Logic Analyzer Screen Captures Follow

Next few pages show a logic analyzer setup with: PDEBUG_HEX8(): 8-bit debug port (yellow) PDEBUG_MODE(): 3-bit mode (green) PDEBUG_TRIG(): 1-bit trigger (magenta) 2 keys on keypad (cyan) serial TX/RX (grey) instantaneous current consumption (green analog) input voltage (red).

Each page is a zoom of the prior page. The zoom area is shown by the magenta rectangle.



Setup ExtrasExample

zoom



Setup ExtrasExample



Setup Extras Example

Information shown in prior slides is simply unavailable when using other debug methods

Order of magnitude faster to pinpoint problems Problems that would take days to find now take an hour Glitches that would take months to pinpoint now found in a

day Problems found by observing anomalies often represent

bugs before even testing can catch it.1000 times more data per second collected

Data is visually presented in timing graphs Export listings to create your own scripts to search or

analyze data



Setup ExtrasMonitoring Serial Transmissions

Serial Communication Protocol Analyzers exist to monitor integrity of data.

• Built-in to some logic analyzers In many cases, you don’t need anything that complex. Rather, you just want to know how many bytes, and when. Use PDEBUG_HEX8() macros to print the bytes being

transmitted.



Serial CommunicationE.g. I2C

CLK

Data

The code that

immediately follows the

I2C transfer can then be

traced line by line.

Some Analyzers will convert clock/data to HEX values and

validate bus protocols. (Example later)




Actual Signals. Verify:

- Clock Rates

- Glitches and Noise

- Sequence

- Timing between edges of different signals

Protocol. Verify:

Number of Bytes Transmitted –

Acks and NAcks –

Start and Stop Bits –

Value of each byte –



Serial CommunicationReal-Time Gotchas

Code will “write” register, but serial transmission is usually NOT done when code continues, as operations continue in parallel.

For many hardware platforms, then the code in fact is pre-loading the NEXT byte, while previous byte is being transmitted, but then it waits on a status flag of the current byte to be completed.• This could get confusing when creating the code or

debugging the low-level read and write functions.• Use logic analyzer coupled with PDEBUG to view precisely

what is happening in the code.



Serial CommunicationMiscellaneous Notes

Use built-in interpreters provided by logic analyzer software to analyze the protocol• I2C, SPI, RS232

For more complex protocols, use a protocol analyzer, not a logic analyzer.• E.g. USB or Ethernet• Use a oscilloscope to verify signal integrity• Use a protocol analyzer to validate the protocol

Collect data in parallel with PDEBUG information, to see relationship between code execution and serial transmission



Using Logic Analyzer toMeasure Thread Execution Time

Resolution: measuring limitation of hardwareAccuracy: depends on method, x y x=measurement, y=accuracyGranularity: what can be measured: program, function, or line of codeDifficulty: effort (and time) needed to make a measurement

Techniques available to measure execution time:

© 2010 InHand Electronics, Inc. – www.inhand.com

Troubleshooting Real-Time

Software Issues

Using a Logic Analyzer

Dave Stewart

Director of Software Engineering

InHand Electronics

[email protected]

www.inhand.com

Embedded Systems Conference Silicon Valley 2010 Class ESC 364/366




Resolution: measuring limitation of hardwareAccuracy: depends on method, x y x=measurement, y=accuracyGranularity: what can be measured: program, function, or line of codeDifficulty: effort (and time) needed to make a measurement

Techniques available to measure execution time:




A logic analyzer provides the most accurate measurements of execution time

The resolution is sufficient to capture operating system overhead, fast interrupts, and even the shortest of code segments

It provides an “independent” time base, so that if hardware timers are inaccurate for any reason (such as a faulty oscillator), it will still catch the problem.

The main drawback is that it is harder to use than some other tools. • This is a typical trade-off: quality vs. effort.



For Most Embedded Systems:Execution Time = $$$

Money ($$$) Have a limited amount Bills have deadlines Can’t spend more than you have* Count it to know how much you are spending Don’t buy without knowing the price Good budgeting leads to good cash flow

CPU Execution Time Have a limited amount Real-time deadlines Can’t use more than you have* Measure it to know how much you are using Don’t execute code without knowing its CPU usage Good real-time design leads to good CPU usage

*except if you use credit



For Most Embedded Systems:Execution Time = $$$

Measuring Execution Time It’s like counting $$$

Analyzing Real-Time PerformanceIt’s like budgeting $$$



Timing and Synchronization

Model of a Good Software Task.

Read Inputs/Events

Do Processing

Write Outputs

Wait for EventFor periodic tasks, event is time-based.

For other tasks, event could be an interrupt, message arrival, semaphore

wakeup, or any other signal.



Timing and Synchronization

Give each module two unique codes. One for Begin Another for End

Add unique codes to interrupt handlers One as handler is entered Other as it exits

Read Inputs/Events

Do Processing

Write Outputs

Wait for Event

PDEBUG_HEX8(stopcode)

PDEBUG_HEX8(startcode)



MEZ_START()MEZ_STOP()

#define MEZ_START(taskid) \PDEBUG_HEX8(taskid);

#define MEZ_STOP(taskid) \PDEBUG_HEX8(0x80|(taskid));



MEZ_START()MEZ_STOP()

Read Inputs/Events

Do Processing

Write Outputs

Wait for Event

MEZ_STOP(gettid())

MEZ_START(gettid())

Assumes a threaded

environment with Task ID’s 0..127.

Modify macros to accommodate different OS task ID assignments.



Measuring Execution Time of TasksSample Logic Analyzer Event Log

Trace Data Rel Time Abs Time

0 51 0ns 0ns

1 61 3.0669ms3.0669ms

2 51 5.9341ms9.0010ms

3 61 3.0669ms12.067ms

4 52 240.69us12.308ms

5 62 5.0739ms17.382ms

6 51 1.5974ms18.980ms

7 61 3.0669ms22.047ms

8 53 6.6304ms28.677ms

9 51 436.53us29.114ms

10 61 3.0669ms32.180ms

11 52 773.17us32.954ms

12 62 5.0944ms38.048ms

13 51 896.05us38.944ms

14 61 3.3741ms42.318ms

15 51 6.6099ms48.928ms

16 61 3.0464ms51.975ms

17 63 3.1078ms55.083ms

18 52 2.8006ms57.883ms

19 51 1.0086ms58.892ms

20 61 3.0464ms61.938ms

21 62 4.3162ms66.255ms

Task Thread ID Period(msec) Priority

TaskA 01 0.010 High

TaskB 02 0.025 Medium

TaskC 03 0.040 Low

Task A

Task B

Task C



Tabulate Execution Time by TaskExample for set of 8 tasks and 1 interrupt handler

Use one of the measurementmethods to create the following table (in seconds):

Total sum of execution times = 10.183 secID Cref Tref Cavg Cmax Deadline0 0.000100 0.001000 0.000025 0.000056 01 0.001000 0.004000 0.001094 0.001096 02 0.002000 0.008000 0.002288 0.002472 03 0.002000 0.010000 0.002530 0.002922 94 0.001000 0.040000 0.000149 0.000587 05 0.003000 0.050000 0.005756 0.006311 76 0.002000 0.100000 0.005229 0.006910 487 0.004000 0.200000 0.009570 0.011306 148 0.005000 0.400000 0.017208 0.017208 1

T= Period

C = Extended Time

ref = reference (i.e. from specs

avg = measured average

max = measured worst-case



Measuring Execution Time of Tasks Sample Logic Analyzer Event Log

Trace Data Rel Time Abs Time0 51 0ns 0ns1 61 3.0669ms 3.0669ms2 51 5.9341ms 9.0010ms3 61 3.0669ms 12.067ms4 52 240.69us 12.308ms5 62 5.0739ms 17.382ms6 51 1.5974ms 18.980ms7 61 3.0669ms 22.047ms8 53 6.6304ms 28.677ms9 51 436.53us 29.114ms10 61 3.0669ms 32.180ms11 52 773.17us 32.954ms12 62 5.0944ms 38.048ms13 51 896.05us 38.944ms14 61 3.3741ms 42.318ms15 51 6.6099ms 48.928ms16 61 3.0464ms 51.975ms17 63 3.1078ms 55.083ms18 52 2.8006ms 57.883ms19 51 1.0086ms 58.892ms20 61 3.0464ms 61.938ms21 62 4.3162ms 66.255ms


TaskA 01 0.010 High


TaskC 03 0.040 Low






TaskA 01 0.010 High


TaskC 03 0.040 Low

Task A

Task B

Task C






TaskA 01 0.010 High


TaskC 03 0.040 Low

Task B

execution time for this iteration of

Task B is 5.0739 msec






TaskA 01 0.010 High


TaskC 03 0.040 Low

Task C

execution time for this iteration of Task C is

(55.081-28.677)-(3.0669+5.0944+3.3741+3.0464)

=11.8222 msec






TaskA 01 0.010 High


TaskC 03 0.040 Low

The conversion from event log to table should be automated,

using either a custom program, spreadsheet, or a software

analyzer tool that has the necessary features to give you

execution time for each cycle of every task.



Tabulate Execution Time by TaskExample for set of 8 tasks and 1 interrupt handler

Create the following table (in seconds):

Total sum of execution times = 10.183 secID Cref Tref Cavg Cmax Deadline0 0.000100 0.001000 0.000025 0.000056 01 0.001000 0.004000 0.001094 0.001096 02 0.002000 0.008000 0.002288 0.002472 03 0.002000 0.010000 0.002530 0.002922 94 0.001000 0.040000 0.000149 0.000587 05 0.003000 0.050000 0.005756 0.006311 76 0.002000 0.100000 0.005229 0.006910 487 0.004000 0.200000 0.009570 0.011306 148 0.005000 0.400000 0.017208 0.017208 1

T= Period

C = Extended Time

ref = reference (i.e. from specs

avg = measured average

max = measured worst-case

Task ID 0 is actually an interrupt handler

Estimated

Worst-Case

Execution Time

Desired

Period (or rate)

period = 1/rate

Measured

Average

Execution

Time

Measured

Worst-Case

Execution

Time

Number of

RTOS-Detected

Missed

Deadlines

(requires an

instrumentation

point with the

timing error handler.

Expanded Model of a Task

Wait for Event



Analyzing Effect of Operating System OverheadOverhead is Already Part of the Measurements

Using the logic analyzer technique, overhead becomes part of the measurements, but it is not uniformly distributed across tasks.

This is important to consider when precise measurements are needed.



Analyzing Effect of Operating System OverheadDistribution of Overhead in Measurements



Measuring Operating System Overhead Δthr

Low priority task continuarlly sends pulses to channel 1

High priority task provides a single pulse to channel 2

This results in the following diagram:

High Priority task execution time =

Operating system overhead to switch tasks

Note that overhead is not necessarily constant. The overhead is often a function of the number of tasks in the system. For most analysis, however, assuming that the overhead is

constant is sufficient.



Validating Periods for Periodic Tasks

Are tasks in the application executing at the rates requested?Quite often, they are not, and could be the cause of errors!For Example

Resolution of system clock400 Hz task cannot be implemented if system clock is 1 msec since it requires a tick of 2.5 msec

Resolution of hardware timer affects accuracy of system clockA 1 msec clock might be approximated to 998 usec

This affects time by over 3 minutes per day! Skew due to implementation error

Occurs when using sleep(X) — block task for X msecInstead, must use pause(X) — block task until time=X

Real-Time Scheduling Problems cause actual periods to varyOverloaded processor

Priority InversionMissed DeadlinesInterrupts executing for too long

Note: Many software analyzers are unable to detect these problems.



Validating Periods for Periodic TasksRecall the MEZ_START() and MEZ_STOP Macros



Validating Periods for Periodic TasksFind “Non-Delayed” Starting Points in Event Log

“Non-delayed” starting points starting points occur when MEZ_START() is detected to occur either: after MEZ_START() of a lower-priority task’s cycle, but before the lower-

priority task’s MEZ_STOP() when the idle task is executing

Example:



Validating Periods for Periodic Tasks Compute Time Differences

Compute the time difference between each two non-delayed starting points Divide by (number_of_delayed_starting_points+1) between those starting point. This number should be exactly the period of the task; if it isn’t, there could be timing

errors. Example of what should be expected

period = (90 - 10)(1 + 1) = 80/2 = 40 msec

Repeat for every pair of non-delayed starting points.



Validating Periods for Periodic TasksExample of a Problem

Suppose the following timing diagram was obtained:

period = (88 – 10)(1+1) = 78/2 = 39 msec

In this case, 40 msec period is requested, but because the system clock is set to 3 msec, exactly 40 msec cannot be achieved. The RTOS instead rounded to a 39 msec period, without any notification of error to the designer.

Not 40 msec as requested!



Checking for Missed Deadlines

Missed Deadline: Periodic task is not finished executing an iteration of code,

when it is time to start it’s next iteration.To check for missed deadlines, do the following for each task:

Find non-delayed starting points for each task in the event log(use same method as for validating periods)

For the delayed starting points, compute the requested starting point by using the non-delayed starting points and adding to it the measured period

Compute the deadline time relative to each starting point The MEZ_STOP() event that indicates end of a task should

execute before this computed deadline time every cycle.This procedure can be automated by using data downloaded from

the logic analyzer as input.



Analyzing Real-Time Performance and Potential for Overload

Use Real-Time Scheduling Analysis Real-Time Scheduling Analysis is very theoretical

and mathematically oriented. Here, a practical approximation is presented,

using real measured data, that can lead to quickly converging on resolving real-time loading issues in a system.

We’ll start with the theory, then simplify it to make it more practical for use in system design and debugging.



Practical Real-Time Scheduling AnalysisFixed-Priority Scheduling Theory

A task set consisting of n periodic threads is schedulable using the Rate Monotonic Algorithm if the following equation holds:

Example, lets apply equation to the data collected and shown in the table:

Total sum of execution times = 10.183ID Cref Tref Cavg Cmax Deadline0 0.000100 0.001000 0.000025 0.000056 01 0.001000 0.004000 0.001094 0.001096 02 0.002000 0.008000 0.002288 0.002472 03 0.002000 0.010000 0.002530 0.002922 94 0.001000 0.040000 0.000149 0.000587 05 0.003000 0.050000 0.005756 0.006311 76 0.002000 0.100000 0.005229 0.006910 487 0.004000 0.200000 0.009570 0.011306 148 0.005000 0.400000 0.017208 0.017208 1



Fixed-Priority Scheduling TheorySimplifying the Equations

First, some approximations with negligible effect except whenduring the most critical fine-tuning. Assume ID 0 is highest priority task instead of an interrupt Instead of 0 < t <= D¡, use longest length of trace that is a multiple of the

slowest task

This reduces the equation:

i

This simply leads to conservative estimates, since it results in computing

more overhead than there really is

This means try equation first with i=1, then i=2, then i=3,

until LHS >1, or i = = n

thr

to:



Fixed-Priority Scheduling AnalysisApplying the Equations using Real Data

With 1 Task:


LHS =

i = 1

Plug into equation

This means task 0 is

always schedulable



Fixed-Priority Scheduling AnalysisWith 2 Tasks

With 2 Tasks:


LHS =

i = 2

Plug into equation

This means tasks 0 & 1 are

always schedulable0.156

0.001096 0.0001+

10---------------------------------------------+ 0.455=



Applying the Equations using Real DataWith 3 Tasks

With 3 Tasks:


Plug into equation

i = 3

LHS = 0.1560 + 0.299 + 0.322 = 0.777This means task 0 & 1 & 2 are always schedulable



Applying the Equations using Real DataWith 4 Tasks

With 4 Tasks:


Plug into equation

i = 4

LHS = 0.1560 + 0.299 + 0.322 + 0.301 = 1.079

This means task 0 & 1 & 2 & 4 are NOT always schedulable



Applying the Equations using Real DataSo What?

What does the analysis mean?

It says a lot: Unless changes are made

only Tasks 0, 1, and 2 are guaranteed to not miss deadlines In order for Task 3 to meet deadlines, either

• Decrease collective execution time (Ci) of Tasks 0, 1, 2, and 3

• Increase period (Ti) of Tasks 0, 1, 2, and/or 3 None of Tasks 4 through 8 are guaranteed to meet deadlines

as long as Task 3 is not guaranteed.

Question: To make Task 3 schedulable, by how much do we decrease Ci or increase Ti?



Fixed Priority Scheduling AnalysisMaking an Unschedulable Task Set Schedulable

Suppose goal is to fix the system so that the interrupt handler (PID 0) and first three tasks (PID 1 through 3) are guaranteed to meet their deadlines.

First, revise Cref using the values of Cmax. This refines the estimated worst-case execution times to ensure they are more accurate.

To make tasks schedulable, must reduce left side of equation to below 1.0 (it was 1.079 in example). One of the following must be done: Decrease CPU usage by optimizing code

I.e. reduce Cref for one or more tasks Increase period if application and hardware allow it

I.e. increase Tref for one or more tasksRepeat the math only, without actually performing the above optimization

or period changes. Only when a valid answer is obtained should the actual code be

modified. Increasing period Tref is a lot easier than reducing execution time Cref . To reduce Cref, it is necessary to optimize or delete some code.



Real-Time SchedulingDiscussion: Getting 110% Effort from the CPU

Real-Time Scheduling is often idealized. In reality, the following situations are quite common, and appear to negate the use of proper real-time analysis: A task’s worst-case execution time is much greater than its typical case. The system is mostly event-driven. This is not a hard real-time system: deadlines are soft. A task’s period changes depending on what it is doing. Tasks waste time busy-waiting. It’s a fast CPU, so it doesn’t matter how long the code takes. Can’t use more than 80% utilization to ensure a buffer zone for timing problems “I need 110% utilization!”

• Equivalent to “I need 110% $$$$” => Use Credit!

No matter what the problem, however, knowing the execution time of each task, and using it as the basis for analyzing the system, is essential to ensure a predictable real-time system that meets application requirements.

Regardless of the application, systematically engineering the timing of the system will lead to a system with minimal timing errors and efficient budgeting of the CPU while lowering the amount of time needed to implement and debug the code.



Summary

Logic Analyzer provides a Window into Embedded Code A window with micro-second resolution

Use it in addition to other methods It is complimentary to using print statements and emulators

Defining the right macros requires some creativity Many possibilities. This class only scratches the surface.

Look for Anomalies and anything that just doesn’t look right These are usually the clues to where the problems lie

Use it to troubleshoot bugs or measure performance Wide variety of tricks and techniques using the logic

analyzerGood Luck!

This is a tool to tackle the toughest embedded system bugs. Hopefully weeks will become days, and days will become

hours

© 2010 inhand electronics, inc. – troubleshooting real-time software issues using a logic...

Documents

realtime software issues

realtime execution

realtime systems

esc silicon valley

timing assessment realtime

real time software issues

easy use print statements

adding print statements