Wir schaffen Wissen – heute für morgen

19. April 2023PSI, 19. April 2023PSI,

Paul Scherrer Institut

Embedded PREEMPT_RT Linux for IOxOS VME IOC

Damir Anicic

Hardware - IOxOS (IFC1210) VME IOC

Presented by D. Zimoch: A new intelligent FMC/XMC/PMC carrier board / IOC for VME

Software

We will use RealTime Linux

• PREEMT_RT patched

• XENOMAI (if PREEMPT_RT is not good enough)

Choose Development Kit

• Started with: Linux Target Image Builder (LTIB)

• Initially sponsored by Freescale Semiconductor (www.ltib.org)• Now moved to Savannah project (savannah.nongnu.org)• We have decided to switch to more modern development kit

• Now using: Embedded Linux Development Kit (ELDK)

• Provided by Denx (www.denx.de)• Based on Yocto (www.yoctoproject.org)• Aligned with OpenEmbedded (www.openembedded.org)

Software Development Kit (SDK) and Root File System

• Download pre-built version• OK to start with

• Better build it yourself• Download ELDK source (git, we use eldk5.2)• Configure & build it

• Initially all SW package sources will be downloaded• Later full rebuilds are faster (~ hours)• Better do partial rebuilds (rootFS only) (~ 1/2 hour)

Now you have SDK and Root File System

Components required to boot Linux

• U-Boot, Universal Boot Loader (www.denx.de)

• Flat Device Tree (devicetree.org)• Device Tree file from IOxOS

• Kernel (rt.wiki.kernel.org), we use 3.2.17-rt28

• Root File System (previous page)

• VME and PCI-bridge drivers (PEV & IDT from IOxOS)

Build them using SDK (Install to TFTP/NFS server)

Booting Linux on Embedded System

• Load U-boot from internal storage (NOR/SPI/SDCARD)• Configure network with DHCP• Load FPGA firmware files (*.bit) using TFTP • Load Kernel and Device Tree using TFTP

• Kernel• Mount rootFS over NFS

• SysV Init• Mount other resources over NFS• Load IOxOS VME and PCI-bridge drivers (PEV & IDT)• Start EPICS IOC

RT Interrupt Response Times Histogram

EPICS not running EPICS running

User times (green) of PREEMPT_RT Linux application are acceptable

Kernel times (blue) is what could also be achieved with XENOMAI (should be even better)

Interrupt Response Times Histogram (DMA in RT task)

It is expected that typical RT application reacts on 10 ms interrupts:

• DMA read 250 KB from USR1 area

• Calculate (void in this test)

• DMA write 8 KB to USR1 area

• notify EPICS

Although it is fine with standalone RT application, we have interrupt jitter problems (2000 µs) when same runs in EPICS. We still have to investigate it.

Achieved 250+ MB/s. It is fast enough for our needs.

Conclusions

PREEMPT_RT Linux is good enough for user applications

• EPICS works• Interrupt response times are acceptable• DMA transfer rate is acceptable

The CPU load can be quite high when heavily loaded EPICSand RT application are running on the same IOC

• Solution: run them on two IOCs

We do not need XENOMAI

Acknowledgements

IFC Board Design• Timo Korhonen• Ernst Johansen• IOxOS TechnologiesLLRF Project lead• Roger KaltTransition Module Design• Stefan Scherrer• Werner SturzeneggerFPGA programming• Lionel Schebacher

Linux Kernel Driver• IOxOS TechnologiesLinux / EPICS programming• Babak Kalantari• Damir Anicic• Dirk Zimoch• Mario Jurcevic

Special thanks to• Roger Kalt• Mario Jurcevicfor measurement results.