llrf embedded development · 2018. 9. 28. · –daq analysis and storage –high-speed means what...
TRANSCRIPT
LLRF Embedded Development
Brian Chase, Joshua Einstein-Curtis
Controls Modernization Group
28 September 2018
Introduction
• LLRF is a high-speed, high-precision real time system.
• For the FNAL accelerator infrastructure, have relatively high
data rates
– Similar to beam instrumentation
– Necessitated builds of a Labview client for commissioning,
debug, and diagnostics (Backdoor/custom)
9/27/2018LLRF Embedded Development2
Example:CMTS has 4 ADC and 2 DAC channels per cavity. In addition to the raw data, there are also the baseband IQ and amplitude/phase signals, error, and controller intermediate stages
2 * (4*2 + 2 + 10) * 65 MSPS * 32-bit = 82 Gbps max82/65 = 1.26 Gbps if 1 MSPS sampling
1.26 Gbps / 8 * 360 = 56 TB/hour
0.02 * 56 = 1.12 TB/hourif only 20 ms of 1MSPS data every second
Real-time Needs
• Hard Realtime LLRF (timescale-dependent, of course)
– Failures not allowed
• Bucket-resolution timestamping on data
• Necessary to do extensive hardware testing or simulation
(Qemu, ARM simulator)
– Good debugging tools a necessity
• Utilization and overloading
– Need for optimizations and good compilers
9/27/2018LLRF Embedded Development3
vxWorks
• MVME2400, MVME5500 with Kinetic Systems backplane
• VxWorks 5.5, 6.1, 6.4, 6.9
– 5.5 running on MI, FAST, 6.4 on Recycler, 6.9 dev system provided to controls
• Power PC 603e, 750 or 7457
• Standardized communication between front ends enables faster
development. Shared libraries, etc.
– https://www-bd.fnal.gov/controls/micro_p/mooc_project/welcome.html
• Protocol Compiler
– Custom for each device? Possibly leads to complexity and versioning
issues
– Each device has own receive stack, and each group develops their
own
9/27/2018LLRF Embedded Development4
Current Model
9/27/2018LLRF Embedded Development5
Spreadsheet LLRF Frontendsys_init.cpp
PythonScripts-to-dabbel
Modification Scripts
Dennis’ Frontends
Acnet and Front-end independently record and manage variables
SoC MFC
9/27/2018LLRF Embedded Development6
LLRF Control and Timing
• VUCD
– Developed in mid-nineties
– No replacement for current systems available
– Timing and data decoding using backplane triggers
– Multi-timer system
– No planned replacement for current/deployed systems
• Bucket resolution and accurate timestamping needed for
future diagnostics
9/27/2018LLRF Embedded Development7
Build Environment
• nova build environment (esd tools) well developed
– LLRF projects are all there, including configuration files for new
SoC systems (spreadsheet register maps and defaults)
• Need for a well-developed upgrade plan across departments
• Need for supporting both legacy systems and new systems
using same tools?
• Significant increases in performance with each controller, but
processor-specific programming can make porting difficult
9/27/2018LLRF Embedded Development8
Revision Control
• Currently part of build toolchains on nova
• One of the largest issues we’ve run in to is revision and
versioning
– Separate versions for firmware, software, libraries
– Storing git commits in f/w and software works well
– Xilinx allows for same bitfile from same files
• routing/placement hash/seed is based on files included in project
• Enforcing this across all project levels is a challenge
• PLC versioning? Software versioning? HDL versioning?
9/27/2018LLRF Embedded Development9
Revision Control
• LBL/LCLS-II and CERN have self-describing
firmware/software
– Register maps in control system and front-ends match
• gzip’d JSON in memory on devices
• CERN’s FESA and cheby
• Acnet database
• TFTP/remote image loading is crucial
– Limit wear on flash (limited write) devices
• Remote/central logging is crucial
– See above
• Verification and Continuous Integration are valuable tools
9/27/2018LLRF Embedded Development10
What is plan for future?
• Responsibility of each department?
• High-speed data and diagnostics
– artDAQ? FTP? Something new?
– DAQ analysis and storage
– High-speed means what for each system?
• Data lifetime?
• Scope capabilities
• Event identification
• Design lifetime
– 20 years? 5 years? 1 year?
• Tool availability and automation
– Build system for non-experts
– Nova replacement? Command-line functionality?
9/27/2018LLRF Embedded Development11
Looking Forward
• What will be available in 2 years? 5 years? 10 years?
• Need a long-term plan
• Due to few resources at developer level, really need process
and procedure directed from managment
• Hardware - should there be a standard developed across
groups?
– Shared FPGA/Processor boards?
– Crate standards?
– Physical interfaces?
9/27/2018LLRF Embedded Development12
• Designed for cell base
stations, radar, military
• Single-chip includes all
necessary for an RF system,
but might not meet
performance requirements of
modern accelerators.
• Separate application and
real-time processors
• Shared memory makes for
very powerful, high-
bandwidth toolset
Xilinx Zynq Ultrascale+ RFSoC
9/27/201813 LLRF Embedded Development
https://www.xilinx.com/products/silicon-devices/soc/rfsoc.html
Conclusion
• Concerns for future
– DAQ and timing/precision
– Data retention and high-speed data management
– Revision Control and Build Environment are significant
concerns
– Ease of integrating in to control system: shared protocols,
development architectures, etc
• Need a multi-year, division-wide path forward
– While still supporting legacy systems
• Thanks!
9/27/2018LLRF Embedded Development14
Extra Slides
9/27/2018LLRF Embedded Development15
Switching to Fiber
• Fiber benefits outweigh slightly increased cost
• Decreased power
• Higher bandwidth
• Low cost per bandwidth
• Relative ease of implementation given available tools
9/27/2018LLRF Embedded Development16
Communication Proposals
• 9.027 MHz RF synchronous frame rate
– Local clock source needed on each device
• 1300 MHz harmonic as data rate
• Need known frame size (ie 128-bit)
• Previous State, Events, Beam pattern for next group
• Single-source star
• Send as timing/events -- Return line as MPS
– Data concentrators?
• Throw out proposal
• 3 links – DAQ, timing/MPS, control
9/27/2018LLRF Embedded Development17
9/27/2018LLRF Embedded Development18
Intel Cyclone V and Arria 10
9/27/2018LLRF Embedded Development19
SoC MFC Test DAQ System
9/27/2018LLRF Embedded Development20
Courtesy Ed Cullerton
Arm Product Line
9/27/2018LLRF Embedded Development21
https://www.silabs.com/documents/public/white-papers/Which-ARM-Cortex-Core-Is-Right-for-Your-
Application.pdf
Microsemi SmartFusion2
9/27/2018LLRF Embedded Development22
Single Event Upset Immune design
LPC17xx
Kinetis K60
NXP
9/27/201823 LLRF Embedded Development
i.MX
Why not Pis?
• The question is really: what does real-time mean?
– Bucket-level resolution and repeatability?
– TCLK decoding accuracy? Event decoding accuracy?
• What runs at each layer?
– Firmware/embedded RTOS
– Software/application layer
• Lifetime and long-term support
– Open Hardware
• Interconnect standards
– SPI + I2C support? (1 SPI, 1 I2C, 1 UART)
– Standard interconnects?
– Bus/backplane protocols?
9/27/2018LLRF Embedded Development24
DSP
• LLRF requires higher-performance DSP and DAQ
• Newer FPGAs are offering dedicated floating-point blocks
– Require use of special tools or overloading standard operators
• Integrated simulation and verification environment being
more important
• Collaboration requires the ability to transfer IP easily
9/27/2018LLRF Embedded Development25
SoM Benefits
9/27/2018LLRF Embedded Development26
https://www.intel.com/content/www/us/en/programmable/products/soc/ecosystem/system-on-modules.html
Linux (and variants)
• Linux w/ preemptRT
• RTAI (Linux as task)
• NI Real-Time Linux (dual-schedulers)
9/27/2018LLRF Embedded Development27
RTOS
• FreeRTOS -- https://www.freertos.org/
• Contiki (IoT) -- http://www.contiki-os.org/
• MicroC/OS-IIx
• Mbed (IoT)
• RTEMS
• vxWorks
– Now a Linux variant
• MQX
9/27/2018LLRF Embedded Development28
MPC603e
9/27/2018LLRF Embedded Development29
MPC750
9/27/2018LLRF Embedded Development30
MPC7457
9/27/2018LLRF Embedded Development31
Processing Models
• IOC+Epics
– “Dumb” front-end with managing IOC
– Subscriber model
• Crate processors
– Slot 0 manages all cards in crate. Advanced data processing
happens ???
• NADs
– Multiple models available
• Acnet
• Requirements on who does each portion is not defined
– Controls communication (MOOC(AN), libacnet, erlang, PC)
– Debug communication (Labview, Python)
– Front-end intelligence
9/27/2018LLRF Embedded Development32