modular data acquisition introduction and applicability to lcls daq
DESCRIPTION
Modular Data Acquisition Introduction and applicability to LCLS DAQ Michael Huffer , [email protected] Stanford Linear Accelerator Center December 14, 2006 Representing : Ryan Herbst Chris O’Grady Amedeo Perazzo Leonid Sapozhnikov Eric Siskind Dave Tarkington Matt Weaver. - PowerPoint PPT PresentationTRANSCRIPT
Department of Particle & Particle AstrophysicsDepartment of Particle & Particle Astrophysics
Modular Data Acquisition Modular Data Acquisition
Introduction and applicability to LCLS DAQ
Michael Huffer, [email protected] Linear Accelerator Center
December 14, 2006Representing:Ryan Herbst
Chris O’GradyAmedeo Perazzo
Leonid SapozhnikovEric Siskind
Dave TarkingtonMatt Weaver
Department of Particle & Particle Astrophysics
2
Department of Particle & Particle Astrophysics
OutlineOutline
• Introduction– Concepts– Architecture– Implementation
• Examples…– Petabyte scale, low access latency storage for SLAC Computer Center– LSST camera data acquisition system
• Application design• Discuss applicability for LCLS Data Acquisition?
Department of Particle & Particle Astrophysics
3
Department of Particle & Particle Astrophysics
The ModuleThe Module
• Is the basic building block of the architecture• Specified as:
– A hardware design (schematics, BOM & layout guidelines)– A series of base services implemented as:
• VHDL (interfaced through core IP libraries)• Software (OO interface - provided through header files and shared libraries)
– documentation • Module neither specifies or constrains application’s physical partitioning model• Architecture specifies three different types of modules
– CEM (Cluster Element Module)• Provides a processor + RTOS (the Cluster Element)• Provides many channels of generic, high speed, serial I/O• Provides commodity network interface (10 GE & 100-Base-T Ethernet)
– fCIM (Fast Cluster Interconnect Module)• Provides 10 GE connectivity for up to 64 Cluster Elements
– sCIM (Slow Cluster Interconnect Module)• Provides 100 Base-T & 1 GE connectivity for up to 64 Cluster Elements
Department of Particle & Particle Astrophysics
4
Department of Particle & Particle Astrophysics
Cluster Element Module (CEM)Cluster Element Module (CEM)
• Two variants…• footprint:
– ~ 50 cm2
• power:– ~ 7 watts total + – ~ 3/4 Watt/port
JTAGreset
optionsreset
PHYs (4-20)
10 GE 100B-T
CE
To fCIM
To sCIM
A Cluster Element (CE) is a
processor
Each lane operates
up to 10 Gb/sec
CEM (1 channel)
resetoptions
JTAG reset 100B-T10 GE 10 GE 100B-T
PHYs (0-16)
Common to both elements
CECE
May mix and match lanesto each CE
CEM (2 channel)
• Two variants…– One channel CE– Two channel C2
Department of Particle & Particle Astrophysics
5
Department of Particle & Particle Astrophysics
FastFast Cluster Interconnect Module ( Cluster Interconnect Module (ffCIM)CIM)
• footprint:– ~ 144 cm2
• Power:– ~ 1 ½ Watt/port– 64 elements ~ 110 watts
1 GE10 GE (0 – 8)
fCIM
Supports a variety of electromechanical
standardsX2 & XENPACK MSA
CX4Long haul & short haul
fibers
To CEIs a collection
of managed switches
To management
network
10 GE (0 – 8)
Department of Particle & Particle Astrophysics
6
Department of Particle & Particle Astrophysics
SlowSlow Cluster Interconnect Module ( Cluster Interconnect Module (ssCIM)CIM)
100B-T (2 – 64)
sCIM
1 GE
Supports a variety of electromechanical
standards
To management
or control network
To CE
Is a collection of unmanaged
switches
• footprint:– TBD (less then fCIM)
• Power:– TBD (much less then fCIM)
Department of Particle & Particle Astrophysics
7
Department of Particle & Particle Astrophysics
32 32 ElementElement Cluster Cluster
To management
or control network
sCIM
To data network
1 GE
CE CE CE CE CE CE CE CE CE CE CE CE CE CE CE CE CE CE CE CE CE CE CE CE CE CE CE CE CE CE CE CE
fCIM
10 GE
fCIM is managed
control network
Department of Particle & Particle Astrophysics
8
Department of Particle & Particle Astrophysics
CEM block diagramCEM block diagram
Left side MFD
FPGA (SOC)200 DSPs
Lots of gatesXilinx XC4VFX60
Fabric clock
Right sideMGT clock
Right sidePPC-405
(450 MHZ)
Right sideConfiguration
memory128 Mbytes)
Samsung K9F5608
Right sideMemory
(512 Mbytes)Micron RLDRAM II
Right sideMulti-Gigabit Transceivers
(MGT)8 lanes
Left side 100-baseT
Reset
Reset optionsJTAG
Department of Particle & Particle Astrophysics
9
Department of Particle & Particle Astrophysics
Base services provided by CEMBase services provided by CEM
• “Fat” Memory Subsystem– 512 Mbytes of RAM– Sustains 8 Gbytes/sec– “Plug-In” DMA interface (PIC)
• Designed as a set of IP cores • Designed to work in conjunction with MGT and protocol cores
• Bootstrap loader (with up to 16 boot options and images)• Interface to configuration memory• Open Source R/T kernel (RTEMS)• 10 GE Ethernet interface• 100 base-T Ethernet interface• Full network stack • Utility software to manage I/O
Department of Particle & Particle Astrophysics
10
Department of Particle & Particle Astrophysics
Extended services provided by CEMExtended services provided by CEM• Pretty Good Protocol (PGP)
– Physical interface is serial with 2 LVDS pairs/lane)– Point-to-Point connectivity– Allows clock recovery– Full duplex
• Symmetric capabilities in either direction from either end– Provides reliable frame (packet) transmission and reception– Deterministic (and small) latency
• Lightweight “on the wire” overhead• Specifies 4 VCs in order to provide QOS
– Implemented as an IP core• Small footprint• Interface hides user from protocol details and implementation• Implemented on CE (through the conical model described above)
– Asynchronous• Extensible in both bit-rate and # of lanes
• Flash Memory Module (FSM)– Provides as much as 256 Bytes/CE of persistent storage– Low latency/high bandwidth access(1 Gbyte/sec)– Interfaced using PGP
Department of Particle & Particle Astrophysics
11
Department of Particle & Particle Astrophysics
Cluster Element Cluster Element as used inas used in petacache petacache
FSM FSM FSM FSM
CE10 GE
To/From fCIM
100B-T
To/From sCIM From management
network
PGP 1 lane @
250 Mbytes/se
c
To client nodes on
client network
Called a SAM(Storage Access Module)
65 Gbyte flash memory(Flash Storage Module)
PGP core & interface
Application specific
Department of Particle & Particle Astrophysics
12
Department of Particle & Particle Astrophysics
Cluster Element Cluster Element as used inas used in LSST DAQ LSST DAQ
CE10 GE
To/From fCIM
100B-T
To/From sCIM
From Camera Control System on CCS network
To client nodes on DAQ
network
Called a RNA(Raft Network Adapter)
Services 9 CCD mosaic288 Mbytes/sec
Application specific
Raft Readout System
PGP (fiber-300 M) 1 lane @ 300 Mbytes/sec
PGP core & interface
In cryostat
(Replicated 25 times)
Department of Particle & Particle Astrophysics
13
Department of Particle & Particle Astrophysics
The “Chassis”The “Chassis”
AcceptsDC power
PassiveBackplan
e
8 U
X2(XENPACK MSA)
1UFan-Tray
1UAir-Outlet
1UAir-Inlet
High-Speed Network Card (8U)
Daughter board Card (4U)
Department of Particle & Particle Astrophysics
14
Department of Particle & Particle Astrophysics
Chassis Physical Interfaces (19”)Chassis Physical Interfaces (19”)
CCS network card1 GE
client network card10 GE to odd raft
to even raft
to CCS
to DAQ network
Science Array (12)Guider Array (2)
WFSArray (2)
8U
Number is
TBD
bank
Daughter cards replicated twice for:Redundancy &
simulation
Department of Particle & Particle Astrophysics
15
Department of Particle & Particle Astrophysics
• Partition problem into three domains:– Device/sensor specific Read-Out (RO)– Device/sensor monitoring and configuration– Data transport and processing
• Define a consistent and regular interface between RO & CE systems– independent of device/sensor
• Define CE customization– How many lanes of I/O necessary between RO and CE?– What are the protocols on these lanes?– Specify data processing
• How should this processing be partitioned between software and hardware?• CE number
– What is the underlying, inherent, parallelism of the data (if any)?– How many CPU cycles and gates should be dedicated per data byte?
• processing effort/byte• Define physical partitioning of design
– How many boards?– What type and number of modules on a board?– Incorporate with custom logic?
A prescription for application designA prescription for application design
The later two are within the realm of the CE
Department of Particle & Particle Astrophysics
16
Department of Particle & Particle Astrophysics
Typical usage patterns Typical usage patterns
CE
RO RO RO RO
RO
CE CE CE CE
RO RO RO
CE CE CE CE
RO
• Many different types of devices• Physically separated• Processing/byte/device is high
• Homogeneous devices• Perhaps physically separated• Processing/byte is high
• Many different types of devices• Physically separated• Processing/byte/device is low