Pulsar IIHardwareOverview

Jamieson Olsen, Fermilab14 April 2014

Pulsar II Hardware Overview / J.Olsen 2

Outline

• ATCA Overview• Pulsar II Hardware– Front Board– Rear Transition Module– Mezzanine

• Data Transfer Options• Pulsar IIa Prototype Results• Conclusion

14 April 2014

Pulsar II Hardware Overview / J.Olsen 3

ATCA Overview• Mature telecom standard

– IPMI Protocol– Hot swap, redundant modules– Fully instrumented

• High power: 300-450W per slot– Front to back airflow– Option for vertical airflow

• Boards measure 8U x 280mm– User defined Zone-3 for RTM I/O

• 2 to 14 slots per shelf• Full mesh backplane option

– 40 Gbps per channel (>7 Tbps)– 100 Gbps per channel coming soon

14 April 2014

ASIS ATCA shelf at Fermilab. This is a 40G full mesh backplane rated for 450W per slot.

Pulsar II Hardware Overview / J.Olsen 4

Pulsar IIb Front Board• Xilinx Virtex 7 FPGA

– XC7VX415T – XC7VX690T• Up to 80 GTH transceivers

– 40 for RTM– 28 for Fabric– 12 for Mezzanines

• Four FMC Mezzanines– 35W each– LVDS + SERDES up to 60 Gbps

• Compatible with LAPP IPMC module– 10/100/1000 Base Interface

Port• Backplane clock distribution

14 April 2014

Pulsar II Hardware Overview / J.Olsen 5

Block Diagram

14 April 2014

RTM Optical Transceivers

Full Mesh Fabric Interface

Dual 48VDCand platformManagement (IPMI)

Pulsar II Hardware Overview / J.Olsen 6

Programming the FPGA

• JTAG bus driven by the IPMC• Mux to automatically select the Xilinx

programmer cable• FPGA also has SPI flash attached– “Indirect” Flash programming

• Time to program?– XC7V690T = ~100 Mbits

14 April 2014

Pulsar II Hardware Overview / J.Olsen 7

IPMC Module (LAPP)

• Dual ARM microcontrollers– IPMI, I2C sensors, AMC/RTM, LEDs, etc.– General User I/O, JTAG, 100BASE-T Ethernet

14 April 2014

Pulsar II Hardware Overview / J.Olsen 8

Pulsar IIb Power Distribution

Yellow = LDO Linears

Orange = Switchers, GE/Lineage DLYNX with PMBus Interface

14 April 2014

48VDC

Pulsar II Hardware Overview / J.Olsen 9

Rear Transition Module

• Fiber optic transceivers– 8 QSFP+– 6 SFP+

• Up to 380 Gbps• Intelligent FRU

– µC monitors optical modules

• Hot Swap• PICMG 3.8

“Zone-3A” compliant

14 April 2014

Pulsar II Hardware Overview / J.Olsen 10

Test Mezzanine Card

• Kintex XC7K160T FPGA• 4 SFP+ Transceivers

– 6.6 Gbps GTX• 128MB DDR3• Socket for testing ASICs• FPGA Mezzanine Card

(FMC) connector is compatible with FPGA development boards

• 74 mm x 149 mm

14 April 2014

Pulsar II Hardware Overview / J.Olsen 11

System Integration• All ATCA platforms support a dual star Base

Interface network– 10/100/1000BASE-T Ethernet– Used for slow controls, downloading firmware,

etc.• COTS “switch blades” support separate

networks on the Base and Fabric Interfaces– Pulsar IIb supports up to 40G Ethernet Fabric

connections to the switch– We are testing the IPBus firmware now– At Fermilab we are using the Emerson F125

switch (10G Ethernet)

14 April 2014

Pulsar II Hardware Overview / J.Olsen 12

Data Sharing• The full mesh backplane is a

natural fit for triggers where hits are shared across tower boundaries– Particularly true if data sharing is

irregular• Effectively blurs the distinction

between FPGAs• First Application: FTK Data

Formatter– 16 φ x 4 η towers– 32 boards in 4 crates

14 April 2014

http://www.youtube.com/watch?v=wgDbm_izOXk

Pulsar II Hardware Overview / J.Olsen 13

Time Multiplexed Transfers

• The full mesh backplane is also useful in systems which consist of many parallel processing engines

• In this example– 10 input boards

• 40 Processor Mezzanines– 1 “Gateway” board– 1 output board

14 April 2014

Pulsar II Hardware Overview / J.Olsen 14

Multipath Transfers• If the firmware supports

packet retransmission then there are many ways to get from A to B

• Effective use of under-utilized backplane channels

• Must account for additional latency, however.

• Currently we are simulating firmware to support “route through” packet handling.

14 April 2014

Pulsar II Hardware Overview / J.Olsen 15

Pulsar IIa Prototype• Designed and tested in

2013• Dual Kintex K325T FPGAs• 10 Gbps GTX Transceivers• Many Unknowns:

– First ATCA board design– First FMC mezzanine

design– First Xilinx 7-Series FPGA

design– First high speed design

using high performance PCB materials

14 April 2014

Pulsar II Hardware Overview / J.Olsen 16

Pulsar IIa Bench Testing• A single slot “Mini Backplane” was

developed at Fermilab– Provides power, a base interface RJ45 port,

and loopback for all backplane channels• IPMC microcontroller software was

created to support Telnet and FTP access– Download firmware images to SDHC flash

memory card– Board control via command line

• FPGA local bus is working well at 10 Gbps• Backplane and RTM loopback tests are

working well at 6.25 Gbps– 10 Gbps operation is possible, but requires

fine tuning GTX settings

14 April 2014

6.25 Gbps

10Gbps

Pulsar II Hardware Overview / J.Olsen 17

Pulsar IIa Crate Testing• Our first ATCA shelf is 10G

– 4 x 2.5 Gbps– ELMA rated for ~4 Gbps

• Our crate level testing has demonstrated BER of 4.2E-17 @ 6.5 Gbps– No apparent degradation across

length of backplane • Tested communication with F101

and F125 switch blades• Recently we acquired an ASIS 40G

shelf and tests are ongoing

14 April 2014

Seven Pulsar IIa boards installed in our ELMA 10G ATCA shelf at Fermilab

Pulsar II Hardware Overview / J.Olsen 18

Pulsar IIb Layout• Incorporate everything we learned from

designing and testing the Pulsar IIa• The goal is to have large receiver margins and

wide open eyes at 10 Gbps and beyond• Keep high speed traces direct and clean (“less

is more” philosophy)• Compensate for every connector, via, and

discrete component in the signal path• At 10 Gbps signal transitions are 10-20ps, so

every layer transition and stub matters!• Work closely with the PCB manufacturer to

develop a stackup and trace geometry that meets performance requirements and can be manufactured!

• 14 Layers• Nelco N4000-13 EP SI materials

14 April 2014

Pulsar II Hardware Overview / J.Olsen 19

Conclusion• The ATCA platform is a natural fit for

many HEP applications– Excellent bandwidth, power, and

telecom reliability– Many interesting data transfer options

• Our first ATCA prototype board works quite well– GTX transceivers are solid at 6.25

Gbps, some tuning needed to get to 10 Gbps

• The Pulsar IIb prototype boards are being fabricated now

14 April 2014

Pulsar II Hardware Overview / J.Olsen 20

Backup Slides

14 April 2014

Pulsar II Hardware Overview / J.Olsen 21

Pulsar IIb FPGA Power

14 April 2014