rce platform technology (rpt) gregg thayer ([email protected]) atca and the cob v4
TRANSCRIPT
2
Outline
ATCA
• Advanced Telecommunications Computing Architecture- PICMG 3.0 Standard
- Developed by the telecom industry
• Features- High Availability
• Redundancy
• Hot Swap
• Uses Integrated Platform Management (IPM)
- Rear Transition Module• Separates physical data interface from processing
- High-speed, protocol agnostic backplane
COB
• Cluster On Board
• Designed and built at SLAC
• Compliant ATCA Front Board
3
The ATCA Shelf
The ATCA Shelf Provides
• Monitoring and Management- Superset of Integrated Platform Management (IPM)
• Power and Cooling- Up to 400W/slot
- AC and DC options allow rack power aggregation
- Power options are vendor specific
- Fans provide cooling
• Intra-shelf data transport- Base Interface
- Fabric Interface
- Synchronization Clock Interface
- Update Interface
• 2 to 16 Slot configurations- Small shelves usually have horizontally oriented slots
- Large shelves often have vertically oriented slots
- 14 slots is the largest to fit in a 19” rack
- 16 slot Euro standard uncommon
• High reliability supported by redundant systems
• Slot addressing (1 – 16)- Physical: Left to Right or Bottom to Top
- Logical: Vendor specific
- Use Physical Address when referring to slots
4
ATCA Slot
• Front Board- Management Controller (IPMC)
- Payload
- Power• Management Power
- 30W @ 3.3V always on
- Powers IPMC and all other
management functions
• Payload Power- Negotiated with Shelf Manager
• Rear Transition Module (RTM)- Powered by Front Board
- Has no separate IPMC
5
ATCA Backplane
The ATCA Backplane is divided into 3
Zones
• Zone 1 – Power and Management- Power from dual-redundant -48VDC
supply rails
- System Management via redundant IPM
Bus (IPMB) connection to Shelf Manager
- Hardware Address
• Zone 2 – Data Transport Interface- Protocol agnostic
- Up to 200 Differential pairs
- Up to 10Gbps signaling
- Backplane inter-slot topology varies by
interface
• Zone 3 – Rear I/O- Connects Front Board to RTM
- Not defined by standard
- RTM must be powered through Front
Board
6
Shelf Management
• The Shelf Manager watches over managed devices- Retrieves inventory information, sensor readings
- Receives event reports and failure notifications from Boards
- Reports anomalous conditions• Takes whatever corrective actions it can
- Altering cooling fan speed
- Deactivating boards
- Managed devices are Field Replaceable Units (FRUs)• ATCA Front Boards and RTMs are FRUs
• FRUs communicate with the Shelf Manager through their IPMC - Called an Intelligent FRU
• FRUs may be represented by the IPMC of another FRU - Called a Managed FRU
• RTM is a Managed FRU represented by the IPMC on the Front Board
• Communication between the Shelf Manager and the IPMCs is done over the IPM Bus
(IPMB)- Two-way, redundant, I2C distributed over the ATCA backplane (Zone 1)
- Topology can be bussed or radial
• Shelf Manager can integrate into a larger IPM system in a number of ways- Simple Network Management Protocol (SNMP)
- Remote Management Control Protocol (RMCP)
7
FRU States
• Each IPMC in the Shelf tracks
the state of the FRUs it controls
• Each FRU can be hot swapped
- FRUs spend most of their time
in states M1 and M4
• Some state transitions are
initiated by changes detected by
the FRU itself
- Insertion and Ejection Criteria
• Some state transitions are
initiated by commands from the
Shelf Manager
• Some can be initiated by either
• The state of the FRU is not the
state of the Payload
- Payload operation largely
takes place while the FRU is
in state M4
8
Shelf FRU Information
• Every Shelf stores information describing itself and its capabilities- Storage location is vendor specific, but is often non-volatile storage
located on the shelf itself
- Readable by and through the Shelf Manager
• The Shelf FRU Information contains (among other things)- Shelf Address (name)
- Address mapping table• Physical Address
• Logical Address
• IPMB Address
• Hardware Address
- Data Transport Backplane topology
- Shelf Manager Network configuration (optional)
- Power and cooling capabilities of the shelf
- Can be extended with custom records
9
ATCA Hot Swap, Power Negotiation and E-Keying
• Hot Swap
- Boards can be inserted and removed from the Shelf while the system is live
- The Shelf Manager is responsible for negotiating with IPMCs when FRUs enter and leave the
system
• Power Negotiation
- The IPMC is powered by a 3.3V always present Management Power• Separate from payload power
- When a FRU is inserted, the Shelf Manager inquires as to its power needs• A FRU can be capable of multiple power levels
- The Shelf Manager retrieves the power capabilities of the shelf from the Shelf FRU
Information
- If the needs of the Front Board can be supported, the Shelf Manager allows the IPMC
controlling the FRU to activate the payload power
• Electronic Keying
- The Shelf Manager requests the Data Transport capabilities of each Front Board
- The Shelf Manager retrieves the backplane mapping from the Shelf FRU Information
- Combining these, the Shelf Manager sends commands to enable all compatible channels
- When a board is inserted or extracted, E-Keying will cause the commensurate configuration
changes in the boards with which it communicates over the Data Transport Interfaces
10
Data Transport Backplane Topologies
• Dual Star- Each of two Hub slots connect to
every other slot
- Implies two types of boards: Hub
and Node
- Hub boards must occupy Logical
Slots 1 and 2
• Full Mesh- All slots connect to all other slots
- Mesh capable boards can be used
as either Hub or Node boards in
other topologies
- Can be used to implement all other
non-replicated topologies
• Replicated Mesh- Multiple connections between slots
- Pairs of slots need not share same
number of replications
- Increases capacity, doesn’t change
connectivity
• Other Topologies- Dual-Dual Star
- Multi-plane Switch
11
Data Transport Interfaces
• The specification assumes boards need to interact with one another- Defines four different types of board communication
• The Base Interface
• The Fabric Interface
• The Synchronization Clock Interface
• The Update Interface
• The specification is as protocol agnostic as possible- Only determines connectivity between boards
- Assumes data is transmitted/received serial-differential
- Assumes communication is full-duplex (independent transmit and
receive)
• Limitations to agnosticism- Board must provide IP support on either Base or Fabric Interface
- The Base Interface (if implemented) must satisfy an Ethernet MAC• 10/100/1000BaseT
12
Data Transport Interface Channels
• All point-to-point transport interfaces are characterized in
terms of channels
• A Channel is a group of differential signal pairs that are
physically routed together on the Backplane to provide an
interconnect trunk between two Slots
• The number pairs per channel and maximum number of
channels varies by interface type- A Base Channel consists of 4 differential pairs
- A Fabric Channel consists of 8 differential pairs
- An Update Channel consists of 10 differential pairs
• Slot interconnect topology varies by interface type
• The Synchronization Clock Interface is bussed and uses 6
differential pairs
13
Data Transport – Fabric Interface
• The Fabric Interface is comprised of 15 Channels providing connectivity among
up to 16 boards in a Full Mesh or Star configuration- 120 differential pairs
• Front boards must be capacitively coupled to the backplane to isolate transmitter
and receiver common mode voltages- Limits protocols to DC balanced signals
• The IPMC should be able to disable transmitters as part of the E-Keying process- When disabled, transmitters do not transmit signaling voltages to the backplane
• The Fabric Interface can be partitioned into multiple fabrics among boards- The Dual Star can be used to support two distinct, redundant fabrics by placing hub
boards in Logical Slots 1 and 2
- Replicated Mesh works similarly in shelves with fewer than 9 slots
• Replicated Mesh configurations can also be used to increase capacity in a single
fabric
• All of this is obviously contingent on the capabilities of the Front Boards in the
system- ATCA ensures that only compatible channels are enabled through the Electronic
Keying (E-Keying) process
14
Data Transport – Base Interface
• The Base Interface is comprised of 16 channels providing
10/100/100BaseT Ethernet connectivity among 16 boards in
a Dual Star configuration and an optional connection to the
Shelf Manager- 64 differential pairs
• The Base Interface drivers do not need to be isolated from
the backplane- The Ethernet PHYs are allowed to auto-negotiate prior to system
management enable (E-Keying)
- Base Interface is still subject to E-Keying negotiation
15
Data Transport – Synchronization Clock Interface
• The Synchronization Clock Interface provides a set of clock busses
to enable applications that require the exchange of synchronization
timing information among multiple boards in a shelf- Each bus is a multi-drop, differential pair
• For redundancy, six busses are divided into three redundant groups- CLK1 (A&B) – Telecom Specific
• 8 kHz A/B failover (Digital Telephony)
- CLK2 (A&B) – Telecom Specific• 19.44 MHz A/B failover (SONET reference clock)
- CLK3 (A&B) – User defined• A and B can be used independently, but limited to 100MHz
• Usage is problematic in multi-tenant systems- E-Keying is key to self-consistent configuration
• Negotiates which boards drive the bus and which listen
• Resolves conflicts among multiple master requests
The Cluster On Board
17
• ATCA Front Board
• Base Interface Node Board
• Fabric Interface Mesh Board
• 10G Ethernet
• Synchronization Clock Interface
• Payload Function
• Hosts a Cluster of RCEs
• On mezzanine boards
• Decouples COB development
from mezzanine development
• Cluster Interconnect
• 10G Ethernet
• Connects all RCEs
• Faceplate SFP+
• Connects to other clusters
over ATCA Fabric Interface
• Timing Sources
• ATCA Synchronization Clock
Interface
• External through Rear
Transition Module
• Internally Generated COB
Cluster on Board (COB) Data Transport
18
RCE Synchronization
• The DTM can distribute timing
signals- To DPMs through fan-out on
COB
- To RTM for external transport
- To the ATCA Synchronization
Interface for intra-shelf timing
• The timing signals can
originate- Internal to the DTM
• For simulating external timing
• For local COB synchronization
- On the RTM• On RTM Mezzanine Board
(RMB)
• From an external source
- From the ATCA Synchronization
Interface
19
• ATCA Front Board
• Management via IPMC
• Power Negotiation
• E-Keying
• Temperature Sensing
• Payload Function
• IPMC controls the payload over I2C
busses to functional components
located in Bays
• COB IPMC is based on software
licensed from Pigeon Point Systems
• Extended and modified to
support COB payload
functions
• IPMC communicates to RCEs
through the Bootstrap Interface (BSI)
• IPMC controls the RCE with General
Purpose I/O (GPIO) I2C devices on
the DPM/DTM
• Status Lines
• Reset Line
• Power Usage
• Extensive monitoring of
temperatures, voltages, and currents
Cluster on Board (COB) Management
20
RCE Bootstrap Interface (BSI)
• IPMC uses the BSI to coordinate the RCEs in a Cluster- The RCE contains an I2C slave that is connected to the IPMC
• Visible to the IPMC as a 2 kByte register space
- The RCE signals the readiness of this interface by asserting a signal
connected to the GPIO device on the DPM/DTM• The interface is not ready until the RCE provided information below has been written
• Information provided by RCE
- BSI Version
- Network PHY type
- CE MAC Address
- CE Interconnect Definition
- RCE Status
• Information provided to RCE- Mezzanine board serial number (from ID PROM)
- Cluster Address (Slot/Bay/RCE)
- Cluster Group Name (Shelf Address)
- External Interconnect Definition (From RTM)
• Information provided to RCE on DTM- Cluster Switch Configurations
• From network PHY types for intra-COB links
• Results of E-Keying for inter-COB links
• Presence of FP SFP+ transceivers
- CE Interconnect Definitions
- Shelf IP Information
21
Rear Transition Module (RTM)
• Connects to COB Zone 3- Power and Management on blue connector
- Most management function contained on a
COB standard daughter card
• Physical adaptation layer for DPM signals- 96 CML, full duplex lanes driven by MGTs on
DPM RCEs
- External interface may be application specific
• RTM Mezzanine Board (RMB) connects to
DTM- Doesn’t actually need to be a mezzanine
- 6 Pairs of LVDS signals to DTM• 4 pairs to clock capable I/O
• 2 pairs to general purpose I/O
- 2 pairs of CML signals to DTM MGT (Tx/Rx)
22
Do-it-yourself RTM
• RTMs are the adaptation layer between
the front-end and the COB- It is expected that some users will need
to design their own RTM
- SLAC will have built a few “generic”
RTMs which may also serve
• There are only a few requirements to
build a COB compatible RTM- Obey the COB Zone 3 pinout
- Include the Management (I2C)
Daughterboard• This is supplied by SLAC
• Provides all COB required Management
functionality
- Must include ATCA specified Face Plate
devices• Hot Swap Handles
• LEDs
• SLAC intends to create an RTM kit which
will include what is needed to build an
RTM- Mechanical drawings
- I2C Management daughter board
- COB Zone 3 pinout description
- Bill of Materials
23
COB and RTM
24
COB and RTM
DPM0 DPM1
DPM3 DPM2
DTM CI
IPMC
RTM
Zone1
Zone2
Zone3
Power Conversion
SFP+
25
COB and RTM
BAY0 BAY1
BAY3 BAY2
BAY4
BAY5
BAY6
RCE0
RCE2
RCE0
RCE2
RCE0
RCE2
RCE0
RCE2
26
Data Transport Module (DTM)
• 1 RCE- SD card stores all code to operate
RCE• SOC configuration file
• RCE Core Software
• Application specific software
• Manages the Cluster Interconnect
Switch
• Connected to ATCA Clock
Synchronization Interface
• Connected to RMB
• Connected to ATCA Base Interface
• Connected to DPM timing fanout
• Connected to DPM consoles and
JTAG
27
Data Processing Module (DPM)
• 2 RCEs- SD card stores all code to operate RCE
• SOC configuration file
• RCE Core Software
• Application specific software
• 12 lanes of MGT per RCE to RTM
• 4 lanes of MGT per RCE to CI
• Timing interface per RCE- 1 pair from DTM to MGT reference clock
- 2 pairs from the DTM to clock capable I/O
- 1 pair feedback from user I/O to DTM
• Serial Console and JTAG- Connected to DTM
28
IPM Controller (IPMC)
• ATCA Functions- Communicates with Shelf
Manager
- Power Negotiation
- Hot-Swap
- E-keying
- Temperature Control
• Cluster Configuration- Controls power and reset lines
of RCEs
- Communicates cluster
configuration information to
RCEs
29
COB Activation
• When a COB is inserted into a Shelf- The Management Power is applied and the IPMC boots
- The IPMC requests Shelf FRU information from the Shelf Manager• Uses Shelf Address Map to determine the Physical Slot number
• Retrieves Shelf Address (shelf name)
• Retrieves Zone 2 backplane topology to forward to DTM
• Retrieves Cluster IP information
- The IPMC requests RCE provided information from the BSI of each RCE
- The IPMC requests FRU Information from the RTM• Type, Power requirements
- When the handle switch is closed the IPMC requests permission to
activate from the Shelf Manager• Shelf Manager and IPMC negotiate Power
- When Payload power is applied• The COB FRU enters the Active state
• The RCEs configure and boot
30
RCE States
• The State of the RCE is not the same as the state of the FRU
• The RCE state is constructed from the following bits that the IPMC can read- Mezzanine present (from COB)
- Mezzanine payload power enabled (from COB)
- Mezzanine reports all voltage regulators OK (from mezzanine GPIO)
- SOC Reset line assertion (from mezzanine GPIO)
- RCE Ready line asserted (from mezzanine GPIO)
- RCE Boot Status (from RCE BSI)
• The RCE States are- Not Present
• IPMC can detect the presence of COB Mezzanine
• Each COB Mezzanine reports the number of RCEs
- Powered Off• Prior to payload power application
- Voltage Not OK• Each COB Mezzanine monitors the state of its voltage regulators
- In Reset• The SOC Reset line is held until the Voltage is OK
- Not Ready• Once SOC Reset has been released, the SOC configures and boots
• When the BSI is present, the SOC asserts Ready, the IPMC begins reading/loading the BSI with the values required to complete
RCE booting
- Not Booted• While the RCE boots, it reports a status value to the IPMC
- Running• Once Booted, the IPMC continues to monitor the state of each RCE
31
Summary
• ATCA is a standard that provides solutions for- Monitoring and Management
- Power and Cooling
- Intra-shelf data transport
• The COB is an fully compliant ATCA Front Board- IPMC based on a licensed commercial product
• Extended to support our payload needs
- Hosts a Cluster of RCEs
- Hosts a Cluster Interconnect
- Supports synchronous timing
• The RTM is customizable to the physical interface of your
front-end- A kit will be available to ensure compatibility with the COB
