Multi-wavelength Optical Transceivers Integrated On Node

PI: Dan Kuchta, IBM and Finisar1

Outline

‣ Overview of MOTION Project and Technology

‣ SAFE (Simplified Analog Front End) ICs

‣ Flip-chip VCSELs and Photodiodes

‣ Transceiver Module Development

‣ Initial measurement results

‣ Network Modelling & Simulation results

‣ Tech-to-Market

‣MOTION Phase 2

2

The Trends for Higher Bandwidth

3

First-level package

Switch ASICoptical

module

• Bandwidth limited by pin density at package / board interface

• Large energy costs for driving > 10 cm transmission lines

Avoid distortion, power, & cost of ASIC-interfacing electrical links Move beyond chip & module pin-count limits Agnostic to upgrades in signaling rates and formats

• Bandwidth limited by pin density at chip / package interface

• Some energy required for few cm long transmission lines but

dominated by E/O and O/E

Pluggable Optics

Co-Packaged Optics

• Bandwidth limited by spectral efficiency – not pin counts

• No E/O or O/E. Energy spent on steering pipes rather than

processing or transmitting bits

Optical Switch

optical

module

Now

Soon

Future

IBM ONRAMPS Program

IBM MOTION Program

XExtra Pain with No Gain

Why are we interested in Co-Packaging?

‣ Primarily for increasing BW from ASICs

– Large ASICs are package pin constrained

– Co-Packaging permits I/O from both sides of the package

‣ Reduction in Power Consumption

– Juxtaposed die do not require high power SERDES

– Lower power I/O cells also use less Si Area

‣ Reduction in Cost

– Stripped down optical packages and reduced function ICs should cost less

– Reduced ASIC area will have higher yield

‣ Expansion of ASIC performance

– Instead of reducing ASIC Area and Power, other choice is to maintain size and add more functionality to the chip up to the original Power constraint

– i.e. Additional switch ports or Additional Memory Hubs

4

Co-Packaging on Organic Laminates: MOTION Phase 1

5

ARPA-E (U.S. Department of Energy) sponsored project, Phase 1: 2 years

– IBM and Finisar Inc.

Target specifications 56GBd NRZ; BER tested to <1E-12 pre-FEC

0oC to 70oC Case

6dB (electrical) link budget (XSR-like)

2 dB optical link margin (30m w/connectors)

< 4 pJ/bit (3.2W, 16 channels)

W:13mm x D:13mm x H:4mm

Package can withstand reflow onto ASIC 1st level laminate

=MOTION=: Multi-wavelength

Optical Transceivers

Integrated on Node

MOTION Transceiver Package Overview

Chip-Scale Optical Package

(CSOP)

MOTION Vision: Multi-Component Carrier

with CSOP for high speed I/O

Final Assembly with lens and clip

attached Fully Assembled with fiber cable

and strain relief

4mm total height

Cu Heat Spreader

Glass Carrier

SAFE ICs,

VCSELs,

PDs

Keel

Simplified Analog Front-End (SAFE2) ICs

‣ SAFE2: Builds on previous designs

‣ 16 channels × 56 Gb/s NRZ = 896 Gb/s/IC

‣ No retiming / CDR

‣ 4 pJ/bit power consumption

‣ Fully DC-coupled: passes 64b/66b &

PRBS31

‣ 55 nm SiGe BiCMOS

‣ CMOS-compatible electrical signal levels

‣ Built-in pattern generators and error

detectors

7

TX Channel 0

TX Channel 1

TX Channel 2

TX Channel 3

TX Channel 4

TX Channel 5

TX Channel 6

TX Channel 7

TX Channel 8

TX Channel 9

TX Channel 10

TX Channel 11

TX Channel 12

TX Channel 13

TX Channel 14

TX Channel 15

SAFE2 TX IC

Bias, Control, & Monitoring

RX Channel 0

RX Channel 1

RX Channel 2

RX Channel 3

RX Channel 4

RX Channel 5

RX Channel 6

RX Channel 7

RX Channel 8

RX Channel 9

RX Channel 10

RX Channel 11

RX Channel 12

RX Channel 13

RX Channel 14

RX Channel 15

SAFE2 RX IC

Bias, Control, & Monitoring

1

6 P

ho

tod

iod

e A

rra

y

1

6 V

CS

EL

Arr

ay

56

Gb

/s

16

Da

ta In

fro

m A

SIC

56

Gb

/s

16

Da

ta O

ut to

AS

IC

1

6 M

on

ito

r P

ho

tod

iod

e A

rra

y

16

SAFE2 TX Channel Architecture

‣ Differential data input with 85Ω terminationand ESD protection

‣ Level-Shifting Attenuator (LSATT)

– Compatible with CMOS signal levels:0.35 to 0.65 Vcm, 0.2 to 1.0 Vppd

– Provides consistent performance across corners and input common modes

‣ Continuous Time Linear Equalizer (CTLE)

– Provides 0 to 6 dB peaking @ 28 GHzequalizing channel from ASIC to TX

‣ Limiting Amplifier (LA)

‣ Multiplexer (AMP)

– Switches between input data,PRBS pattern generator, and fixed 0/1

‣ VCSEL Driver with 3-tap FFE

– LC delay line for power-efficient delay

– 0.75 UI delay per tap

– 2 drivers for VCSEL sparing(only one powered at a time)

‣ Improved performance over SAFE1

‣ Monitor photodiode current amplifier

– Analog output to measure VCSEL optical power

8

ESD& Term

ChipEdge

UnretimedData Input

Monitor PD Current Amp

AnalogTest

PointsAMUX

LSATT

DRV w/ 3-tap FFE

DRV w/ 3-tap FFE

6b DAC(x4)

DRV Biasing

Digital Buffers & Logic

Supply Voltages

3.3V

1.8V

2.5V

1.2V

MU

X

CTLE LA

PRBS Data fromPattern Generator

LC Delay Line

4

To ChipAMUX

ChipEdge

MainVCSEL

SpareVCSELAnalog

Biasing

MonitorPD

SAFE2 TX Summary

‣ Initial hardware undergoing lab test

– Pre-MOTION VCSELs without sparing

‣Multiple channels measured error-free to 52Gb/s

‣ Full testing awaits faster MOTION VCSELs with

sparing

9

Power supply Data mode PRBS mode

1.8 V 820 mA 1.6 A

3.3 V 256 mA 256 mA

Total power 2.3 W 3.7 W

Energy efficiency 2.6 pJ/bit 4.1 pJ/bit

TX Chip Layout

4.6

3m

m

1.64mm

Simulated Power Consumption

TX Chip Photo

Measured energy efficiency@50 Gbps : ~2.4 pJ/bit

Error-free to 52G with low BW VCSEL

50G 56G 60G 64G

SAFE2 Tx IC Electrical driving signal

SAFE2 Tx IC w/pre-MOTION VCSEL (< 20GHz BW)46G 48G 50G 52G

SAFE2 TX Initial Results

SAFE2 RX Channel Architecture

‣ Transimpedance Amplifier (TIA)

– Self-referenced low-noise designconverts photocurrent to voltage

‣ Amplifiers (AMP)

– Convert unbalanced inputto differential output

‣ Analog feedback

– Low pass filtering (LPF) andopamp set low-frequency cutoff

– Vertical slicing level control tofine-tune decision level

‣ Limiting Amplifier (LA)

‣ Continuous Time Linear Equalizer (CTLE)

– Provides 0 to 6 dB peaking@ 28 GHz, equalizingchannel from RX to ASIC

‣ Output driver with 1kV HBM ESD protection

‣ PRBS Checker / Error Detector

– Normally powered down,only used for on-chip testing

11

TIA UnretimedDataOutput

ChipEdge

T-Coil & ESD

LPF

AMUXAnalog

TestPoints

ToChip-levelAMUX

CTLEOUTDRV

PeakingControl

ChipEdge

LA

PD Cathode Bias & DC Sig Det

Digital Buffers & Logic

Vertical SlicingLevel Control

PRBS Checker / Error Detector

Half-RateClock

Supply Voltages

3.3V

1.8V

2.5V

1.2V

AMP

Analog Biasing

ErrorRate

DAC

BU

F

1.0V

SAFE2 RX Summary

‣ Initial hardware undergoing lab test

– Using pre-MOTION PDs with limited BW

‣ Fully functional to 40Gb/s

‣ Full speed testing awaits assembly with MOTION

PDs

12

Power supply Data mode PRBS mode

1.8 V 480 mA 1.2 A

3.3 V 120 mA 130 mA

Total power 1.3 W 2.7 W

Energy efficiency 1.5 pJ/bit 3 pJ/bit

Simulated Power Consumption

RX Chip Layout

4.6

3m

m

1.64mm

RX Chip Photo

Glass Carrier Substrate

‣ Successfully designed, modeled and fabricated a glass substrate that can support at least a 16 channel transceiver with BW >30GHz

‣ Substrate is floor planned for production capability on 4 metal layers with embedded passives

‣ Overcame line impedance issues by recharacterizing the supplier’s capability of dielectric thickness.

‣ Extended modeling capability to include host substrates, uBGAs, ICs and optical devices.

13

56GBd VCSELs and Photodiodes

‣ Designed and fabricated flip chip 56GBd

940nm VCSELs on a production epi and

Wafer fab process

– Realized yields needed for production

– Extending the results to 112PAM4 for

phase 2 feasibility

‣ Implemented a dual aperture structure to

realize laser sparing which dramatically

improves overall system reliability

‣ Designed and Fabricated TWO different

56GBd Photodiodes structures: GaAs

based and InP based.

14

56Gbps (error free)

112Gbps (1E-8)

Glass Carrier Subassembly

15

MOTION glass carrier assembly – top view

Chip join process• SAFE chip – Cu pillar w SnAg cap

• PD/VCSEL – Au/Sn w formic acid reflow

• Tight spacing btw chips ~ 10µm

• Reflow self alignment precision < 1µm

• No significant challenges remaining

Underfill process• Structural UF for electrical chips

• Epoxy based Optical UF for PD/VCSEL

• Material survey / 4 candidates evaluated / one selected

• New process developed

• Stable optical performance after solder reflow

• Remaining challenges

• Automated dispense manufacturability (small distance

btw SAFE and PD/VCSEL using distinct UF materials)

• UF void control

UF void under PD chip

SAFE Rx

SAFE Tx

4xPDs

4xVCSELs

Glass substrate

VCSEL Au/Sn interconnect

Electrical Connection

‣ We have realized a high speed socket that is capable of >30GHz operation.

‣ 400um Pitch LGA to LGA

‣ Allows placement of the Co-Packaged transceiver after the ASIC is mounted to the substrate and/or test infrastructure

– Tested to >10K insertions

‣ We have also designed and fabricated a solderable interconnect for MOTION transceiver.

– Allows the same part to be soldered directly to the host ASIC

– Allows for common assembly and test infrastructure

16

Solderable

Interposer

Transceiver on interposer

LGA

Test Card Emulating MCC

‣ MCC laminate assembled

– SLC 2-2-2 stack up

– GL102F low loss material

– 105 x 105 mm

– Mini-SMP connector for high speed

electrical input/output/clock

– Micro-berg headers for power and

control

– LGA sockets

– Electrical link length matched

‣ Emulates a 1st Level ASIC package

‣ Full Link (both E and O) is possible with

only a clock signal provided.

17

TX SMPs

RX SMPs

Power

Control

IBM Confidential

Optical Solution and Transceiver Assembly

‣ Developed a disconnectable optical

cable solution allowing the transceiver

to be reflowed

‣ Transceiver footprint: 13mm x 13mm

x 6mm high (not including heat sink)

18

How much additional bandwidth

can MOTION provide?

19

HBM HBM HBM HBM HBM HBM

170

GB/s

2 x 12.5 GB/s

Network

TOR

NIC

(1) Co-packaging enables higher-radix switches flatter network.

(2) Optics on CPU & GPU modules enables higher on-node

bandwidth.

(3) Co-packaged optics may free up electrical package pins for more electr. DRAM channels.

SUMMIT-like Node

Possible insertion points of co-packaged optics on switches

example of co-packaged optics enabling

51.2-Tb/s switch on 90 x 90-mm2 carrier

with 13x13 mm2-modules

40% fill factor

The benefits of higher-radix switches enabled by MOTION

20

324x

TOR TOR

P

A A A

P

A A A…

NIC

CORE SWITCH CORE SWITCH…

18x

TOR TOR

P

A A A

P

A A A…

NIC…

18x

SPINE SPINE SPINE…18x

LEAF LEAF LEAF LEAF…36x

36 leaf switches x 18 ports 648 ports

2-level fat tree in a box built from 54

36-port switches

18x

648x

18x

18x

18x

36x

… … … …

SUMMIT-like: 1620 36x36 SWITCH MODULES

(1x)

…324x

20

MOTION: 1280 128x128 SWITCH MODULES

18x

TOR TOR

A A A … A A A

P P

NIC

TOR TOR

A A A … A A A

P P

NIC

CORE SWITCH CORE SWITCH64x

64x

512x

…

64x

SPINE SPINE SPINE…4x

LEAF LEAF LEAF LEAF…8x

8 leaf switches x 64 ports 512 ports

2-level fat tree in a box built from 12

128-port switches64x

64x

128x

… … … …

(16x)

648x…

512x…

2 links / node 6 links / node

• 3x more network end points for 21% fewer switch modules

• 2.8x higher bisection BW for 100 Gb/s per port (11.2x for 400 Gb/s)

• MOTION opens the way to direct-network-attached accelerators

Network performance analysis: MOTION vs. Summit

21

Venus discrete event

network simulator

Traffic and Network

Generator

RX buffer size Infinite

Message size 1024 B

Generation distribution Bernoulli

Data rate 100/400

Gbps

Load [0.1-1]

Adapter/Switch

Type InfiniBand

Data rate per link 100/400

Gbps

Delay 100 ns

Switch Buffer per port 128 KB

Packet size 1024 B

Routing algorithm Random

Uniform, BitComplement, BitReverse: Linear throughput increase for all systems

2.8x and 11.2x higher throughput for 100 and 400 Gb/s data rates

BitTranspose: earlier saturation due to the significantly fewer destination nodes

4.3x and 17.2x higher throughput for 100 and 400 Gb/s data rates

MOTION-400: best mean packet delay for all cases

MOTION (16x56Gbps NRZ Co-packaged Optical Assembly)

‣ The MOTION Co-packaged Optical Assembly(COA) is targeted at low-latency optical-interconnect applications which value;

– High bandwidth density per square-millimeter

– Protocol Agnostic capability up to 56Gbps-NRZ/channel on 16-duplex channels with 1:1 redundancy of optical transmitter

– Low latency, and power, by maintaining NRZ encoding with pre-FEC BER performance <1e-12, and new short-reach (XSR) electrical interface

‣ Three market segments Identified and Pursuing COA applications:

– Massively Parallel Processing / High Performance Computing & AI-Deep Learning

– Metro-access Edge Compute Equipment for network-edge datacenters & AI-Inference

– High-performance FPGA interconnects serving Aerospace, High-resolution Imaging, & future compute-accelerator technologies

‣ Key challenge to commercialization is customer timeline for releasing applications & aligning funding to commit to a production contract

– Concern with OEM funding coming available without an adequate lead time required for a commercial production ramp

– Team will utilize Phase 2 to continue advanced development milestones to prove feasibility and expand product capability to support 56Gbaud (112Gbps)-PAM4 modulation to increase COA value proposition to all markets including “OEM Switch”

22

Technology-to-Market

Co-Packaging on Organic Laminates: MOTION Phase 2

23

ARPA-E (U.S. Department of Energy) sponsored project, Phase 2: 2 years

– IBM and Finisar Inc. (now II-VI Inc.)

Target specifications

Electrical Interface: 80 channels @ 56G NRZ, single ended encoded bus (SE-BUS)

Optical Interface: 32 channels @ 112G PAM-4, 16 fibers, 2 wavelengths

< 2 pJ/bit (< 7 W, 32 channels)

0oC to 70oC Case

6 dB (electrical) link budget (XSR-like)

2 dB optical link margin (30m w/connectors)

W:13 mm x D:13 mm x H:4 mm

Package can withstand reflow onto ASIC 1st level laminate

=MOTION=: Multi-wavelength

Optical Transceivers

Integrated on Node

Phase 2 CMOS Transmitter and Receiver ICs

‣ Tx integrates 32x 112G PAM4 transmit lanes

‣ Quarter rate architecture

‣ Input C4 clock, which comes from a single clock

multiplier, runs at ¼ of the baud rate (56/4=14GHz)

‣ Power budget

– Electrical: 0.3pJ/bit, Optical interface: 0.7pJ/bit

24

‣ Rx integrates 32x 112G PAM4 receive lanes

‣ Quarter rate architecture

‣ Input C4 clock, which comes from a single clock

multiplier, runs at ¼ of the baud rate (56/4=14GHz)

‣ A phase rotator is used to adjust the phase of the

incoming ¼ rate clock. The output of the phase

rotator goes to a quadrature clock phase generator

‣ Power budget

– Electrical: 0.5pJ/bit, Optical interface: 0.5pJ/bit

MOTION Phase 2: 100GPAM4 VCSELs and PDs

‣ In Phase 2 we will continue to extend

the work on high speed VCSELs

112PAM4 on a high volume production

EPI and Wafer Fab

‣ Lowering the cost of fiber optics

communication to DAC cost levels

– Necessary for server to switch

applications

‣ Lowering the total power to <2pJ /bit

(including the light source)

‣Multi Wavelength to further utilize the

fiber bandwidth (cost)

25

MOTION2: Glass Substrate and Optical Assembly

‣ During Phase 2 we will extend the

capability to create a 2l x 16ch x

112PAM4 Transceiver

‣ The glass substrate will remain the

same size (4x the BW density) however

the LGA pitch will shrink to 300um.

‣ Optics will include the 32ch WDM MUX

& Demux in the same vertical space.

‣ Our target is to retain reflowability and

develop a 300um pitch socket for the

transceiver as well.

‣ Thermal path for 105mW/mm2

26

Concept FloorplanConcept I/O Footprint

105mW/mm2

2l

VC

SE

Larr

ay

Optic

Phase 2: IBM SYSTEMS TECHNOLOGY EVALUATION

• Goal: To assess the technology readiness of co-packaged optics

• Optical transceivers soldered directly on the top surface of a laminate package will be built

• Positive results after two evaluation cycles (two cycles of learning) would result in a recommendation that this technology could move into productization phases of work

• Socketed optical transceivers will be evaluated through modeling

• The IBM Systems group will perform this technology evaluation focusing on the thermal & mechanical robustness of packaging:

• Four (4) optical transceivers and one (1) test site die on the top of a single FC-PLGA laminate, assembled with a thermal lid

• Evaluation challenges:

• Assembly Processing

• Package Reliability

• Thermal Performance

Demonstrating a viable path to system integration is necessary before this technology can be considered in a product plan.

Acknowledgements: MOTION Phase 1 Team members & Sponsor

‣ IBM Research

– C. Baks, A. Benner, R. Budd, T. Dickson, F. Doany, B. Lee, W. Lee, M. Meghelli, P. Pepeljugoski,

J. Proesel, M. Taubenblatt, L. Schares, M. Schultz, P. Maniotis, P. Stark, D. Kuchta,

‣ IBM Bromont

– L-M. Achard, P. Fortier, C. Dufort, E. Tucotte, C. Bureau, M. Pion, Y. Cossette, P. Ducharme,

S. Desputeau, A. Janta-Polczynski, P. Minier

‣ Finisar Corporation

– D. Case, P. Chen, F. Flens, J. Glover, C. Kocot, K. Koski, G. Light, T. Nguyen, S. Pandy, S Quadery,

K. Szczerba, B. Wang, P Westbergh

2828

Acknowledgment: “The information, data, or work presented herein was funded in part

by the Advanced Research Projects Agency-Energy (ARPA-E), U.S. Department of

Energy, under Award Number DE-AR0000846. The views and opinions of authors

expressed herein do not necessarily state or reflect those of the United States

Government or any agency thereof.”