new optical lab facility @garr: set-up and first results · white rabbit: python+influxdb+grafana...

New optical Lab facility @GARR: set-up and first results

GLORIA VUAGNIN

INFRASTRUCTURE DEPARTMENT GARR

Prague, September 3rd, 2019

10th CEF Networks workshop

Status of the current infrastructure

Gloria Vuagnin // CEF 2019// Prague, 09/03/2019 2

(2011) Huawei 1/10 G• 10G/40Gbps channels not enough for the

needs of the core• Close to the end-of-life• Maintenance (+2Y) ends 2020

(2015) Infinera 10/40/100 G• 500Gbps superchannels• Fine for the capacity needs of Southern

Italy• High power and space consumption

(2017) AW 2x100G among Core PoPs• Buffer solution – it works super fine

(2019) New fiber footprint on North-East Area

• North-Eastern region: closed topology / ARNES direct Interconnection / Lightnetinterconnection

• ECMWF new site in Bologna

MI2MI1

BO1

RM2

BA1

• Open Line System (partially disaggregated) to replace the oldest infrastructure

partially disaggregated == decoupling of photonic elements and transponders

• April 2019: we issued a RFI on open line systems and optical transport network equipment (Juniper, ADVA, ECI, Huawei, Infinera)

• August: tender for DCI (Infinera G30) for an INFN-GARR join project on Distributed Datalake for Science

• By the end of 2019 -> tender (hopefully into GÉANT framework)

How GARR sees its evolution


We are ready to stand a period of possible inconvenience

(several different DWDM technologies in our network)

in order to build, for the near future, an open infrastructure

Resounding concepts:

Disaggregation

Alien wave

Automation

Analytics

Telemetry

Green

4

What do we have in our mind

Gloria Vuagnin // CEF 2019// Prague, 09/03/2019

We need and we want:

• Acquire knowledge of different available DCI technologies

o Data we are interested: o Reach (in several different scenarios)

o Tolerance to our old infrastructure in the transition time,

o AW working solutions (management and monitoring)

o Figure of merit and acceptance criteria

o New valuable features (analytics …)

• Practice North-Bound Interfaces

o Monitoring & Provisioning

• Start to build an environment for automation

A test facility is a must-have

Building skills and competences for evolution


An independent-from-production NMS control the lab photonic infrastructure

Optical lab test: the set-up

Gloria Vuagnin // CEF 2019// Prague, 09/03/2019 6A

MP

100 km

75 km25 km

25 km

A/D MUX/DEMUX

F-O

AD

M

F-O

AD

M

A/D MUX/DEMUX

Photonic Layer

NODE A3 degree ROADM• Flexgrid• Colorless/Directionless

NODE B3 degree ROADM• Flexgrid• Colorless/Directionless

NODE CF-OADM – 8 channels• Flexgrid• C-Band passthrough

EDFA amplifierExtended C-Band

75 km

DNA LAB

The lab nodes: XTM Infinera


A

BAmp

C

16 spools of single G.652d fiber

(vendor: Fujikura)

• 8 spools 50km -> 4 pairs

• 8 spools 25 km -> 4 pairs

In total 600km single fiber

The flexible topology changes through a patch panel on the rack top

Fiber spools


From these PoPs we can configure links that are as long as:

• ~60 km

• ~1000 km

• ~1500 km

Links towards the production environment


From the lab two cables go down and reach the

RM2 and RM1 PoPs

Paths on the production environment


End Sites km dB Ila Raman ROADM

RM1 – PI 430 118 4 0 2

PI – FI 136 40 0 2 1

FI – RM2 360 104 3 4 1

Total 926 262 7 6 4


RM1 – RMT 22 10 0 0 2

RMT -- FRA 8 5 0 0 1

FRA – RM2 29 12 0 0 1

Total 59 27 0 0 4


RM1 – PI 430 118 4 0 2

PI–FI-BO1 259 74 0 4 2

BO1-BO–PE 466 144 4 0 3

PE – RM2 361 96 2 0 2

Total 1516 432 10 4 9

Short Metro Ring

Long Ring

Intermediate Ring

ECMWF access to GARR (temporary solution)

For the Italian Data Lake initiative:

Large bandwidth over hundreds of kms

Decisions to make: which transponder for 100GE

11

Site 1 Site 3

Site 2

200Gbps

400Gbps

200Gbps

~700km -220 dB

MI1

BO3

ECMWF


1. Transponder TP100GOTN XTM Infinera

Client: 100G Ethernet

Line: CFP1 100G ACO - QPSK coherent

2. Juniper ACX6360 (and MX240)

Client: 100-Gbps (QSFP28) pluggable interfaces

Line: 200-Gbps CFP2-DCO coherent DWDM pluggable interfaces, which support 100-Gbps QPSK and 200-Gbps 8QAM, and 16QAM modulation options

3. Infinera Groove G30

Client: up to 4x100GbE QSFP28

Line: CHM1 sled – 400G 2xCFP2-ACO (100G QPSK, 150G 8QAM, 200G 16QAM)

Test transponders 100G

12Gloria Vuagnin // CEF 2019// Prague, 09/03/2019

Transponder TP100GOTN XTM Infinera

13

Client: 100G EthernetLine: CFP1 100G ACO - QPSK coherent DWDM

•OTU-FEC-DEG (forward error correction degraded)

•OTU-FEC-EXE (excessive errors, FEC_FAIL from the transponder)

reach

[km]

channel number

(Huawei convention)FEC

adjacent

channels

OSNR

margin

node A

[dB]

OSNR

margin

node B

[dB]

alarms

b2b 43 SD-FEC n 7 7

60 41 SD-FEC n 5 5.5

60 41 SD-FEC y (40) 4.9 5.5

1000 43 SD-FEC n 1 1.5 FEC_EXE

1000 41 SD-FEC y (40) 1.2 1.8


Juniper ACX6360


Client: 100-Gbps QSFP28Line: 100-Gbps QPSK CFP2-DCO coherent DWDM

•OTU-FEC-DEG (forward error correction degraded)

•OTU-FEC-EXE (excessive errors, FEC_FAIL from the transponder)

•Channel Huawei convention: 43 (193.95), 41 (194.05)

reach

[km]

channel number

(Huawei convention)FEC

adjacent

channels

BER

node A

BER

node B

Qvalue node

A [dB]

Qvalue

node B [dB]alarms

b2b 43 SD-FEC n 4.3E-11 1.1E-11 16.1 16.2

60 41 SD-FEC n 2.5E-07 1.1E-07 13.7 13.9

60 41 SD-FEC y (40) 7.8E-07 1.8E-07 13.6 14.1

1000 43 SD-FEC n 1.00E-03 1.00E-03 7 7 FEC_DEG

1000 41 SD-FEC y (40) 1.60E-03 4.00E-03 9.3 8.4 FEC_EXC

Client: 100GbE QSFP28

Line: CHM1 sled – CFP2-ACO 100G QPSK

Groove test


ModulationReach

[km]

Channel

number

(Huawei

convention)

adjacent

channels

BER

node A

BER

node B

Qvalue

node A

[dB]

Qvalue

node B

[dB]

OSNR

margin

node A

[dB]

OSNR

margin

node B

[dB]

alarms

60 43 n 1.3E-08 1.2E-08 14.8 15 20 20

60 41 y (40) 2.7E-08 3.5E-08 14.7 14.7 20 20

1000 43 n 3.00E-03 2.20E-03 8.7 9 15.2 14.5

1000 44 n 2.43E-03 5.33E-04 8.9 10.3 15.5 16.5

1000 41 y (40) 6.34E-03 1.58E-03 8 9.5 14.5 15.5

1000 44 y (43) 3.36E-03 1.12E-03 8.6 9.6 15 16.2

1500 43 n 1.50E-02 1.60E-02 6.7 6.5 14 13.6

1500 41 y (40) 5.5 5.6 BER Test errors sync

QPS

K

Client: 100GbE QSFP28

Line: CHM1 sled – CFP2-ACO 150G 8QAM – 200G 16QAM

More modulation tests on Groove


ModulationReach

[km]

Channel

number

(Huawei

convention)

adjacent

channels

BER

node A

BER

node B

Qvalue

node A

[dB]

Qvalue

node B

[dB]

OSNR

margin

node A

[dB]

OSNR

margin

node B

[dB]

alarms

59 43 n 1.42E-04 2.60E-03 11.1 8.9 20.3 20.7

59 41 y(40) 2.80E-04 3.10E-04 10.7 10.6 20 19.7

926 43 n 1.10E-02 3.40E-03 7 8.6 15.3 15.6 Loss of Frame

926 41 y (40) 5.40E-03 6.40E-03 8.1 7.8 15 14.7 Loss of Frame

59 43 n 4.20E-03 4.20E-03 8.4 8.4 20.5 19.4

59 41 y (40) 8.12E-03 8.12E-03 7.5 7.5 18.8 18.8

926 43 n Loss of Frame

8QA

M16

QA

M

Groove Monitoring using SNMP-influxDB-Grafana


ECMWF access to GARR (temporary solution)

For the Italian Distributed Data Lake initiative:

Large bandwidth over hundreds of kms

Which transponder for 100GE

18

Site 1 Site 3

Site 2

200Gbps

400Gbps

200Gbps

~700km -220 dB

MI1

BO3

ECMWF

ECMWF IDDLS

INFINERA

XTMOK NOK

Juniper

ACXOK NOK

INIFNERA

GROOVEOK OK


Time & Frequency in Italy


INRIM in Italy has a dedicated fiber infrastructure

• The aim of the infrastructure is to provide T&F service to Italian Research Institutions (CNR, ASI, INAF) and Industrial Users.

• INRIM main priority is Research

NMI fundamentals: redefinition of the second, better timescale etc.

fundamental physics: geodesy (relativistic or not), VLBI, relativistic experiment, quantum technologies, space.


WR for a time service: a collaboration with INRIM

• Several research institutions might profit from a source of “INRIMcertified time”

• Not all of them have the same requirements in terms of time accuracy

In July 2019, in collaboration with INRIM, we started to test the White Rabbit protocol over DWDM in our network

• The aim is to check which accuracy we can reach using the WRprotocol on our optical infrastructure (using a pair of fibers, amplifiers and ROADMs)

• We want to measure the average accuracy/stability we can get in different parts of our network

• We want to understand the complexity of operating the WR devices

Playing with a White Rabbit


LEN1

WRS 1

WRS 2 LEN2

10 MHzIN

10 MHz pps

Free running Clock

Gran Master

Counter

MM M

SS S

Ch1 Ch2Int Ref out

LEN3

Lab set-up

White Rabbit: Python+InfluxDB+Grafana


We set-up an optical lab to have the possibility to test new devices and/or solutions in different scenarios.

The lab is a versatile facility, it can work as:

• a completely independent environment

• as part of GARR infrastructure

We built it to enable GARR to evolve its network with:

• a step-by-step approach

• independently checking the value of each solution

Having the possibility to start building our system for:

• a systematic approach to collect data from the network

• good debugging processes

Thank you

Conclusions

23Gloria Vuagnin // CEF 2019// Prague, 09/03/2019

new optical lab facility @garr: set-up and first results · white rabbit: python+influxdb+grafana...

Documents