building a lambda switching capable infrastructure

Ontario’s gateway to research and discovery

Le portail ontarien de la recherche et la découverte

Thunder Bay

Timmins

Sudbury

North BaySault Ste. Marie

BarriePeterborough

BellevilleKingston

Ottawa

Guelph OshawaToronto

Oakville

St. Catharines

Welland

London

Waterloo

Sarnia

Windsor

Hamilton

Building a Lambda Switching Capable Infrastructure

Sam Mokbel, Project DirectorOptical Regional Advanced Network of Ontario



Network Highlights

• 20 year access to dark fibre as a strategic long-

term investment

• Complements the government’s investment in

research and education

• Co-ownership of optical fibre with private sector

partners

• Full ownership of equipment

• Customized to specific requirements of a research

and education network

• Fully operational by Spring, 2003 and financially

sustainable within 3 years



ORION

• Ontario’s Research and Innovation Optical

Network (ORION) is a new high-speed, optical

network that brings broadband access and

connectivity to Ontario’s research and

education (R&E) institutions

• Created to support innovative and collaborative

projects and activities in Ontario’s publicly

funded research and education community

• Will connect more than 100 Ontario institutions

and organizations with other local and global

networks



User-Identified Applications

• ORION will connect users to each other, the

Internet, CA*net4 , and other R&E networks

around the world

• Once fully deployed, ORION will allow: • virtual classrooms and laboratories

• complex multimedia-based interactivity

• real-time collaborative research

• shared and ready access to large genomic and biotech

databases.

• grid computing

• Transport of operation level data such as email, web browsing, ftp



ORION Operating Guidelines

• The ORION network manager will seek the help of a user technical advisory panel on important engineering issues

• The ORION and member network managers may classify user traffic and apply traffic engineering rules to optimize bandwidth utilization and enhance performance

• Research and Internet traffic of users share the same physical infrastructure

• All ORION backbone circuits are to be at OC192 speeds

• All ORION PoPs will have Gigabit capable equipment to interconnect the members

• PoP site hosts are ORION partners

• Users can interconnect to ORION directly or through a CBN

• RANs or CBNs that interconnect local institutions manage their own local infrastructure



ORION Functions

• Collects traffic from directly-connected

institutions and community networks

• Transports inter-member and external traffic

• Provides advanced services

• Enables exchange of knowledge

• Peers with other advanced R & E networks

• Transports Internet traffic for some members

• Nucleus of a new telecommunication

infrastructure to the R&E community



Operations Requirements

• Define interface to the members• Layers 1, 2, and 3• Demarcation• Bandwidth• Reporting• Administration

• Set objectives for equipment and fibre

availability

• Develop processes and guidelines for• Network management

• Network maintenance

• External connectivity



PoP Requirements

• Interconnection points for local members

• Physically accessible to ORANO staff on a 7x24 basis

• Supply the required environment to operate the ORION equipment

• Host ORION optical, data, and management equipment

• Integrate with ORION’s remote management approach

• Neutral to the termination of 3rd party fibre

• Located at partner operated and member endorsed premises



Transport Requirements

• Optical transport

• Lambda at 2.5Gbps or higher between PoPs

• Scalable up to 8 lambdas per PoP-to-PoP section

• Dynamic lambda setup and tear down

• No OADM between PoPs

• Terminal equipment located at partner hosted

PoPs

• Transport IP traffic

• Capable of provisioning private layer 2 circuits and

lambdas between users



Service Requirements• QoS capable point-to-point paths between ORION

members

• Provide layer 2 or optical light path connectivity between

users and external advanced networks

• IPV6 and IPv4

• IPV4 Multicast

• Support multi-homing of members

• Internal and external routing of member IP traffic

• Shape Internet traffic to/from members

• Accommodate new technology testing

• Enterprise level network availability



Thunder Bay

Timmins

Sudbury

North BaySault Ste. Marie

Barrie

Peterborough

Belleville

Kingston

Ottawa

GuelphOshawa

TorontoOakville

St. Catharines

WellandLondon

Waterloo

Sarnia

Great Lakes

Hamilton

WindsorChicago

Cleveland

Duluth



ArchitectureThunderba

y

Sudbury

NorthBay

Toronto UT

Wat

erl

oo

Lond

on

Win

dsor

Kin

gsto

nO

ttaw

a

Toronto: York

U

Hamilton

DataSwitching/

Routing

WDM

OpticalFiber

Pet

erbo

roug

h

Gue

lph

SSMarie

Sarnia

Welland

Oshawa

Belleville

Timmins

St.

Cat

harin

esOakvill

e

Barrie



Fibre Routes

TORO2(York)

TORO1(UofT)

TORO3(Front)

Oakville(Sheridan)

Hamilton(McMaster)

St.Catharines

(Brock)

Welland(Niagara)

SOUTH2f

Windsor(Windsor Library)

London(Western)

Waterloo(Waterloo)

Guelph(Guelph)Sarnia

(Lambton)Oshawa

(Durham)

Belleville(Loyalist)

Kingston(Queen’s)

OTWA(City Hall)

Peterborough(Fleming)

NORTH 2f

BARIE(SCBN)

Sudbury(Sudbury Wired)

Sault S

aintM

arie (Sault C

)

ThunderBay(Lakehead) N

orth Bay

(Nippissing)

WEST 2f

Canarie

East 2f

Tim

mins

Northern

Sault-Timmins 2f

OpticalPoP

Repeater



PoP

Active OpticalEquipment

Member Interface andData Equipment

Power Manager

Power ConditioningEquipment

Passive DWDMEquipment

DWDM Shelves

DWDM Shelves

7 ft. Bay7 ft. Bay7 ft. Bay7 ft. Bay

Fiber Management Tray

Extension shelf

Environmental control unit

Rectifiers Shelves

Battery Shelves

Battery Shelves

Rectifiers Shelves

Out of Band Management

Fiber Patch Panel

Data Routing/Switching

GE

Supervisor Engine

GE

powerControlShelf

Input/Output

Tributary Shelf2.5 GWT, 10GWT

Main Shelf - MOR+, OSC,2.5 GWT, 10GWT



Routing Layer

Thunder Bay(Lakehead U.)

SS. Marie(Sault C)

Sudbury(Laurentian

U.)

Toronto(York U.)Guelph

(U. of Guelph)Waterloo

(U. of Waterloo)

London(U. of Western Ontario)

St. Catharines(Brock U.)Windsor

(Public Library)

Hamilton(McMaster U.)

Toronto(U of Toronto)

Peterborough(SS Flaming C)

Kingston(Queens U.)

Ottawa(City Hall )

North Bay(Nipissing U.)

Toronto(151 Front Street)

GE

Express GE links

Backup DS3

Backup OC-3

Ottawa(CANARIE PoP)



Integration

• Multiple suppliers for• Long haul and local fibre • PoP space and repeater huts• Optical, data switching, power, and remote

management equipment

• Challenges in• Finalizing full design requirements• Finding and communicating compatible specs and

coordinated installation and testing• Uniform configuration• Who does what?



What Fibre Glut?

• Fibre-rich sections not necessarily where we needed them

• Significant number of local loops are not readily available

• Toronto had the most severe shortages• Where is all that fibre that has been deployed over the

past decade?• Have the carriers used their fibre in the most efficient

way?

• Optimal network design may not necessarily be feasible due to

• fibre shortage• fibre stranding• fibre holes



Power and Grounding

• Long haul optical equipment uses large amounts of DC power

• Most PoP sites located at universities and colleges do not have DC power readily available

• Initial vendor requirements were for an average of 150A per node and isolated dedicated ground

• Would have required massive power conversion systems and physical construction to provide grounding and floor enforcement to support massive batteries

• Working closely with the vendor resulted in• Reduced power requirements to 50A• Shared building ground



Remote Management• We are not a carrier, yet we are deploying a carrier class

network• No capacity or resources to maintain 3700KM of fibre

and 31 repeater sites • Must rely on outsourcing for maintenance, but must

maintain control over member connectivity• Designed the network into static and dynamic

components• fibre and repeaters on long routes don’t require many changes, can

be hosted and maintained by a carrier partner• minimized the number of regen repeater sites• PoP sites where members connect are hosted by member partners

and jointly operated and maintained• building remote access capability into the PoP design



Partnership With PoP Sites

• ORION uses a distributed operations model that is centrally coordinated

• PoP hosts are an integral part of the ORION network operation team

• PoP sites have a stake in the success of ORION to provide service to their own institutions, local region, and the rest of the R&E community

• Not a commercial relationship; the PoP hosts are making significant contributions to the network

• Relationship fulfills the knowledge exchange mandate of ORION



Lessons Learned so Far -1

Change Management

• Be flexible & creative, but stay focused• Final network will be different from initial design• Small details matter, clarify assumptions• Be diligent: this is a network for 20 years• Engage your users early; this is their network• Find out for yourself: your partners perspective may

be different• Formal contracts make you appreciate what is

important to your partners and members• Communication, communication, communication



Lessons Learned -2

Carrier networks are different from enterprise networks

• Repeater hut space can get you: real estate in rural Ontario is not dirt cheap

• Higher speed and longer distance = stringent requirements • Power is not the same as enterprise class• We spent as much time on PoP layout & design as on

transport or IP layer• Think about insurance (liability & loss)• Justify your design by requirements not by best or previous

practices• Documentation, documentation, documentation



Lessons Learned -3

Need a first class team

• Engineers, communications staff, and project managers must have been there before

• Mistakes can be costly• No room for experimentation• Team needs telecom/carrier experience

• Startup organization and a new network can be stressful

• Everyone on the team must be a self starter



www.orion.on.ca

Thank You – Merci !

building a lambda switching capable infrastructure

Documents