1

Lessons Learned from SC2004

Jin Tanaka Satoshi Matsui APAN-JP/JGN2 NOCs

KDDI

Chris RobbGlobal NOC

Indiana University

2

1. Background

2. Outline and requirements of demo conducted by Asian participants for SC2004

3. System of APAN/JGN2 NOCs for SC2004

4. Preparation works

5. Reports on demo traffics

6. Analysis of problems and lessons

7. Proposal for improvement

Agenda

3

• APAN-JP/JGN2 NOCs and Global NOC supported a lot of demonstrations conducted by Asian participants at Super Computing 2004 in Pittsburgh. In Bandwidth Challenge, we were related to the supports of all challenge except that of SDSC (San Diego Supercomputer Centre) group.

• However, various problems had happened on the network performance and the inter-domain operation in this SC2004.

• It is important to analyze the highlighted problems from operator's viewpoint, and to propose improvement plans for future high performance demonstrations.

Background

4

Outline and requirements of demos conducted by Asian participants for SC2004

Project Name Contact PointBandwidth of

International LinkBandwidth of Domestic

Link

Prefixes ofDomesticNetwork

Prefixes of SC2004Venue

MTU Routing

Hiraki

hiraki@is.s.u-tokyo.ac.jp

AIST, GTRC Grid Tatebe JGN2 10 G for (1) JGN2 Domestic 10 G for (1) 163.220.52.0/23 and  9K for (1)

There are 4 Demos

TransPAC LA 1 G for(0)&(2)&(3) Peak 1 G,Average 150~200 M MAFFIN 1 G for (0)&(2)&(3)

203.181.194.128/27for (1)

1.5K forOthers

 (1) BWC (10G) Okazaki 163.220.0.0/19 for (0)

 (0) GTRC Generic Demos f-okazaki@aist.go.jp163.220.60.0/24 for(0)&(2)

 (2) Realtime Data Analysis Demowith KEK (100M)

163.220.108.0/24 for(3)

 (3) Gfarm Demos

Fujita

9K forTsukubaRC

fujita@chofu.jaxa.jp1.5K forChofu

JGN2 KitaKyushu RC & KitaKyushu RC 140.221.186.0/24

University of Illinois e2ecc@kyushu.jgn2.jp 140.221.187.0/24

Akiyama 133.1.33.0/25

akiyama@cmc.osaka-u.ac.jp 133.1.69.0/24

192.50.1.192/26

Hirabaru 203.181.194.0/28 140.173.174.0

masaki@nict.go.jp 203.181.194.48/28 MIT Heystack(Westford)

Okamura TransPAC LA 155M

oka@ec.kyushu-u.ac.jp APII/Genkai 155M

TransPAC LA 1 G 203.255.252.192/29 140.221.199.128/25

APII/Genkai 1 G 155.230.20.120/32 140.221.199.0/25

140.221.198.128/25

U-Tokyo Data Reservoir JGN2 10 G T-LEX 10G 192.31.116.0/24 140.221.202.0/259K (enough8K)

Static and

BGP

140.221.184.0/27

Static and

BGP

JAXA JGN2 10 G JGN2 Domestic 10 G203.181.194.96/27TsukubaRC 140.221.217.192/27

Static and

BGP

Static and

BGP

Osaka Univ. TransPAC LA 160 M JGN2 Domestic 160 M 140.221.218.64/27 1.5K

Static and

BGP

JGN2 10 G

TransPAC LA 1 G JGN2 Domestic 1 G

9KJGN2 Domestic 10 G 202.180.40.0/28

1.5K BGP

CJK WIDE 155M 203.38.64.0/18 140.221.214.128/27 1.5K BGP

eVLBI

9Kstatic and

BGPCaltech kihwan@knu.ac.kr (JGN2 Domestic 1 G)

5

System of APAN-JP/JGN2 NOCs for SC2004

• Most of participants were expected to use JGN2 domestic line, TransPAC line, and JGN2 international line

• A concrete work flow was made, involving both NOCs of APAN-JP and JGN2 (See the next slide)

• Information-sharing was done by e-mail or tool between NOCs in order to realize coordinated operation

– Utilization of mailing list with all demo participants and operation staff registered: sc2004@jgn2.jp

– Providing information by web page of both NOCs• APAN-JP/JGN2 International NOC: a special page regarding SC2004 for public us

e　　 http://www.jp.apan.net/NOC/sc2004/sc2004-requirements.htm

• JGN2 domestic NOC: an operation information page for people concerned.　 （ Providing topology maps 、 traffic graphs 、 and looking glass for routers/switche

s

– Adopting tools for task progress management • Using Request tracker http://rt3.jp.apan.net which had been already used success

fully on APAN-JP NOC.• Tickets created per project enabled sharing, managing and updating of task progr

ess on both NOCs among people concerned.　

6

●24-hour NOC engineering support was provided. (Usually, only NOC monitoring is supported 24-hours)

• Shift work was adopted: day-shift and night -shift●International circuit carries were asked to strengthen circuit monitoring and avoid planned outage

• During SC2004, there was one planned outage postponed ●Coordination was reinforced with international circuit carriers and JGN

domestic NOC

24-hour NOC24-hour NOC

Project administrationProject administration

APAN-JPJGN2 InternationalNOC

APAN-JPJGN2 InternationalNOC

JGN2Domestic NOC

JGN2Domestic NOC

InternationalNOCs

InternationalNOCs

Japanese researcher participants

Local researcherparticipants

①Project application

①Project application

⑤⑩Coordination、information-sharing、confirmation of operationcoverage

④Indication

②Awareness of project contents、 Network resource control

③Approval

③Approval

⑦Provisioning

⑦Provisioning

⑨Demos,supporting end users

⑧Demos,supporting end users

⑥Request for international cooperation and network management

⑨Acceptable of filing、troubleshooting,escalation

⑪Project report

Asian researcher participants

System of APAN-JP/JGN2 NOCs for SC2004

7

Preparation works 1• Requirements for network utilization by each participant were

confirmed actively in advance– Path to be used– Network address for participant side and SC2004 venue side– Bandwidth expected to be used– MTU size to be used– Routing methods

• With information shared by participants on the venue, testing schedules were adjusted

• SC2004 group was created on the task-progress management tool “Request Tracker”

– http://rt3.jp.apan.net/ ( see the next slide）– Ticket were generated per demo or per event– For information-sharing, e-mails were sent to all the people registered when

tickets were updated.

• Information page for SC2004 was created– Showing requirement items for each demo, network designs, schedules,

relevant traffic graphs http://www.jp.apan.net/NOC/sc2004/sc2004-requirements.htm

8

For reference on Request Tracker

• There were 99 tickets generated during SC2004– 59 with requests etc.– 40 for welcoming new users

• Tickets were updates on web and by e-mail.

• Updating e-mails were sent to multiple addresses of people concerned.

9

• Configuration within APAN Tokyo XP was modified for SC2004

– MS6(BigIron15K) was installed for the AIST cluster server, providing uplink of 10(8)Gbit/sec.

• JGN2 domestic and T-LEX were newly setup– 4 new VLANs were made in cooperation with JGN2 domestic NOC: JGN2, JA

XA, AIST, Osaka UNIV.– In order to activate Jumbo Frame of Tokyo UNIV. Data Reservoir team, 10G li

nk between APAN TokyoXP and T-LEX was modified to include VLAN ID tags.

• The following routing controls were done on circuits between Japan and US

– In using TransPAC LA 2.4G circuit• The original route: Tokyo – Abilene LA – SC2004 venue was set on general routing policy. 　

– In using JGN2 10G circuit• For controlling traffics going to the venue

　 Static routes to US were set after checking each user’s network address for each booth on the venue.

• For controlling traffics coming from the venue　　 Routes with longer subnet masks than those advertised with BGP to TransPAC LA circuit were s

et to be advertised toward TranPAC Chicago -> Abilene routers, making themselves preferred more within Abilene. For this, Global NOC (Mr. Chris Robb) was asked to accept prefixes of /27 or more at Abilene routers.

Preparation works 2

10

• TransPAC Chicago router terminating JGN2 10G circuit was set to allow remote login

– With coooparation of Global NOC, an account was made on TransPAC tpr-procket (Procket 8801) located at Chicago (StarLIGHT).

– The status of the above router was able to be monitored as needed from Tokyo.

Preparation works 2

11

　 For Reference for Network Topology and Routing for SC2004U-Tokyo

AIST,GTRC

JAXA

Kitakyushu RC

Osaka-U

eVLBI

CJK

Caltech

JGN 2Domestic L2

Network

T-LEXAS23814

WIDEAS2500

QGPOPAS2523

APANTokyoXPAS7660 Abilene

(Chicago)AS11537

TransPAC(Chicago)AS22388

GenkaiXPAS7660

10 Gbps2.4 GbpsLess than 1 Gbps

APII/Genkai Link 1 Gbps

TransPAC LA 2.4 Gbps

JGN 2 10Gbps

Abilene(LA)

AS11537KORENAS9270

CERNETAS4538

192.31.116.0/24

U-Tokyo

AIST,GTRC

JAXA

Kitakyushu RC

Osaka-U

eVLBI

CJK

Caltech

140.221.184.0/27

140.221.192.0/27

140.221.186.0/27140.221.187.0/24

140.221.202.0/25

140.221.218.64/27

140.173.174.0

140.221.199.128/25140.221.199.0/25140.221.198.128/25

SnetAS7082

Abilene(WASH)AS11537

140.221.214.128/27

SC2004 Venue

Haystack

T-LEX 10Gbps

AISTServers

JGN2 Domestic 10Gbps

For BWC163.220.52.0/23 203.181.194.128/27For others163.220.0.0/19163.220.60.0/24163.220.108.0/24

203.181.194.96/27

202.180.40.0/28

133.1.33.0/25133.1.69.0/24192.50.1.192/26

203.181.194.0/28203.181.194.48/28

203.28.64.0/18

203.255.252.192/29155.230.20.120/32

MAFFINAS18125

TsukubaWANAS18127

For BWC

For Others

JGN 2AS17934

MAFFIN 1Gbps

eVLBI 1Gbps

WIDE 1Gbps

12

TransPAC LA circuit JGN2 circuit

JGN2 DomesticJAXA 、 Illinois UNIV (Kitakyusyu RC) 、 AIST 、Osaka UNIV 、 Caltech (From Korea via Genkai XP)

Tokyo UNIV. via T-LEX

CJK

CalTech

JAXA

Illinois UNIV.

Illinois UNIV.eVLBI

AIST

CalTech

Illinois UNIV. & JAXA

Illinois UNIV. & JAXATraffic Reports during SC2004

Tokyo UNIV.

13

Problem and Analysis (1)1.Utilization of Request Tracker

– With 40 members newly registered for SC2004, there were 15 members who did not login. It seems that Request Tracker was not utilized fully.

　【 Lessons & Learned 】– Quite a few members did not understand how to use Request Tracker.– Full explanation about this tool should be given to people concerned in the futur

e

2.Requirement for schedule adjustment of preliminary tests

– In contrast to the actual performance of bandwidth challenge which schedule was under control, on the venue, no schedule adjustment was made for preliminary tests.

　【 Lessons & Learned 】– Schedule adjustment should be done among participants, NOCs and network r

esource managers– Although schedule adjustment was done by NOCs this time, it seems possible f

or persons other than NOCs to do schedule adjustment.– Filing of schedules from participants should be indispensable – The demo information page created by APAN-JP NOC was effective for domest

ic participants, but it was rather confusing for folks of international NOCs.

14

Problem and Analysis (2)3.Time difference between Japan and US

– With 14-hour time difference between Japan and Pittsburgh, there were some requirements for setting modification and troubleshooting. Some of them could not be attended till the following day until a special shift work system was adopted at APAN-JP NOC.

　【 Lessons & Learned 】– As it is a general situation with regards to any international network operation, It

seems a must to build up a system that can cover requirements anytime of the day.

4.Speed-up in selecting route for the venue– With multiple networks existing in Japan, Japan-US and US domestic, there we

re multiple possible routes toward SC2004 venue. There were some signal cases where set-up was delayed due to indecisiveness in selecting the route.

　【 Lessons & Learned 】– It seems necessary to hold early consultation between network managers, NOC

s and participants long before domo performance.– NOCs should provide criteria parameter for selecting suitable route （ bandwidt

h, traffic, MTU size, RTT ） .– NOCs should know network topology not only for home but also for abroad

（ Asia, US domestic including End-to-End).– Cooperation between networks home and abroad is important

15

Problem and Analysis (3)5.Reliability of Procket router (Pro8812 version 2.4.4.1-P)

– Although it was required to set up for advertising separate BGP routes only toward Chicago in addition to usual aggregated routes, unsuppress-map could not be set with Procket. As a result, filtering was applied for the remaining BGP peers.

– With the interface counter indication not precise enough, there were difficulties in real-time monitoring. Problem with SNMP MIB values regarding POS/OC192 interface had also effect on traffic graphs.

– With no SNMP MIB value for 1G/10G Ethernet sub-interface, collection of traffic data per project was not possible.

– L3 discards were constantly monitored at 10GbE-LR interface connecting with T-LEX BigIron, though with no effect on performance. It is presumed that Procket was receiving packets it could not recognize.

　【 Lessons & Learned 】– AS throughput performance over 1G had not been experienced with Procket router

in actual operation, router performance was in question. For this type of important demo, coordination with vendors should be built up in assuming hardware replacement.

– Our domestic vendor suggested that setting for collecting flow data might have effect on performance; the relevant configs were all removed accordingly.

– Fortunately, information on QOS parameters was provided by Mr. Clayton Wager, who was developer of Procket (now Cisco) routers, at SCinet.

– Interface counter and SNMP MIB problems have recovered with version up to 2.5.0.173-B. We should have done version up for reliability.

16

Problem and Analysis (4)

6.Utilization of high-speed & precise traffic graphs – Actual throughput could not be grasped by general MRTG that graphs data of 5-

minutes average/5-minutes interval.

　【 Lessons & Learned 】– Utilization of close-to-real-time precise traffic graphs has been found very useful

in this type of demo both for participants and operation staffs.– The original traffic graphs system developed by Mr. Hirabaru of NICT makes gr

aphs by collecting data every 10 seconds. It very much helped operation staffs grasp the situation during SC2004.

　　 http://mrtg.koganei.itrc.net/

– Recently, Mr. Ikeda also completed a traffic graphs by system that collects data every 10 seconds using SNAPP as a tool of APAN/TransPAC observatory.

http://nms2.jp.apan.net/cgi-bin/snapp/index.cgi

– The Traffic Weather MAP provided by SCinet helped grasp the overall network condition of the venue. We did not notice its existence until the bandwidth challenges were over, coordination with SCinet should have been built up at an early stage.

http://weathermap.sc04.org/　 (disabled)

17

Problem and Analysis (5)

7.JGN2 Chicago circuit trouble– A link flapping trouble occurred on JGN2 Chicago circuit on Nov.10 18:30 – Nov

.11 07:30(JST) – Tokyo UNIV. was not able to do the measurement test due to this trouble– The cause of flapping trouble was a problem on the transport equipment in Salt

Lake City. After the end of SC2004, it was replaced and JGN2 circuit became stable.

　【 Lessons & Learned】– We should have measured and offered the high-speed throughput data regard

to this OC-192 circuit– The importance of the ensuring backup circuit is confirmed again.– If there is no protected circuit, It is desirable we are able to use other circuits or

networks as backup as much as possible. e.g. TransPAC2, SINET, and IEEAF

18

Problem and Analysis (6)8.Difficulties in measuring network performance

– It was filed that expected throughput performance was not gained due to packet loss between Tokyo UNIV.’s machine and the SC2004 venue

• With TCP, packet loss always occurred when exceeding 3Gbit/sec, which made throughput poor.

• With UDP, only 7-8 Gbit/sec throughput was maximum.• As packet loss always occurred when exceeding a certain threshold, some shaping seem

ed to be made somewhere in the backbone network.

– Investigation was made in coordination between Tokyo UNIV., APAN/JGN2 NOCs 、Global NOC in US and the vendor. However, it was not detect within the time frame where the loss was occurring.

– A similar issue, found on the previous day, had been cleared though the cause had not been found.

– With interface counters of all the equipment checked between Tokyo UNIV. and SC venue, no error or loss was found.

19

Problem and Analysis (7)8.Difficulties in measuring network performance (continue)

– While L3 incomplete discards were detected on the Abilene Chicago router (Juniper T320) side connecting with StarLIGHT switch (Force10), Chris tried to modify the set up so as to include a Catalyst, however with no change in the situation. （ This was only 1 hour before the entire SC2004 was over, which made us realize the importance of coordination with SCinet. ）

– This issue has not been solved yet and continuous investigation is necessary. – After time-consuming cause-determination work, it has been confirmed that NO LOS

S occurs even in generating over 3Gbit/sec traffic, though in case of multi-stream, between:

• T-LEX <-> CERN• TPR4 OC-192 POS i/f <-> Abilene Chicago router

JGN2 Abilene

T－ LEX

StarLIGHT

SC2004

1.5G

2G

Total 3.5G

No packet loss !

APAN

20

Problem and Analysis (8)8.Difficulties in measuring network performance (continue)　【 Lessons & Learned 】

– It is preferable to be able to check and view the status of as many machined included in the topology as possible. Therefore, status-checking tools such as Router Proxy are important in addition to coordination between NOCs etc.

– Substantial preliminary measurement and actual operation were necessary with regards to 10G networks such as OC-192 and 10GbE.

– It is necessary to develop a system that enables precise determination of trouble cause ( trouble occurring at backbone portion? Or End-user portion?)

– Troubleshooting for cases with no error or loss on interface is a challenge. Attending emergent software failure should be also counted.

– Network topology toward SC2004 venue, especially US domestic portion, should be grasped precisely.

– With cause-determination work this time, it is not likely that loss is occurring on JGN2 Chicago circuit portion.

– Information –sharing with Request Tracker was useful in troubleshooting, though the cause was not found.

21

Proposal for improvement (1)• From 1G-based measurement to 10G-based measurement

– Making 10G Iperf and BWCTL machines– Preparation of 10G Router Tester (i.e. Agilent Router Tester 900) http://advanced.comms.agilent.com/n2x/– Multiple use of 1G measurement machine– Cooperated performance measurement on users’ ends:

• Heavy-traffic users might be requested to conduct measurement regularly and provide the measurement data to NOCs.

• Cooperation of heavy traffic users would be preferable for troubleshooting.

• Strengthening of collaboration with SCinet and Global NOC– Making a clear operating flow– Building up a special system for collaboration – Enhanced Information-sharing among NOCs and participants in Japan and US

• Substantial preliminary tests including US backbone networks

– Provide NOCs with detailed information as much as possible– Perform preliminary tests as early as possible

• On-the-venue arrangement by APAN NOC staff to realize quick response and precise status-grasping

– APAN NOC member will go and support the Asian demonstrations at SC2005 venue.

22

Proposal for improvement (2)• Ensuring of another path and flexible routing change

– Investigation by using TransPAC2/SINET/IEEAF JP-US circuits for temporary use, comparison etc.

– Preparation of connecting 10G with Jumbo frame to SINET,T-LEX/IEEAF– Building up of a special cooperation system between networks including internationa

l circuits

• Development of operation-status publication tool with regards to domestic equipments

– Development of APAN Router Proxy is under study

• Utilization of advanced schedule management tool– A simple tool was used this time. It is preferable to prepare a web-based tool enablin

g easy writing and browsing.– Detailed information and document with contact person of project are described

• Setting up measuring machines for each node and the venue

– Setting up measuring machine with necessary software/parameter sush as IPv4/IPv6/Jumbo Frame/Iperf at each node. * Utilization of Abilene Observatory

– Setting up a measuring machine on the venue

• Strengthening of collaboration with vender for emergency– To cope with any new issues, coordination should be strengthen with vendors

23

Proposal for improvement (3)• Merits to be maintained

–Information-sharing:• Coordination between international and domestic NOC in Japan

• Utilization of Request Tracker

• Publication of information page for the event

–Flexible APAN-JP NOC system• Attendance during night time

• Allocation of operator per project for efficient management of multiple projects

–Utilization of Abilene Observatory tools–10G-based designing of APAN Tokyo XP with consideration for

Jumbo Frame–Monitoring by high-speed & high-performance traffic graphs

24

Proposal for improvement (4) Difficulty of hop by hop 10G measurement

• The boxes enabling 10G test are installed in the subordinate of every hop • This measurement scheme can most accurately find where the bottleneck is• NOC requires 10G BOX which enables high performance test

Implementation might be difficult, because 10G interfaces are still expensive stillNIC which can achieve the 10G throughput enough, are not appearing on the marketvery often

NOC hopes to set up a 10G box that NOC staffs can handle!For high-speed performance, researchers’ cooperation is necessary !

25

• Schedule tests and real experiments• Communication tool between NOCs and researchers • Share Information (provisioning/trouble/maintenance)

Proposal for improvement (5)Example of central management system

APANJGN2

TransPAC Abilene

SCinet

Other NOCs e.g. EU

Administration

Coordination

Requirement

ParticipantsResearchers

Ticketing

Coordination

Collaboration Collaboration

• Everyone of NOCs and participants can see this system on the web • Design related to traffic graph (e.g. Animated traffic map on SCinet) is useful for grasping the network situation • Possibility of using scheduling tool is use examined by APAN/TransPAC Observatory team

Indication

26

Proposal for improvement (6)

Establish the know-how for lower layers operation

• The importance of the grasping circuit condition is confirmed again– SONET/SDH

• For operation, it is very important to check the status of circuits in cooperation with circuit carriers

• As a recent trend, backbone network based on L2 or Lambda is conspicuous

– Layer2• Difficulty in finding bottle-necks • Apply L3 monitoring technology e.g. ICMP ping, traceroute, other measurement tool• VLAN ID management from end-to-end

– Lambda• Operators can’t monitor and measure performance of circuit/link• Burden for operation on end router/user