linking u.s.-russian science, education, research & development with high performance networking...

Linking U.S.-Russian Science, Education, Research & Developmentwith High Performance Networking

Natasha Bulashova, Friends and Partners Foundation

Greg Cole, University of Tennessee

Joe Gipson, University of Tennessee

N S F H I G H P E R F O R M A N C E I N T E R N A T I O N A L I N T E R N E T S E R V I C E S P R O G R A M

Intro Statistics Engineering Outreach Impact Funding Future

MIRnet

• NSF Cooperative Agreement (ANI-9730330) to the University of Tennessee with matching funds from Russian Ministry of Science and Technology to Russian partners

• A 6 Mbps IP/ATM service between STAR TAP in Chicago and the M9 switch in Moscow for purpose of linking high performance science and education networks in US and Russia

• A program to encourage applications of high performance networking for US-Russian scientific collaboration

• Network running reliably since July, 1999

Intro


Introduction

• Connectivity

• Peering

• Management

• Project Development

Intro


Connectivity

Intro

• 6 Mbps IP/ATM Service

• U.S. Terminates at STAR TAP in Chicago

• Russia Terminates at M9 switch in Moscow

• US/Transatlantic transport - Teleglobe

• Russian transport - RASCOM


Network Topology

NAP in Chicago is represented with router and ATM switch connected to STAR TAP

NAP in Moscow is represented with router and ATM switch connected to Internet Exchange in M9 which is managed by Russian Institute of Public Networks.

NAP in Chicago is represented with router and ATM switch connected to STAR TAP

NAP in Moscow is represented with router and ATM switch connected to Internet Exchange in M9 which is managed by Russian Institute of Public Networks.

Intro

MIRnet at

Chicago

Moscow

M9

6 Mbps VBR-nrt

TransPAC

Euro-Link

ANL

vBNS

NREN

ESNET

NISN

Abilene

DREN

STARTAPRASCOM

Teleglobe domestic

ATM

RAS

RBnet

MSUnet

FREEnet

SINP


U.S. Users

Intro

• Traffic to U.S.– 93% educational

– 7% .gov/.mil

• Traffic from U.S.– 92% educational

– 8% .gov/.mil

QuickTime™ and aGraphics decompressorare needed to see this picture. DOE (15,533)NIH (4,606)NASA (3,234)NOAA (2,941)MIL (956)Other (404)QuickTime™ and aGraphics decompressorare needed to see this picture. DOE (43,093)NIH (29,038)NASA (39,632)NOAA (1,668)MIL (8,993)USGS (2,886)US Government agency use of MIRnet (megabytes transferred since July, 2000)


Russian Users

Intro


International Traffic Flow to Russia

Intro

#1 U.S.

63% (508 G)

#7 Canada

3% (22 G)

#6 U.K.

3% (24G)

#5 Netherlands

3% (27G)

#3 France

4% (33G)

#12 China

0.4% (3G)

#13 Japan

0.3% (2.8G)

#2 Sweden

12% (99G)

#4 Finland

4% (31G)

Primary Country Providers of Traffic to Russia since July 1, 2000


Introduction

• Connectivity

• Peering

• Management


Intro


MIRnet Consortium

Intro

• US Partners include:– University of Tennessee

– STAR TAP

– Indiana University

– Funding provided by US National Science Foundation

– Connectivity provided by Teleglobe

• Russian Partners include:– Russian Institute of Public Networks

– Friends and Partners Foundation

– Moscow State University

– Russian Academy of Science

– Funding provided by Ministry of Science & Technology

– Connectivity provided by RASCOM/Teleglobe


Budget

• NSF Funding– $800K annually ($4.0

million total)

– $600K for transport (Chicago - Denmark)

– $135K operations ($65K overhead)

– $150K cost sharing (UT)

• Russian Ministry of Science Funding– $500K annually

– ~$300K for transport (Denmark - Moscow)

Intro


The Russian Team

Intro

• Friends & Partners Foundation

• Russian Institute of Public Networking

• Moscow State University

• Russian Academy of Science


Friends & Partners Foundation

Intro

QuickTime™ and aPhoto - JPEG decompressor

are needed to see this picture.

• Originating member of Russian MIRnet team

• Worked with US staff for 7 years on several US-Russian networking projects

• Focus on developing domestic Russian infrastructure via CIVnet program

Natasha Bulashova, President, F&P


Russian Institute for Public Networking

Intro

Russian Backbone Network (RBnet)

Alexei Platonov, Exec. Director, RIPN


Moscow State University (MSU)

Intro

• Added late to original NSF proposal

• ATM decision changed leadership to MSU

• Recipient of Ministry Funds

• Controls Project in Russia

• Determines who in Russia is routed across MIRnet


Russian Academy of Science

Intro


Introduction

• Connectivity

• Peering

• Management


Intro

Intro Statistics Engineering Outreach Impact Funding FutureIntro

Project Development

F & PJan’94

KORRnetJun’94

NATOSep’95

CIVnetJan’97

NSF RFPJun’97

ProposalAug’97

Revised ProposalMar’98

Cooperative Agreement

Jul’98

TeleglobeSep’98

AnnounceSep’98

Moscow/MASSNov’98

MADAS 1.0Sum’99

IP UpJun’99

IP DownJul ‘99

LaunchSep ‘99

MeetingNov ‘99

RenegotiateFall/Winter ‘97

DemoFeb 22 ‘00

DemoJun ‘00

DemoAug 8 ‘00 MADAS 2

Sep ‘00

NewsletterSep ‘00

TeleglobeSep ‘00

ReviewOct ‘00


Usage Statistics

• Usage Analysis

• MIRnet Administrative Data Analysis System (MADAS)

Statistics

Intro Statistics Engineering Outreach Impact Funding FutureStatistics

• More traffic has transited MIRnet in the 4.5 months since June 1, 2000 (1.2 terabytes) than in the 9 previous months (1.1 terabytes)

Usage Statistics

(total daily network throughput in gigabytes)


Monthly Traffic Flow - destination country

Statistics

-

50,000

100,000

150,000

200,000

250,000

300,000

350,000

400,000

1999-09 1999-10 1999-11 1999-12 2000-01 2000-02 2000-03 2000-04 2000-05 2000-06 2000-07 2000-08 2000-09 2000-10

France

Sweden

Finland

Denmark

USA

Russia


Monthly Traffic Flow - source country

0

50,000

100,000

150,000

200,000

250,000

300,000

350,000

400,000

1999-09 1999-10 1999-11 1999-12 2000-01 2000-02 2000-03 2000-04 2000-05 2000-06 2000-07 2000-08 2000-09 2000-10

Netherlands

France

Finland

Sweden

Russia

USA

Statistics


Russian Use by Institution (% total)

0%

20%

40%

60%

80%

100%

1999-09

1999-10

1999-11

1999-12

2000-01

2000-02

2000-03

2000-04

2000-05

2000-06

2000-07

2000-08

2000-09

2000-10

Other

Ural State Univ(Ekaterinburg)RU Space Science Internet

MIRnet Multicast Service

Russian Academy ofScienceMEPHI

FREEnet

Chernogolovka

Moscow State Univ

Statistics


0%

20%

40%

60%

80%

100%

1999-09

1999-10

1999-11

1999-12

2000-01

2000-02

2000-03

2000-04

2000-05

2000-06

2000-07

2000-08

2000-09

2000-10

USC

Univ of Wisconsin

Univ of Chicago

UW-Madison

Brown Univ

UC Berkely

UIUC

Georgia Tech

MIT

NLANR

U.S. Top 10 Institutions (% total)

Statistics


Russian Use by Protocol (% total)

0%

20%

40%

60%

80%

100%

1999-09 1999-10 1999-11 1999-12 2000-01 2000-02 2000-03 2000-04 2000-05 2000-06 2000-07 2000-08 2000-09 2000-10

Other

ICMP

UDP-Other

TCP-WWW

TCP-FTP

TCP-Other

Statistics


0%

20%

40%

60%

80%

100%

1999-09

1999-10

1999-11

1999-12

2000-01

2000-02

2000-03

2000-04

2000-05

2000-06

2000-07

2000-08

2000-09

2000-10

GrandTotal

Other

TCP-SMTP

ICMP

UDP-Other

TCP-FTP

TCP-WWW

TCP-Other

U.S. Use by Protocol (% total)

Statistics


MIRnet Administrative Data Analysis System (MADAS)• Tracks all MIRnet usage

• SQL database fed every 10 minutes from Cisco Netflow data

• All non-proprietary software tools

• Analysis by countries, domains, hosts, protocols, time periods, traffic volume

• Generates standard graphics and allows user-based queries

• Supplements standard tools (mrtg, etc.)

Statistics


Network Engineering

• Link Configuration

• MIRnet Access Scheduling System (MASS)

• NOC Services

Engineering


Link Configuration

Engineering

Router CISCO7507

Chicago

Router CISCO7507

Chicago

Router CISCO7507

Moscow

Router CISCO7507

Moscow

PVC for regular applications (PVCr), 4 Mbp/s

PVC for multicast applications (PVCm)

PVC for special application (PVCs)


MIRnet Access Scheduling System

Engineering

• Proposed system to provide users mechanism for registering themselves and their applications and to reserve use of the network for certain applications and time periods with reasonable assurance that resources they need will be available.

• To provide management with mechanism for implementing policy in an on-line database system which, for the most part, will run unattended.

• To provide differentiated classes of services (via PVCs) which can be reserved and which are managed in such a way as to ensure reasonable network service



Engineering

• The reservation system will involve a registry of users, applications, and networks, and a reservation database with which users can reserve service for specified dates and times.

• The actual approval of request is to be policy driven as much as possible and mostly automated so that users receive immediate response.

• The database system will interact with the MIRnet end point routers in Moscow and Chicago periodically updating access control lists which govern which PVCs the IP flows traverse.



Engineering

• Through this system we can formalize the provision of network services so that users can feel confident about the network meeting their needs.

• System satisfies concerns of DOE for authorizing users and applications - and for providing an audit trail of actual use. Meets the same security concerns on the Russian side.

• Use of the system should lead to an increased confidence in the network and resulting increased use.



Engineering

• Elements in place include:

– MIRnet administrative Data Analysis System (for back-end monitoring and reporting)

– User/application registry

• Elements to be developed include:

– Network description component

– Calendar/scheduling component

– Policy component


MIRnet NOC Services

• Currently using public domain software for monitoring network and contacting series of pagers (24/7) if problems found

• Transitioning to arrangement with Indiana University for providing 24/7 NOC services

Engineering


Outreach Activities

• Presentations

• Demonstrations• F&P Outreach

• Information Dissemination

Outreach


Information Dissemination

Outreach

• MIRnet Users Database System

• 1000 US-Russian Partnerships Described

• Partnerships Funded by NSF, DOE, CRDF, NIH, IREX

• US Russian/NIS Language Programs

• Searchable by Institution, City, Investigator, Keyword


Information Dissemination

Outreach

• Letter/Newsletter distributed to over 500 US-Russian partnerships


The MIRnet WWW Site

Outreach

• Bilingual

• Database driven

• Dynamic Page Construction

• Browser independent

• Mirrored in US, Russia

• Content– Proposal

– Project Information

– Applications

– Services

– Usage Statistics


Impact

• Institutional Users

• Applications

• Benefits

Impact


Russian Institution UsersInstitution Source Percent Destination Percent

Moscow State University 77,966 24.8% 298,249 30.4%Chernagolovka 19,178 6.1% 241,040 24.5%MEPHI 7,380 2.3% 78,194 8.0%MIRnet Multicast Service - 0.0% 52,818 5.4%Moscow Power Engineering Institute 1,966 0.6% 30,725 3.1%Gubkin Russian State University of Oil and Gas 8,469 2.7% 30,217 3.1%Chelyabinsk FREEnet 1,990 0.6% 29,018 3.0%Ural State University (Ekaterinburg) 11,818 3.8% 26,363 2.7%State Public Scientific-Technical Library of Russia 461 0.1% 17,767 1.8%Russian Space Science Internet 4,521 1.4% 14,444 1.5%RAS Ural Regional Academic Network 1,733 0.6% 13,679 1.4%Shemyakin and Ovchinnikov Institute of Biochemistry 3,232 1.0% 10,114 1.0%Plantetary Geophysics 200 0.1% 9,597 1.0%RAS Steklov Mathematical Institute 531 0.2% 9,021 0.9%Kurchatov Institute 7,480 2.4% 8,870 0.9%Institute of Precision Mechanics and Computers 486 0.2% 7,248 0.7%Yaroslavl Regional Network 1,951 0.6% 7,207 0.7%RAS Institute of Radioengineering and Electronics 669 0.2% 4,798 0.5%Semyenov Institute of Chemical Physics 1,901 0.6% 4,703 0.5%Zelinsky Instiute of Organic Chemistry 676 0.2% 4,497 0.5%Novgorod State University 496 0.2% 4,265 0.4%Nesmeyanov Institute of Elemental-Organic Compounds 636 0.2% 4,023 0.4%Other 160,548 51.1% 75,640 7.7%

Total 314,288 100.0% 982,497 100.0%

Impact

Megabytes transferred July 1 - October 14, 2000


US Institution Users

Impact

Institution Source Percent Destination Percent

NLANR 69,224 13.1% 5,540 4.3%University of Chicago 5,980 1.1% 5,208 4.1%Root Servers.net 11,109 2.1% 4,317 3.4%Penn State University 6,042 1.1% 3,701 2.9%University of Wisconsin 24,189 4.6% 3,279 2.6%STAR TAP 3,094 0.6% 3,091 2.4%Brown University 251 0.0% 2,927 2.3%Oak Ridge National Lab 1,170 0.2% 2,837 2.2%University of Illinois UC 13,131 2.5% 2,826 2.2%NOAA 1,248 0.2% 2,765 2.2%University of Tennessee 11,905 2.3% 2,738 2.1%Georgia Tech 28,450 5.4% 2,345 1.8%University of Southern California 449 0.1% 2,195 1.7%University of Texas 2,367 0.4% 2,078 1.6%MIT 28,648 5.4% 2,072 1.6%Ohio State University 1,754 0.3% 2,009 1.6%Georgetown University 143 0.0% 1,832 1.4%Argonne National Lab 8,659 1.6% 1,712 1.3%University of Michigan 5,400 1.0% 1,695 1.3%University of Virginia 683 0.1% 1,680 1.3%University of Kansas 4,197 0.8% 1,605 1.3%Rutgers University 1,019 0.2% 1,539 1.2%Other 297,532 56.5% 68,208 53.2%

Total 526,645 100% 128,200 100%

(megabytes transferred from June 1 - October 14, 2000)

Intro Statistics Engineering Outreach Impact Funding FutureImpact

.

US-Russia Networking Project: Web Cache for Science and Education

Averaging nearly 1.2 gigabytes of total traffic per day, the heaviest single application of MIRnet during its first year has been between the U.S. National Laboratory for Applied Network Research (NLANR) web cache and the Russian academic web cache located in Chernogolovka (scientific city located 50 km northeast of Moscow). Managed by the Theoretical Physics Institute, the goal of the project is to study the properties of WWW traffic and determine optimal path calculations for receiving and sending information via the distributed cache servers. The Cherogolovka web cache functions as the root web-cache for the Russian science and education community.


.

Telecommunications for Technology Transfer and Distributed Learning

The primary goal of the project is to develop an advanced infrastructure able to supply video conferencing (both point-to-point and multi-point) as a regular service for institutions involved in US-Russia research and academic collaboration and the commercialization of technologies. During the first year of MIRnet’s operation, the two involved institutions (the Zelinsky Institute and the University of Missouri-Columbia) have engaged in regular video-conferences supporting a variety of programs - and have achieved good results using very low bandwidth (64 kbps) H.323 and H.120 technologies.


.

Moscow State/Iowa State University Cooperation in Nuclear and High-Energy PhysicsThe project is in support of cooperation between three units of Moscow State University -- the Institute of Nuclear Physics, the Department of Theoretical High Energy Physics and Department of Physics of Atomic Nucleus -- and the International Institute of Theoretical and Applied Physics at Iowa State University.

MIRnet traffic between Iowa State University and Moscow State University often exceeds 500 megabytes daily. The graph above shows total traffic transiting MIRnet between the two institutions since September, 1999.

0

200

400

600

800

1000

1200

1400

1600

Megabytes


.

Nonproliferation and Arms Control Projects in the FSU

Since February, 2000, MIRnet has been used by the Oak Ridge National Laboratory to support its various collaborations with the Kurchatov Institute and other Russian research laboratories. In June, 2000, peering arrangements were put in place enabling all DOE laboratories in the US to use MIRnet. The primary applications discussed and observed related to US-RF programs are in nonproliferation and arms control. Several video-conferencing demonstrations have been held - including with senior US government leaders - to demonstrate the utility of high performance networks in supporting existing US-RF collaborative efforts. All of these demonstrations have been coordinated with Oak Ridge National Laboratory and the Kurchatov Institute.

Oak Ridge - Kurchatov Institute traffic from February, 2000.


Weapons Disposal: Reactor Physics / Thermal hydraulics Analyses for Plutonium

Disposition The US and RF are engaged in a cooperative program to dispose of weapons usable plutonium in pressurized water reactors in both countries. While plutonium fuels have been routinely used in western pressurized water reactors (PWRs) in France and Germany, such fuels have not been used in Russian reactors (VVERs). For the past four years, ORNL has been responsible for transferring technology for mixed oxide (MOX) fuel to Russia and Kurchatov Institute has been responsible for educating ORNL staff as to the design and operation of VVERs.

MIRnet has supported this cooperative program through video-conferencing, ftp, use of the Centra Symposium web-based teaching package,

whiteboard, and applications sharing.

Academician Ponimarev-Stepnoi (seated, on screen from Moscow) of the Kurchatov Institute addresses the June 13, 2000 video-conference involving participants from Moscow, Washington, Argonne National Lab in Chicago and Oak Ridge. Included was Howard Baker, former US Senate Majority leader, heading up a Blue Ribbon Panel observing US-Russian programs.


US-Russia Materials Protection, Control & Accountability programThe objective of the lab-to-lab MPC&A Program is to enhance, through U.S.-Russian technical cooperation, the effectiveness of nuclear materials protection, control, and accounting in Russian nuclear facilities. The enhancements are implemented by Russian institutes. The U.S. laboratories provide funding for the Russian institutes through laboratory-to-laboratory contracts.

The two partnering institutions, the Oak Ridge National Laboratory and the Kurchatov Institute in Moscow, have used MIRnet since February, 2000 to support a variety of applications - including video-conferencing (demonstrations, seminars, contract negotiation), large database (200+ Mbytes) transfer,

radio propagation analysis, and application sharing.

US participants interact with Dr. Evgenii Velikhov, President of the Moscow Kurchatov Institute, and other Russian academicians during the February 22 video-conference between Moscow, Washington, Knoxville and Oak Ridge. In addition to the video-conferencing, a demonstration was conducted of application sharing

for joint control of a radio propagation analysis


NASA-Russian Space Science Internet (RSSI) CollaborationFTP dominates as the primary application between hosts on NASA’s high performance NREN network and the Russian Space Science Internet. With daily flows occasionally reaching 500 megabytes or more, MIRnet supports a wide variety of NASA-

RSSI programs and activities.

Traffic from NASA hosts to RSSI hosts is shown above. The primary application is ftp which often

exceeds 256 Megabytes transferred daily.


US-Russian Military Officers Training: Humanitarian LawThis program is in support of a new Russian Federation Defense order which requires humanitarian law training for all members of the Russian Armed Forces. Working with the University of Virginia Center for National Security Law, the Russian Ministry of Defense will begin utilizing MIRnet to support curriculum development, transfer of training materials, and live classroom training between sites in Moscow and Charlottesville, Virginia. The initial demonstration

event is scheduled for November 30, 2000.


Sample Academic Partnerships Utilizing MIRnet Services

• Experimental High Energy Physics High energy physicists in the Skobeltsyn Institute of Nuclear Physics at Moscow State University use MIRnet for collaborating with US partners at the Fermi National Accelerator and the Stanford Linear Accelerator. Data is shared and analyzed from experiments performed in the particle accelerators for the purpose of obtaining insight into macroworld structures.

• Investigation of Magnetospheric Perturbances Moscow State University and the Space Physics Research Laboratory at the University of Michigan are using MIRnet in support of a joint project involving a heavy computational component and the creation/maintenance of a shared cosmophysical information database addressing experimental and theoretical research in the area of magnetospheric and space physics.

• Research of on-line collaboration methods in distance learning technology development Based on the collaboration of Ural State University, Perm State University, and North Caroline State University in the framework of their 1997-2000 project “Collaboration in the field of distance learning in the area of business management” supported by the US State Dept., the project is exploring methods of remote collaboration via video- and audio- data exchange

(mBONE and H.32x). • US-Russia Collaboration in Plasma

Astrophysics Project is multi-disciplinary, involving specialists in numerical magnetohydrodynamics, plasma physics, and astrophysics. Russian scientists work at Cornell during one-month visits, and the US scientists visit Russia on a regular basis. Collaboration on developing 2D computer codes between Keldysh Institute and Cornell.


Benefits

• Increased network capacity for data sharing

• Some capacity available for richer communications (video-conferencing)

• Spurring development of high performance infrastructure in Russia

• Encouraging US and Russian partners to learn about new network technologies

Funding


Funding

• US Potential– NSF REU Supplement

– NASA, DOE, DOD

• Russian Reality

Funding


The Future

• Applications

• Capacity Expansion

• Russian High Performance Network Expansion

• Extension to other NIS countries

Future

Intro Statistics Engineering Outreach Impact Funding FutureFuture

Application of Scientific Visualization to MarineEcosystem AnalysisCenter for Coastal Physical Oceanography at Old Dominion Univ. with oceanographic colleagues in Moscow and Vladivostok. MIRnet services for visualization (fly-thrus of oceanographic models), large data sets (ftp), video-conferencing, etc.


Radiation Epidemiology Collaboration

The Radiation Epidemiology Branch of NCI proposes to use MIRnet to cooperate with partners in Russia, Kazahkstan, Ukraine and Belarus. Part of the project focuses on epidemiological follow-up on the Chernobyl disaster.


Distance Learning: International Security in a Changing World

Stanford University will offer a course "International Security in a Changing World" to students at Yaroslavl State University and Southern Ural State University. Both Russian universities are now (recently) routed across MIRnet. Efforts focus now on identifying mutually available technology to be used for the coursework.

IS/IDL, Institute for International Studies, Stanford University


US-Russian InternetPhysics OlympiadIn April '00 high school students in Novosibirsk and St.-Petersburg (Russia) combined with students in Seattle and San Diego (America) to participate in an internet-based science competition. Each of eight teams were composed of four American and four Russian students. Each pair of team halves were linked with their own private chat and whiteboard connection so they could consult in solving the problems. They were presented with questions and then submitted answers through an internet-based platform which was controlled at a central site; the system delivered each team's answers to separately located panels of judges.

Novosibirsk State University and Cornell University propose to use MIRnet to enhance opportunities and expand participation.

Seattle, Washington

San Diego, California

Saint Petersburg, Russia

Novosibirsk, Siberia

Sixty Four Students

And a Significant Number of Physicists and Technologists!


Collaboration on Spallation Neutron Source (SNS), Oak Ridge National LaboratoryInterest from and potential collaboration with Kurchatov Institute, Obninsk, other Russian scientific communities on the new Spallation Neutron Source project being constructed at the Oak Ridge National Laboratory.


Planned Capacity Expansion

• Negotiations with Teleglobe and Russian partners to expand to 155 Mbps

• Link split between MIRnet (80 Mbps) and academic commodity traffic (75 Mbps)

Future


Russian High Performance Network Expansion

• Critical need to expand beyond Moscow region

• First priorities: St. Petersburg and Novosibirsk

• Success depends upon working with Russian academic network partners (RBnet)

• Recent dramatic decrease in regional telecommunications costs for non-profit purposes will help

Future


Extension to other NIS countries

• Conversations with Ukrainian and Kazakh academic communities

• Interest from US side in Belarus

• Can happen only after MIRnet upgrade

Future

Russia

Belarus

Ukraine Kazakhstan

linking u.s.-russian science, education, research & development with high performance networking...

Documents

russia intro

russian science

users intro traffic

mirnet slide

teleglobe russian partners

fp slide

ripn slide

rascomteleglobe slide