PacificWaveandPRP/GRPBigNewsforBigData

DaveReese

29TH NORDUNET CONFERENCEHELSINKI,FINLANDSEPTEMBER 22,2016

Six Charter Associates:

• California K-12 System

• California Community Colleges

• California State University System

• Stanford, Caltech, USC

• University of California

• California Public Libraries

• CENIC is a non-profit created to serve California’s K-20 research & education institutions with cost-effective, high-bandwidth networking

CENIC: California’s Research & Education Network• 3,800+milesofopticalfiber• US$75Mannualoperatingbudget• Membersinall58countiesconnectvia

fiber-opticcableorleasedcircuitsfromtelecomcarriers

• Over10,000 sitesconnecttoCENIC• 20,000,000 CaliforniansuseCENIC• Governedbymembersonthesegmental

level• Collaboratewithover500privatesector

partners• 88 other peering partners

(Google, Microsoft, Amazon …)• Enables worldwide collaboration

PacificWave• Beganasfirstgeographicallydistributedexchangein2004

• PacificWaveisanopenexchangesupportingbothcommercialandR&Epeers

• Currentlyserves29countriespeeringacrossthePacificandWesternUnitedStates

• WithPNWGPandTransPac,announcedthefirst100GbpsTrans-Pacific linkfromTokyotoSeattlein2015

PacificWaveandWRN

• PacificWaveandtheWesternRegionNetworkprovidefora100GbpsnetworkspanningtheWesternUnitedStatesservingPNWGP,CENIC,FRGP,ABQGPandUH.

• PacificWaveandNSFIRNCawardeePIREN(Univ ofHawaii)worktogethersupportingAARNet linkstoCaliforniaandWashingtonandexpansionofhigh-speedservicethroughthePacificIslandsRegion

www. p n w - g i g a p o p . n e t

Nx100GAcrossthePacific• CURRENT:

– TransPac/PacificWave(Tokyo-Seattle)– SINGAREN/Internet2(Singapore-LosAngeles)– SINET/SoftBank/PacificWave(Tokyo-LosAngeles)– AARNET/PIREN/PacificWave(Australia-SEA)– AARNET/PIREN/PacificWave(Australia-LA)

• FUTURE:– UH/PIREN/PacificWave(Guam-Hawaii-LA)

PacificWaveandNSF/IRNC• PacificWavehasbeenpartiallysupportedthroughthreeseparatefive-yearNationalScienceFoundationgrantssupportinggrowth,connectivityandinnovation

• Currentawardpromotes100GexpansionandimplementationofSDXcapabilitieswithinPacificWave(ACI-1451050)

SDX=SDN+IXP

9

AS A Router

ASCRouter

ASB Router

BGPSession

SDNSwitch

SDXControllerSDX

AbstractionLayer(FlowSpace Firewall)OpenFlowSwitches

On-rampLocations(Ethernet/virtualcircuits)

NetworkTestbedEnivironments

CircuitBuilding(NSI)

SDXmiddlewareOpenFlow Controllers

(plural)

Testbed Resources/OtherUses(DTNs) ScienceGroupApplications /Uses

Pacific Wave SDX Testbed Control Plane

Vision: Creating a Pacific Research Platform

Use Optical Fiber Networks to Connect All Data Generators and Consumers,

Creating a “Big Data” Freeway System

“The Bisection Bandwidth of a Cluster Interconnect, but Deployed on a 20-Campus Scale.”

This Vision Has Been Building for 15 Years

Creating a “Big Data” Freeway on Campus:NSF-Funded Prism@UCSD and CHeruB Grants

Prism@UCSD, Phil Papadopoulos, SDSC, Calit2, PI (2013-15)CHERuB, Mike Norman, SDSC PI

CHERuB

These Are Twoof Over

100 NSF Campus Cyberinfrastructure

GrantsMade in the Last 4 Years

How Prism@UCSD Transforms Big Data Microbiome Science:Preparing for Knight/Smarr 1 Million Core-Hour Analysis

12 Cores/GPU128 GB RAM3.5 TB SSD48TB Disk

10Gbps NIC

Knight Lab

10Gbps

Gordon

Prism@UCSD

Data Oasis7.5PB,

200GB/s

Knight 1024 ClusterIn SDSC Co-Lo

CHERuB100Gbps

Emperor & Other Vis Tools

64Mpixel Data Analysis Wall

120Gbps

40Gbps

1.3Tbps

Next Step: The Pacific Research Platform Creates a Regional End-to-End Science-Driven “Big Data Freeway System”

NSF CC*DNI Grant$5M 10/2015-10/2020

PI: Larry Smarr, UC San Diego Calit2Co-Pis:• Camille Crittenden, UC Berkeley CITRIS, • Tom DeFanti, UC San Diego Calit2, • Philip Papadopoulos, UC San Diego SDSC, • Frank Wuerthwein, UC San Diego Physics and

SDSC

ThePacificResearchPlatform(PRP)• NSFCC-NIEandsimilarprojectsrepresentsignificant investmentsincampus

infrastructureincluding SDN,ScienceDMZ’s (~130projects)

• Butthescientistsarestillstruggling withthecomplexityofusingthenetworkandinteroperabilitybetweendifferentimplementations ofScienceDMZ’s

• PRPfocusesonenabling thesciencecommunitiesacrossthePacificregion tomakeeffectiveuseof thehighperformance infrastructure

• Kick-off inDecember2014:takeadvantageoftheregionalinfrastructure;perfSONAR formeasurement/analysisandMaDDashforvisualization

• IncludeDTN’s:useacommonsoftwaresuitefordatamovement; reflectdisk-to-diskperformanceonMaDDash

• Demonstratedasaproof-of-conceptattheCENICSpringmeeting (March2015)

DOE ESnet’s Science DMZ: A Scalable Network Design Model for Optimizing Science Data Transfers

A Science DMZ integrates four key concepts into a unified whole:

– A network architecture designed for high-performance applications, with the science network distinct from the general-purpose network

– The use of dedicated systems for data transfer

– Performance measurement and network testing systems that are regularly used to characterize and troubleshoot the network

– Security policies and enforcement mechanisms that are tailored for high performance science environments

http://fasterdata.es.net/science-dmz/

PRPv0 - An experiment including:

CaltechCENIC / Pacific WaveESnet / LBNLNASA Ames / NRENSan Diego State UniversitySDSCStanford UniversityUniversity of WashingtonUSC

UC BerkeleyUC DavisUC IrvineUC Los AngelesUC RiversideUC San DiegoUC Santa Cruz

17

18

PRPv0 ExperimentThe PRPv0 experiment concentrated on the regional aspects of the research data movement challenge.

§ High-performance interconnection among campus Science DMZs

§ A mesh of perfSONAR toolkit instances§ perfSONAR MaDDash -- Measurement

and Debugging Dashboard§ Flash I/O Network Appliances (FIONAs)

and Data Transfer Nodes (DTNs)§ GridFTP file transfers to quantify

throughput, with results reflected on MaDDash

§ CalREN HPR / AS2153§ A partial mesh of bilateral BGP

sessions across the Pacific Wave distributed exchange

FIONA – Flash I/O Network Appliance:Linux PCs Optimized for Big Data on DMZs

FIONAs Are Science DMZ Data Transfer Nodes (DTNs) &

Optical Network Termination DevicesUCSD CC-NIE Prism Award & UCOPPhil Papadopoulos & Tom DeFantiJoe Keefe & John Graham

Cost $8,000 $20,000IntelXeonHaswell E5-1650v36-Core 2xE5-2697v314-Core

RAM 128GB 256GBSSD SATA3.8TB SATA3.8TB

NetworkInterface 10/40GbEMellanox 2x40GbEChelsi+MellanoxGPU NVIDIATeslaK80

RAIDDrives0to112TB(add~$100/TB)

UCOP Rack-Mount Build: Source:JohnGrahamandTomDeFanti,Calit2

§ DTNs loaded with Globus Connect Server suite to obtain GridFTP tools.

§ cron-scheduled transfers using globus-url-copy.

§ ESnet-contributed script parses GridFTP transfer log and loads results in an esmond measurement archive.

§ FDT – developed by Caltech in collaboration with PolytehnicaBucharest

20

As of 3/9/15, the Pacific Research Platform (PRPv0) as a facility, logs rather good performance: From To Measured

Bandwidth Data Transfer Utility

San Diego State Univ. UC Los Angeles 5Gb/s out of 10 GridFTP UC Riverside UC Los Angeles 9Gb/s out of 10 GridFTP UC Berkeley UC San Diego 9.6Gb/s out of 10 GridFTP UC Davis UC San Diego 9.6Gb/s out of 10 GridFTP UC Irvine UC Los Angeles 9.6Gb/s out of 10 GridFTP UC Santa Cruz UC San Diego 9.6Gb/s out of 10 FDT Stanford UC San Diego 12Gb/s out of 40 FDT Univ. of Washington UC San Diego 12Gb/s out of 40 FDT UC Los Angeles UC San Diego 36Gb/s out of 40 FDT Caltech UC San Diego 36Gb/s out of 40 FDT Table I.2.1: Bandwidth of flash disk-to-flash disk file transfers shown between several sites

for the existing experimental facility “PRPv0.”

January 29, 2016 PRPV1 (L3)

PRP Point-to-Point Bandwidth MapGridFTP File Transfers-Note Huge Improvement in Last Six Months

June 6, 2016 PRPV1 (L3)Green is Disk-to-DiskIn Excess of 5Gbps

Troubleshooting Unidirectional Performance Issues

Measuring performance – IPv6

Measuring Performance – IPv4

25

PRP Timeline

• PRPv1– A routed Layer 3 architecture – Tested, Measured, Optimized, With Multi-domain Science Data– Bring Many Of Our Science Teams Up – Each Community Thus Will Have Its Own Certificate-Based Access

To its Specific Federated Data Infrastructure.

• PRPv2– Incorporating SDN/SDX, AutoGOLE / NSI– Advanced IPv6-Only Version with Robust Security Features

– e.g. Trusted Platform Module Hardware and SDN/SDX Software– Support Rates up to 100Gb/s in Bursts And Streams– Develop Means to Operate a Shared Federation of Caches– Cooperating Research Groups

Resources

www. p n w - g i g a p o p . n e t

Pacific Wavehttp://www.pacificwave.net/https://ps-dashboard.pacificwave.net

CENIChttp://www.cenic.org/https://ps-dashboard.cenic.net

Pacific Research Platformhttp://prp.ucsd.edu/http://cenic.org/files/publications/PRP_Overview_%C6%92.pdfhttp://prp-maddash.calit2.optiputer.net/maddash-webui/

Calit2http://www.calit2.net/

CITRIShttp://citris-uc.org/

ESnethttp://www.es.net/http://fasterdata.es.net/http://ps-dashboard.es.net/

Invitation-Only PRP Workshop Held in Calit2’s Qualcomm InstituteOctober 14-16, 2015

• 130 Attendees From 40 organizations – Ten UC Campuses, as well as UCOP Plus 11 Additional US Universities– Four International Organizations (from Amsterdam, Canada, Korea, and Japan) – Five Members of Industry Plus NSF

CMS

Pacific Research PlatformDriven by Data-Intensive Research

EarthquakeEngineering

Biomedical‘omics

ParticlePhysics

TelescopeSurveys

Visualization, Virtual Reality, Collaboration

Cancer Genomics Hub (UCSC) is Housed in SDSC:Large Data Flows to End Users at UCSC, UCB, UCSF, …

1G

8G

Data Source: David Haussler, Brad Smith, UCSC

15GJan 2016

30,000 TBPer Year

Two Automated Telescope SurveysCreating Huge Datasets Will Drive PRP

300 images per night. 100MB per raw image

30GB per night

120GB per night

250 images per night. 530MB per raw image

150 GB per night

800GB per nightWhen processed

at NERSC Increased by 4x

Source: Peter Nugent, Division Deputy for Scientific Engagement, LBLProfessor of Astronomy, UC Berkeley

Precursors to LSST and NCSA

PRP Allows Researchersto Bring Datasets from NERSC

to Their Local Clusters for In-Depth Science Analysis

Data Flows Over HPWREN

Global Scientific Instruments Will Produce Ultralarge Datasets Continuously Requiring Dedicated Optic Fiber and Supercomputers

https://tnc15.terena.org/getfile/1939

Square Kilometer Array Large Synoptic Survey Telescope

https://tnc15.terena.org/getfile/1939 www.lsst.org/sites/def ault/files/document s/DM%20Introduction%20-%20K antor.pdf

Tracks ~40B Objects,Creates 10M Alerts/Night

Within 1 Minute of Observing

2x40Gb/s

We are Experimenting with the PRP for Large Hadron Collider Data Analysis Using The West Coast Open Science Grid on 10-100Gbps Optical Networks

Crossed 100 Million

Core-Hours/MonthIn Dec 2015

Over 1 Billion Data Transfers

Moved200 Petabytes

In 2015

Supported Over200 Million Jobs

In 2015

Source: Miron Livny, Frank Wuerthwein, OSG

ATLAS

CMS

40G FIONAs

20x40G PRP-connected

WAVE@UC San Diego

PRP LinksCreates Distributed Virtual Reality

PRP

CAVE@UC Merced

DanCayanUSGSWaterResourcesDiscipline

ScrippsInstitutionofOceanography,UCSanDiegomuchsupport fromMaryTyree,MikeDettinger,Guido Francoandothercolleagues

NCARUpgradingto10GbpsLinkOverWestnetfromWyomingandBouldertoCENIC/PRP

Sponsors:California EnergyCommissionNOAARISAprogramCaliforniaDWR,DOE,NSF

PlanningforclimatechangeinCaliforniasubstantialshiftsontopofalreadyhighclimatevariability

UCSD Campus Climate Researchers Need to Download Results from NCAR Remote Supercomputer Simulations

to Make Regional Climate Change Forecasts

average summer afternoon temperature

average summer afternoon temperature

Downscaling Supercomputer Climate SimulationsTo Provide High Res Predictions for California Over Next 50 Years

36

Source: Hugo Hidalgo, Tapash Das, Mike Dettinger

approximately 50 miles: Note: locations are approximate

to CI andPEMEX

Extending PRP/CENIC Optical Backplane via High Speed Wireless Research and Education Network

Real-Time Network Cameras on Mountains for Environmental Observations

Source: Hans Werner Braun, HPWREN PI

14 May 2014: 9 Simultaneous Active Fires in San Diego County

San Diego County Red Mountain Fire Cameras• Southeast (left) “Highway” Fire• Southwest (center rear) “Poinsettia” Fire• West (right) “Tomahawk” Fire

Interactive Virtual Reality of San Diego CountyIncludes Live Feeds From 150 Met Stations

TourCAVE at Calit2’s Qualcomm Institute

HPWREN Users and Public Safety ClientsGain Redundancy and Resilience from PRP Upgrade

San Diego CountywideSensors and Camera

ResourcesUCSD & SDSU

Data & ComputeResources UCSD

UCR

SDSU

UCI

UCI & UCRData Replication

and PRP FIONA Anchorsas HPWREN Expands

Northward

10X Increase During Wildfires

Data From Hans-Werner Braun

• PRP CENIC 10G Link UCSD to SDSU– DTN FIONAs Endpoints– Data Redundancy – Disaster Recovery – High Availability – Network Redundancy

NSF Has Funded Over 100 Campuses to Build Local Big Data Freeways:Imagine Linking All of Them Like the Pacific Research Platform

Red 2012 CC-NIE AwardeesYellow 2013 CC-NIE AwardeesGreen 2014 CC*IIE AwardeesBlue 2015 CC*DNI AwardeesPurple Multiple Time Awardees

Source: NSF

Next Step: Global Research PlatformBuilding on CENIC/Pacific Wave and GLIF

Current InternationalGRP Partners

Questions?