THE EASTERN REGIONAL NETWORK (ERN):SIMPLIFYING MULTI-CAMPUS RESEARCH COLLABORATIONS

Barr von OehsenAssociate Vice President, Office of Advanced Research ComputingRutgers, The State University of New Jersey

[ 2 ]

• Eastern Regional Network Vision and Mission• Science Drivers• History• Proof of Concept• Commercial Cloud• perfSONAR• Future Projects• ERN Partners

TABLE OF CONTENTS

[ 3 ]

Eastern Regional Network (ERN)

• Vision: To simplify multi-campus collaborations and partnerships that advance the frontiers of research, pedagogy, and innovation.

• Mission: Through a partnership of educational institutions, research facilities, regional network providers, and Internet2, the ERN will provide layered and transparent access to shared data and computing facilities for research projects located at and between partner sites. The resulting regional research and education platform will support a diverse set of science drivers and emergent educational opportunities and offer the educational research community access to a broad range of collaborative multi-institutional resources that are not available on any one campus alone.

[ 4 ]

Solving Challenges Related to:• Supporting and enabling research collaborations that span multiple universities• Sharing and accessing data (containerized workflows)• Building, managing, supporting, and sharing distributed federated

resources/infrastructure that spans multiple campuses and commercial cloud designed to enable research

• Creating performant, software-configurable, and intelligent networks to enable access to research resources

• Creating testbeds around new and emerging technologies, software stacks, etc., being asked for by the research community

• Having a safe space for Central IT to experiment with new and emerging technologies without the worry of breaking enterprise infrastructure

• Sharing expertise and building collaborations across both research and IT silos• Workforce development, especially for schools that don’t typically have access to

campus cyberinfrastructure, research testbeds, new and emerging technologies, etc.

• Writing broader impact statements on grant proposals

[ 5 ]

What Are The Science Drivers?

• Cryo-EM• Proteomics• Genomics• Brain Imaging• Climate Research• High Energy Physics• Digital Humanities• Social Sciences• Marine and Coastal

Science• Medical/Clinical Research

• Precision Health• Systems Biology• Computational Social Science• Computer Science• Network Research• Multi-Scale Modeling• Chemistry

[ 6 ]

ERN Origins and Evolution

• Germ of an idea: at 2017 National Research Platform meetingsCan the regional research platform idea work in the Northeast?Rutgers, OSHEAN, KINBER

• Coalition of the Willing: January 2018 gathering at Rutgers-NewarkCan we find something of common interest to work on?KINBER, Rutgers, OSHEAN, MGHPCC, Internet2, NYSERnet

• Resource Federation Proof of Concept: Can we work together to prototype a potentially beneficial resource federation idea?Rutgers, MGHPCC, Syracuse, NJ Edge, U Maine, Google, Internet2“Early if not elegant” approach; emphasis on getting people to work togetherUsed Rutgers Federated Ecosystem as model

[ 7 ]

Federated Computing Across New Jersey – ERN Model

State-wide multi-campus, distributed HPC and storage, fast, low-latency network that is part of global Science DMZs:• NSF CC* Funded (OAC-1659232)• SDN Based 100 Gbps Network

Core• Data Transfer Nodes (FIONA)• Advanced Computing and Storage• Performance and monitoring

support (perfSONAR, XDMoD)• Testbed as a service• Federated across campuses• Policy driven

Local Storage

HPC

DTN

Local Storage

HPC

DTN

Local Storage

HPC

DTN

New Brunswick

Newark

Camden

Central StorageRepository

DTN

PE

NYSERNet, I2

I2, NJEdge, MagPi, KINBER

[ 8 ]

Origins and Evolution

• Northeast perfSONAR Mesh:Can we work together toward a regional science DMZ?Federation sites plus• BU, Brown, Colby, CEN, KINBER, Network Maine, OSHEAN,

UNHLess-specialized technology allows expanded participation

• Growing interest: Maybe we really can do this?[in progress] expanded steering committee; per-project working groupsAll of the above plus• U Delaware, NJIT, U Buffalo, Bucknell, UMass Amherst, Yale,

Princeton, BTAA, U Kentucky, Case Western Reserve

[ 9 ]

Resource Federation Proof of Concept

Recipe for joining:https://github.com/rutgers-oarc/ern-poc

§ Six sites, five states§ Ability to launch containerized (Singluarity)

jobs as well as traditional HPC jobs, from any site to any other site including commercial cloud, via SLURM scheduler.

§ Working with Google, Cisco, Internet2, SchedMD (SLURM), and OSG on enhancements

§ InCommon authentication and authorization§ Policy based federation of computing

resources§ Reserving slices of the network§ Cloud bursting§ OSG federation with containerized SLURM

(slurmd)§ Testing different data sharing approaches

as well as filesystems

[ 10 ]

Resource Federation Proof of Concept

[ 11 ]

Federation with Commercial Cloud

• Partnership with Google Cloud and SchedMD (SLURM) to develop federated hybrid environment• Allows federation across resources (including cloud services)• Allows us to offer one stop shop for users• Allows on demand elastic computing

• Initial Proof of Concept• Set up federated environment between Rutgers and Google Cloud• Google Cloud instance is set up similar to local resource, includes SLURM• Connection through Internet2 between both environments• Able to submit jobs directly from Rutgers HPC to Google Cloud using

SLURM submit script• Elasticity/bursting capability now being tested• Next steps: Extend to Jetstream, OSG

[ 12 ]

ERN perfSONAR Mesh

• 23 nodes, 14 sites, six states• New sites welcome• Still in teething phase• Improves visibility on

connectivity/bandwidth/delay• Starting point for connectivity

improvements (where they matter)

• Prerequisite for a regional data DMZ

Site: http://mesh.hpc.rutgers.edu/maddash-webui/

Recipe for joining:https://github.com/rutgers-oarc/ern-poc

[ 13 ]

Community Ownership

• ERN is not a Rutgers project, it is a community project• Loosely based on the condo model• Proof of concepts led by partner campuses, regional network providers,

industry, etc.• Test out new ideas to address challenges• Can be funded or lead to funding• For the greater good of the community

• Using CaRCC stakeholders document as a guide• Steering Committee (10 members)• ERN Council in process

• Leverages other community projects

[ 14 ]

Looking ahead – Some Possibilities• Regional technology testbeds • Regional Science DMZ (Data DMZ)• Common interface to resources, independent of provider (Atrio?)

• Campus resources, national resources, commercial cloud • Resource bartering system/Marketplace

• Advanced network technologies (experiment->pilot->production)• Simplified engagement with national resources and commercial cloud

• National Research Platform (NRP)• Open Storage Network (OSN)• Big Data Hubs• Open Science Grid• National Resources (XSEDE, GENI, Cloudlab, Chameleon)• Commercial cloud providers (Azure, AWS)

• Regional infrastructure proposals (NSF CC*, CICI, Mid-Scale, etc.)• Improve competitiveness on larger solicitations• Mosaic of smaller proposals that benefit the whole

• New security models• Zero trust? Google BeyondCorp

[ 15 ]

ERN PartnersUniversities:• Brown University• Case Western Reserve• Columbia University• Franklin and Marshall• George Washington University• Indiana University• NJIT• Penn State University• Princeton University• Rutgers University• Syracuse University• Yale University• University of Chicago• University of Delaware• University of Illinois

• University of Kentucky• University of Maine-Orono• University of Massachusetts-Amherst• University of Michigan• University of New HampshireNetwork Providers:• CAAREN• CEN• Internet2• KINBER• NetworkMaine• NJEdge• NYSERNET• OARnet• OSHEANOrganizations:

MGHPCCNortheast Big Data Innovation Hub