the$science$dmz$...the$science$dmz$ eli$dart,$network$engineer$ esnetscience$engagement...
TRANSCRIPT
The Science DMZ
Eli Dart, Network Engineer ESnet Science Engagement Lawrence Berkeley Na;onal Laboratory
Moving My Data at High Speeds over the Network
TNC16
Prague, CZ
June 12, 2016
Overview – XXX
• Science DMZ Mo;va;on and Introduc;on
• Science DMZ Architecture
• Science DMZ Security
• Wrap Up
2 – ESnet Science Engagement ([email protected]) - 06.07.16 © 2015, Energy Sciences Network
• Networks are an essen;al part of data-‐intensive science – Connect data sources to data analysis – Connect collaborators to each other – Enable machine-‐consumable interfaces to data and analysis resources (e.g. portals), automa;on, scale
• Performance is cri;cal – Exponen;al data growth – Constant human factors – Data movement and data analysis must keep up
• Effec;ve use of wide area (long-‐haul) networks by scien;sts has historically been difficult – Especially for large-‐scale data movement
Mo,va,on
3 – ESnet Science Engagement ([email protected]) - 06.07.16 © 2015, Energy Sciences Network
The Central Role of the Network
• The very structure of modern science assumes science networks exist: high performance, feature rich, global scope
• What is “The Network” anyway? – “The Network” is the set of devices and applica;ons involved in the use of a remote resource • This is not about supercomputer interconnects • This is about data flow from experiment to analysis, between facili;es, etc.
– User interfaces for “The Network” – portal, data transfer tool, workflow engine – Therefore, servers and applica;ons must also be considered
• What is important? Ordered list: 1. Correctness 2. Consistency 3. Performance
4 – ESnet Science Engagement ([email protected]) - 06.07.16 © 2015, Energy Sciences Network
TCP – Ubiquitous and Fragile
• Networks provide connec;vity between hosts – how do hosts see the network? – From an applica;on’s perspec;ve, the interface to “the other end” is a socket
– Communica;on is between applica;ons – mostly over TCP
• TCP – the fragile workhorse – TCP is (for very good reasons) ;mid – packet loss is interpreted as conges;on
– Like it or not, TCP is used for the vast majority of data transfer applica;ons (more than 95% of ESnet traffic is TCP)
– Packet loss in conjunc;on with latency is a performance killer
5 – ESnet Science Engagement ([email protected]) - 06.07.16 © 2015, Energy Sciences Network
A small amount of packet loss makes a huge difference in TCP performance
Metro Area
Local (LAN)
Regional
Con;nental
Interna;onal
Measured (TCP Reno) Measured (HTCP) Theoretical (TCP Reno) Measured (no loss)
With loss, high performance beyond metro distances is essentially impossible
6 – ESnet Science Engagement ([email protected]) - 06.07.16 © 2015, Energy Sciences Network
Working With TCP In Prac,ce
• Far easier to support TCP than to fix TCP – People have been trying to fix TCP for years – limited success – Like it or not we’re stuck with TCP in the general case
• Pragma;cally speaking, we must accommodate TCP – Sufficient bandwidth to avoid conges;on – Zero packet loss – Verifiable infrastructure
• Networks are complex • Must be able to locate problems quickly • Small footprint is a huge win – small number of devices so that problem isola;on is tractable
• What if I don’t use TCP? – TCP benefits are significant, but are not the only reason for Science DMZ – Architecture, cost, opera;onal benefits
7 – ESnet Science Engagement ([email protected]) - 06.07.16 © 2015, Energy Sciences Network
PuDng A Solu,on Together
• Effec;ve support for TCP-‐based data transfer – Design for correct, consistent, high-‐performance opera;on – Design for ease of troubleshoo;ng
• Easy adop;on is cri;cal – Large laboratories and universi;es have extensive IT deployments – Dras;c change is prohibi;vely difficult
• Cybersecurity – defensible without compromising performance
• Borrow ideas from tradi;onal network security – Tradi;onal DMZ
• Separate enclave at network perimeter (“Demilitarized Zone”) • Specific loca;on for external-‐facing services • Clean separa;on from internal network
– Do the same thing for science – Science DMZ
8 – ESnet Science Engagement ([email protected]) - 06.07.16 © 2015, Energy Sciences Network
Dedicated Systems for Data
Transfer
Network Architecture
Performance Tes;ng &
Measurement
Data Transfer Node • High performance • Configured specifically
for data transfer • Proper tools
Science DMZ • Dedicated network
loca;on for high-‐speed data resources
• Appropriate security • Easy to deploy -‐ no need
to redesign the whole network
perfSONAR • Enables fault isola;on • Verify correct opera;on • Widely deployed in ESnet
and other networks, as well as sites and facili;es
The Science DMZ Design Pakern
9 – ESnet Science Engagement ([email protected]) - 06.07.16 © 2015, Energy Sciences Network
Abstract or Prototype Deployment
• Add-‐on to exis;ng network infrastructure – All that is required is a port on the border router – Small footprint, pre-‐produc;on commitment
• Easy to experiment with components and technologies – DTN prototyping – perfSONAR tes;ng
• Limited scope makes security policy excep;ons easy – Only allow traffic from partners – Add-‐on to produc;on infrastructure – lower risk than rebuilding exis;ng infrastructure
10 – ESnet Science Engagement ([email protected]) - 06.07.16 © 2015, Energy Sciences Network
Science DMZ Design PaLern (Abstract)
10GE
10GE
10GE
10GE
10G
Border Router
WAN
Science DMZSwitch/Router
Enterprise Border Router/Firewall
Site / CampusLAN
High performanceData Transfer Node
with high-speed storage
Per-service security policy control points
Clean, High-bandwidth
WAN path
Site / Campus access to Science
DMZ resources
perfSONAR
perfSONAR
perfSONAR
11 – ESnet Science Engagement ([email protected]) - 06.07.16 © 2015, Energy Sciences Network
Local And Wide Area Data Flows
10GE
10GE
10GE
10GE
10G
Border Router
WAN
Science DMZSwitch/Router
Enterprise Border Router/Firewall
Site / CampusLAN
High performanceData Transfer Node
with high-speed storage
Per-service security policy control points
Clean, High-bandwidth
WAN path
Site / Campus access to Science
DMZ resources
perfSONAR
perfSONAR
High Latency WAN Path
Low Latency LAN Path
perfSONAR
12 – ESnet Science Engagement ([email protected]) - 06.07.16 © 2015, Energy Sciences Network
Modular Architecture – Mul,ple Science DMZs
Dark Fiber
DarkFiber
10GE
DarkFiber
10GE
10G
Border Router
WAN
Science DMZSwitch/Routers
Enterprise Border Router/Firewall
Site / CampusLAN
Project A DTN(building A)
Per-project securitypolicy
perfSONAR
perfSONAR
Facility B DTN(building B)
Cluster DTN(building C)
perfSONARperfSONAR
Cluster(building C)
13 – ESnet Science Engagement ([email protected]) - 06.07.16 © 2015, Energy Sciences Network
Supercomputer Center Deployment
• High-‐performance networking is assumed in this environment – Data flows between systems, between systems and storage, wide area, etc. – Global filesystem omen ;es resources together
• Por;ons of this may not run over Ethernet (e.g. IB) • Implica;ons for Data Transfer Nodes
• “Science DMZ” may not look like a discrete en;ty here – By the ;me you get through interconnec;ng all the resources, you end up with most of the network in the Science DMZ
– This is as it should be – the point is appropriate deployment of tools, configura;on, policy control, etc.
• Office networks can look like an amerthought, but they aren’t – Deployed with appropriate security controls – Office infrastructure need not be sized for science traffic
14 – ESnet Science Engagement ([email protected]) - 06.07.16 © 2015, Energy Sciences Network
HPC Center
© 2014, Energy Sciences Network 15 – ESnet Science Engagement ([email protected]) - 06.07.16
Routed
Border Router
WAN
Core Switch/Router
Firewall
Offices
perfSONAR
perfSONAR
perfSONAR
Supercomputer
Parallel Filesystem
Front endswitch
Data Transfer Nodes
Front endswitch
HPC Center Data Path
© 2014, Energy Sciences Network 16 – ESnet Science Engagement ([email protected]) - 06.07.16
Routed
Border Router
WAN
Core Switch/Router
Firewall
Offices
perfSONAR
perfSONAR
perfSONAR
Supercomputer
Parallel Filesystem
Front endswitch
Data Transfer Nodes
Front endswitch
High Latency WAN Path
Low Latency LAN Path
Common Threads
• Two common threads exist in all these examples
• Accommoda;on of TCP – Wide area por;on of data transfers traverses purpose-‐built path – High performance devices that don’t drop packets
• Ability to test and verify – When problems arise (and they always will), they can be solved if the infrastructure is built correctly
– Small device count makes it easier to find issues – Mul;ple test and measurement hosts provide mul;ple views of the data path • perfSONAR nodes at the site and in the WAN • perfSONAR nodes at the remote site
17 – ESnet Science Engagement ([email protected]) - 06.07.16 © 2015, Energy Sciences Network
But What If I Don’t Use TCP? • Some sites use non-‐TCP tools/protocols
– Open source (e.g. UDT) – Commercial (e.g. Aspera)
• Does this mean we don’t need a Science DMZ? – The short answer is no…a Science DMZ is s;ll very valuable – There are many different reasons
• Tension between security and performance • Offload bandwidth hogs from enterprise network • Cost savings – consolidate high-‐performance services, reduce device count • Flexibility of provisioning, policy applica;on, enforcement • Flexibility of technology adop;on
• Flexibility offered by Science DMZ is cri;cal – Decouple enterprise network (stability is key) from science infrastructure – How fast can you adapt? How fast must you adapt?
18 – ESnet Science Engagement ([email protected]) - 06.07.16 © 2015, Energy Sciences Network
Overview
• Science DMZ Mo;va;on and Introduc;on
• Science DMZ Architecture
• Science DMZ Security
• Wrap Up
19 – ESnet Science Engagement ([email protected]) - 06.07.16 © 2015, Energy Sciences Network
• Goal – disentangle security policy and enforcement for science flows from security for business systems
• Ra;onale – Science data traffic is simple from a security perspec;ve – Narrow applica;on set on Science DMZ
• Data transfer, data streaming packages • No printers, document readers, web browsers, building control systems, financial databases, staff desktops, etc.
– Security controls that are typically implemented to protect business resources omen cause performance problems
• Separa;on allows each to be op;mized
Science DMZ Security
20 – ESnet Science Engagement ([email protected]) - 06.07.16 © 2015, Energy Sciences Network
Performance Is A Core Requirement
• Core informa;on security principles – Confiden;ality, Integrity, Availability (CIA) – Omen, CIA and risk mi;ga;on result in poor performance
• In data-‐intensive science, performance is an addi;onal core mission requirement: CIA à PICA – CIA principles are important, but if performance is compromised the science mission fails
– Not about “how much” security you have, but how the security is implemented
– Need a way to appropriately secure systems without performance compromises
21 – ESnet Science Engagement ([email protected]) - 06.07.16 © 2015, Energy Sciences Network
Placement Outside the Firewall
• The Science DMZ resources are placed outside the enterprise firewall for performance reasons – The meaning of this is specific – Science DMZ traffic does not traverse the firewall data plane
– Packet filtering is great – just don’t do it with an enterprise firewall
• Lots of heartburn over this, especially from the perspec;ve of a conven;onal firewall manager – Lots of organiza;onal policy direc;ves manda;ng firewalls – Firewalls are designed to protect converged enterprise networks – Why would you put cri;cal assets outside the firewall???
• The answer is that enterprise firewalls are typically a poor fit for high-‐performance science applica;ons
22 – ESnet Science Engagement ([email protected]) - 06.07.16 © 2015, Energy Sciences Network
Typical Firewall Internals
• Typical firewalls are composed of a set of processors which inspect traffic in parallel – Traffic distributed among processors such that all traffic for a par;cular connec;on goes to the same processor
– Simplifies state management – Paralleliza;on scales deep analysis
• Excellent fit for enterprise traffic profile – High connec;on count, low per-‐connec;on data rate – Complex protocols with embedded threats
• Each processor is a frac;on of firewall link speed – Significant limita;on for data-‐intensive science applica;ons – Overload causes packet loss – performance crashes
23 – ESnet Science Engagement ([email protected]) - 06.07.16 © 2015, Energy Sciences Network
Firewall Capabili,es and Science Traffic
• Commercial firewalls have a lot of sophis;ca;on in an enterprise serng – Applica;on layer protocol analysis (HTTP, POP, MSRPC, etc.) – Built-‐in VPN servers – User awareness
• Data-‐intensive science flows typically don’t match this profile – Common case – data on filesystem A needs to be on filesystem Z
• Data transfer tool verifies creden;als over an encrypted channel • Then open a socket or set of sockets, and send data un;l done (1TB, 10TB, 100TB, …)
– One workflow can use 10% to 50% or more of a 10G network link
• Do we have to use a commercial firewall?
24 – ESnet Science Engagement ([email protected]) - 06.07.16 © 2015, Energy Sciences Network
Firewalls As Access Lists
• When you ask a firewall administrator to allow data transfers through the firewall, what do they ask for? – IP address of your host – IP address of the remote host – Port range – That looks like an ACL to me!
• No special config for advanced protocol analysis – just address/port
• Router ACLs are beker than firewalls at address/port filtering – ACL capabili;es are typically built into the router – Router ACLs typically do not drop traffic permiked by policy
25 – ESnet Science Engagement ([email protected]) - 06.07.16 © 2015, Energy Sciences Network
Security Without Enterprise Firewalls
• Data intensive science traffic interacts poorly with enterprise firewalls
• Does this mean we ignore security? NO! – We must protect our systems – We just need to find a way to do security that does not prevent us from gerng the science done
• Key point – security policies and mechanisms that protect the Science DMZ should be implemented so that they do not compromise performance
• Traffic permiked by policy should not experience performance impact as a result of the applica;on of policy
26 – ESnet Science Engagement ([email protected]) - 06.07.16 © 2015, Energy Sciences Network
Systems View Of Science Infrastructure
• Security is a component, not a gatekeeper
• Think about the workflows • Think about the interfaces to data (tools, applica;ons)
– How do collaborators access data? – How could they access data if the architecture were different?
• Think about costs/benefits – What is a new cancer breakthrough worth? – $30k for a few DTNs – what is that in context?
• Think about risks – What risks do specific technologies mi;gate? – What are opportunity costs of poor performance?
27 – ESnet Science Engagement ([email protected]) - 06.07.16 © 2015, Energy Sciences Network
Overview
• Science DMZ Mo;va;on and Introduc;on
• Science DMZ Architecture
• Science DMZ Security
• Wrap Up
28 – ESnet Science Engagement ([email protected]) - 06.07.16 © 2015, Energy Sciences Network
Context: Science DMZ Adop,on • DOE Na;onal Laboratories
– Both large and small sites – HPC centers, LHC sites, experimental facili;es
• NSF CC-‐NIE and CC*IIE programs leverage Science DMZ – $40M and coun;ng (CC*DNI awards coming soon, es;mate addi;onal $18M to $20M) – Significant investments across the US university complex, ~130 awards – Big shoutout to Kevin Thompson and the NSF – these programs are cri;cally important
• Na;onal Ins;tutes of Health – 100G network infrastructure refresh
• US Department of Agriculture – Agricultural Research Service is building a new science network based on the Science DMZ model – hkps://www.xo.gov/index?s=opportunity&mode=form&tab=core&id=a7f291f4216b5a24c1177a5684e1809b
• Other US agencies looking at Science DMZ model – NASA – NOAA
• Australian Research Data Storage Infrastructure (RDSI) – Science DMZs at major sites, connected by a high speed network – hkps://www.rdsi.edu.au/dashnet – hkps://www.rdsi.edu.au/dashnet-‐deployment-‐rdsi-‐nodes-‐begins
29 – ESnet Science Engagement ([email protected]) - 06.07.16
© 2015, Energy Sciences Network
Strategic Impacts
• What does this mean? – We are in the midst of a significant cyberinfrastructure upgrade – Enterprise networks need not be unduly perturbed J
• Significantly enhanced capabili;es compared to 3 years ago – Terabyte-‐scale data movement is much easier – Petabyte-‐scale data movement possible outside the LHC experiments – Widely-‐deployed tools are much beker (e.g. Globus)
• Metcalfe’s Law of Network U;lity – Value propor;onal to the square of the number of DMZs? n log(n)? – Cyberinfrastructure value increases as we all upgrade
30 – ESnet Science Engagement ([email protected]) - 06.07.16 © 2015, Energy Sciences Network
Why Build A Science DMZ?
• Data set scale – Detector output increasing
• 1Hz à 10Hz à 100Hz à 1kHz … à 1MHz – HPC scale increasing
• Increased model resolu;on à increased data size • Increased HPC capability means addi;onal problems can now be solved
– Sequencers, Mass Spectrometers, … • Data placement
– Move compute to the data? – Sure, if you can…otherwise you need to move the data
• Without a Science DMZ, this stuff is hard – Can you assume nobody at your ins;tu;on will do this kind of work? – If this kind of work can’t be done, what does that mean in 5 years?
31 – ESnet Science Engagement ([email protected]) - 06.07.16 © 2015, Energy Sciences Network
• The Science DMZ design pakern provides a flexible model for suppor;ng high-‐performance data transfers and workflows
• Key elements: – Accommoda;on of TCP
• Sufficient bandwidth to avoid conges;on • Loss-‐free IP service
– Loca;on – near the site perimeter if possible – Test and measurement – Dedicated systems – Appropriate security
• Science DMZ gives flexibility, scaling, incremental provisioning for advanced services
Wrapup
32 – ESnet Science Engagement ([email protected]) - 06.07.16 © 2015, Energy Sciences Network
Links
– ESnet fasterdata knowledge base • hkp://fasterdata.es.net/
– Science DMZ paper • hkp://www.es.net/assets/pubs_presos/sc13sciDMZ-‐final.pdf
– Science DMZ email list • hkps://gab.es.net/mailman/lis;nfo/sciencedmz
– perfSONAR • hkp://fasterdata.es.net/performance-‐tes;ng/perfsonar/ • hkp://www.perfsonar.net
33 – ESnet Science Engagement ([email protected]) - 06.07.16 © 2015, Energy Sciences Network
Thanks!
Energy Sciences Network (ESnet) Lawrence Berkeley Na;onal Laboratory
hkp://fasterdata.es.net/
hkp://my.es.net/
hkp://www.es.net/
Extra Slides – DTN Cluster for HPC Cluster
35 – ESnet Science Engagement ([email protected]) - 06.07.16 © 2015, Energy Sciences Network
HPC/Cluster Environment
• Common cluster architectural elements – Head/Login nodes
• Primary user access • SSH typically required for access (for security reasons) • Job submission tools • Small-‐scale test jobs
– Compute nodes • Run user jobs • Typically not accessible from outside
– Central filesystem • Input data sets • Results of simula;on/analysis • Available on compute nodes, login nodes, etc.
06.07.16 36
Abstract Cluster
10GE10GE
Border Router
WAN
Firewall
Enterprise
perfSONAR
perfSONAR
10GE10GE
Filesystem
HEADCluster
Head/LoginNodes
Cluster compute nodes
HEAD
06.07.16 37
Limita,ons of Abstract Design – no DTNs
• Abstract cluster has limited data capabili;es
• All data transfers must traverse Head/Login nodes – Firewall in the path – Configura;on for data transfer tools conflated with cluster configura;on – User interac;ve jobs keep CPUs busy
• Solu;on – add DTNs – Connect DTNs to Science DMZ – Mount central filesystem on DTNs – Only permit data transfer tools on DTNs
06.07.16 38
Abstract Cluster With Science DMZ Data Access
10GE10GE
10GE10GE
10GE
10GE
Border Router
WAN
Science DMZSwitch/Router
Firewall
Enterprise
perfSONAR
perfSONAR
10GE10GE
10GE
10GE
10GE10GE
DTN
DTN
Filesystem
HEAD
“Sealed” DTNs(Globus only, no
shell access)
ClusterHead/Login
Nodes
DTN
DTN
Cluster compute nodes
HEAD
perfSONAR
06.07.16 39
Cluster Data Paths
10GE10GE
10GE10GE
10GE
10GE
Border Router
WAN
Science DMZSwitch/Router
Firewall
Enterprise
perfSONAR
perfSONAR
10GE10GE
10GE
10GE
10GE10GE
DTN
DTN
Filesystem
HEAD
“Sealed” DTNs(Globus only, no
shell access)
ClusterHead/Login
Nodes
DTN
DTN
Cluster compute nodes
HEAD
perfSONAR
Data Transfer Path
User Login/Shell Access Path
Compute Data Access Path
06.07.16 40
Cluster Security Controls
10GE10GE
10GE10GE
10GE
10GE
Border Router
WAN
Science DMZSwitch/Router
Firewall
Enterprise
perfSONAR
perfSONAR
10GE10GE
10GE
10GE
10GE10GE
DTN
DTN
Filesystem
HEAD
“Sealed” DTNs(Globus only, no
shell access)
ClusterHead/Login
Nodes
DTN
DTN
Cluster compute nodes
DTN Security Controls
HEAD
Filesystem Security Controls
perfSONAR
06.07.16 41
Cluster With DTNs – Full Picture
10GE10GE
10GE10GE
10GE
10GE
Border Router
WAN
Science DMZSwitch/Router
Firewall
Enterprise
perfSONAR
perfSONAR
10GE10GE
10GE
10GE
10GE10GE
DTN
DTN
Filesystem
HEAD
“Sealed” DTNs(Globus only, no
shell access)
ClusterHead/Login
Nodes
DTN
DTN
Cluster compute nodes
DTN Security Controls
HEAD
Filesystem Security Controls
perfSONAR
Data Transfer Path
User Login/Shell Access Path
Compute Data Access Path
06.07.16 42
Let’s Talk About Risk
• What part of the cluster poses the greatest security risk? • Greatest risk: Head/Login nodes
– Users have shell access (via SSH – the firewall can’t see!) – Users have compiler access, can build/run arbitrary code – Programma;c access to filesystem
• Second greatest risk: Compute Nodes – Users can run arbitrary code – Programma;c access to filesystem
• Lowest risk: DTNs – No user shell access – No user programma;c filesystem access – Data transfer applica;on (e.g. Globus) is the only method of interac;on
06.07.16 43
DTN Cluster Detail
10GE10GE
10GE10GE
10GE
10GE
Border Router
WAN
Science DMZSwitch/Router
Firewall
Enterprise
perfSONAR
perfSONAR
10GE10GE
10GE
10GE
10GE10GE
DTN
DTN
Filesystem
HEAD
“Sealed” DTNs(Globus only, no
shell access)
ClusterHead/Login
Nodes
DTN
DTN
Cluster compute nodes
HEAD
perfSONAR
Configure as DTN Cluster
06.07.16 44
DTN Cluster Design
• Configure all four DTNs as a single Globus endpoint – Globus has docs on how to do this – hkps://support.globus.org/entries/71011547-‐How-‐do-‐I-‐add-‐mul;ple-‐I-‐O-‐nodes-‐to-‐a-‐Globus-‐endpoint-‐
• Recent op;ons for increased performance – Use addi;onal parallel connec;ons – Distribute transfers across mul;ple DTNs (Globus I/O Nodes) – Cri;cal – only do this when all DTNs in the endpoint mount the same shared filesystem
• Use the Globus CLI command endpoint-modify – Use the -‐-‐network-‐use op;on – Adjusts concurrency and parallelism – More info at globus.org (hkp://dev.globus.org/cli/reference/endpoint-‐modify/)
06.07.16 45
Extra Slides – Output Queue Discussion
46 – ESnet Science Engagement ([email protected]) - 06.07.16 © 2015, Energy Sciences Network
Mul,ple Ingress Flows, Common Egress
Background traffic or
competing bursts
DTN traffic with wire-speed
bursts
10GE
10GE
10GE
Hosts will typically send packets at the speed of their interface (1G, 10G, etc.)
• Instantaneous rate, not average rate • If TCP has window available and data to send,
host sends un;l there is either no data or no window
Hosts moving big data (e.g. DTNs) can send large bursts of back-‐to-‐back packets
• This is true even if the average rate as measured over seconds is slower (e.g. 4Gbps)
• On microsecond ;me scales, there is omen conges;on
• Router or switch must queue packets or drop them
47 – ESnet Science Engagement ([email protected]) - 06.07.16
© 2015, Energy Sciences Network
Router and Switch Output Queues
• Interface output queue allows the router or switch to avoid causing packet loss in cases of momentary conges;on
• In network devices, queue depth (or ‘buffer’) is omen a func;on of cost – Cheap, fixed-‐config LAN switches (especially in the 10G space) typically have inadequate buffering. Imagine a 10G ‘data center’ switch as the guilty party
– Cut-‐through or low-‐latency Ethernet switches typically have inadequate buffering (the whole point is to avoid queuing!)
• Expensive, chassis-‐based devices are more likely to have deep enough queues – Juniper MX and Alcatel-‐Lucent 7750 used in ESnet backbone – Other vendors make such devices as well -‐ details are important – Thx to Jim: hkp://people.ucsc.edu/~warner/buffer.html – This expense is one driver for the Science DMZ architecture – only deploy the expensive features where necessary
48 – ESnet Science Engagement ([email protected]) - 06.07.16 © 2015, Energy Sciences Network
Output Queue Drops – Common Loca,ons
10GE
1GE
10GE
1GE
10GE
1GE1GE
1GE
10GE
Site Border RouterSite Core Switch/Router
32+ cluster nodes
Wiring closet switch
Common locations of output queue drops for traffic
outbound toward the WAN
WAN
Department Core Switch
1GE1GE
1GE
WorkstationsDepartment
cluster switch
Department uplink to site core constrained by
budget or legacy equipment
Cluster data
transfer node
Common location of output queue drops for traffic inbound
from the WAN
Inbound data path
Outbound data path
Outbound data path
49 – ESnet Science Engagement ([email protected]) - 06.07.16 © 2015, Energy Sciences Network
Extra Slides – Globus Security Map
06.07.16 50
Security Footprint of a Globus Transfer
Amazon AWS
100GE
10GE10GE
100GE
10GE
10GE100GE
DATA
TCP ports50000-51000
Lab1 Science DMZ
Lab1 Border Router
ESnet 100GEESnet Router
Lab2 Border Router
Lab2 Science DMZ
Lab1 DTN
DTN DTN
OrchestrationOrchestration
Lab2 DTN
ESnet Router
Lab1 DTN security
filters
Lab2 DTN security
filters
TCP ports 443,2811, 7512
TCP ports 443,2811, 7512
Logical data path
Physical data path
Logical control path
Physical control path
Lab1 DTN security filters Lab2 DTN security filters
06.07.16 51
Security Footprint of a Globus DTN
06.07.16 52
10GE
Amazon AWS
100GE
10GE
10GE
100GE
DATA
TCP ports50000-51000 Science DMZ
Site / Campus Border Router
World
DTN
DTN
Orchestration
Remote DTNs
DTN securityfilters
TCP ports 443,2811, 7512
DTN
DATA
Local DTN
Logical data path
Physical data path
Logical control path
Physical control path
Extra Slides – Firewall Internals
53 – ESnet Science Engagement ([email protected]) - 06.07.16 © 2015, Energy Sciences Network
Thought Experiment
• We’re going to do a thought experiment
• Consider a network between three buildings – A, B, and C • This is supposedly a 10Gbps network end to end (look at the links on the buildings)
• Building A houses the border router – not much goes on there except the external connec;vity
• Lots of work happens in building B – so much that the processing is done with mul;ple processors to spread the load in an affordable way, and results are aggregated amer
• Building C is where we branch out to other buildings • Every link between buildings is 10Gbps – this is a 10Gbps network, right???
54 – ESnet Science Engagement ([email protected]) - 06.07.16 © 2015, Energy Sciences Network
No,onal 10G Network Between Buildings
WAN
perfSONAR Building A
10GE 10GE
Building B
Building C
1G1G
1G1G
1G 1G1G
1G
1G1G
1G1G1G 1G1G 1G1G 1G1G
1G
10GE
Building Layout
To O
ther
Bui
ldin
gs
10GE
10GE
10GE
55 – ESnet Science Engagement ([email protected]) - 06.07.16 © 2015, Energy Sciences Network
Clearly Not A 10Gbps Network
• If you look at the inside of Building B, it is obvious from a network engineering perspec;ve that this is not a 10Gbps network – Clearly the maximum per-‐flow data rate is 1Gbps, not 10Gbps – However, if you convert the buildings into network elements while keeping their internals intact, you get routers and firewalls
– What firewall did the organiza;on buy? What’s inside it? – Those likle 1G “switches” are firewall processors
• This parallel firewall architecture has been in use for years – Slower processors are cheaper – Typically fine for a commodity traffic load – Therefore, this design is cost compe;;ve and common
56 – ESnet Science Engagement ([email protected]) - 06.07.16 © 2015, Energy Sciences Network
No,onal 10G Network Between Devices
WAN
perfSONAR Border Router
10GE 10GE
Firewall
Internal Router
1G1G
1G1G
1G 1G1G
1G
1G1G
1G1G1G 1G1G 1G1G 1G1G
1G
10GE
Device Layout
To O
ther
Bui
ldin
gs
10GE
10GE
10GE
57 – ESnet Science Engagement ([email protected]) - 06.07.16 © 2015, Energy Sciences Network
No,onal Network Logical Diagram
10GE
10GE
10GE
10GE
10GE10GE
Border Router
WAN
Internal Router
Border Firewall
perfSONAR
58 – ESnet Science Engagement ([email protected]) - 06.07.16 © 2015, Energy Sciences Network
06.07.16 59
What Is A Firewall? • Marketplace view
– Specific security appliance, with “Firewall” printed on the side – Lots of protocol awareness, intelligence – Applica;on awareness – User awareness (VPN, specific access controls, etc.) – Designed for large concurrent user count, low per-‐user bandwidth (enterprise traffic)
• IT Organiza;on view – “Firewall” appliance, purchased from the commercial marketplace – The place in the network where security policy gets applied – Owned by the security group, not by the networking group – Primary risk mi;ga;on mechanism
• NIST view (Publica;on 800-‐41 rev. 1, Sep. 2009) – “Firewalls are devices or programs that control the flow of network traffic between networks or hosts that employ differing security postures”
– This is very general, and does not match marketplace view or IT org. view
60 – ESnet Science Engagement ([email protected]) - 06.07.16 © 2015, Energy Sciences Network
NIST Sees Two Firewalls, IT Shop Sees One
10GE
10GE
10GE
10GE
10G
Border Router
WAN
Science DMZSwitch/Router
Enterprise Border Router/Firewall
Site / CampusLAN
High performanceData Transfer Node
with high-speed storage
Per-service security policy control points
Clean, High-bandwidth
WAN path
Site / Campus access to Science
DMZ resources
perfSONAR
perfSONAR
perfSONAR
Stateful
Stateless
61 – ESnet Science Engagement ([email protected]) - 06.07.16 © 2015, Energy Sciences Network
Stateful Inspec,on For Science DMZ Traffic?
• Science DMZ traffic profile – Small number of connec;ons or flows – Large per-‐connec;on data rate (Gigabit scale or higher) – Large per-‐connec;on data volume (Terabyte scale or higher)
• Stateless firewall – Address/port filtering (which systems use which service) – TCP connec;on ini;a;on direc;on (ACK flag)
• Stateful firewall adds – TCP sequence number tracking (but Linux stack is as good or beker compared to firewall TCP mi;ga;ons)
– Protocol/app analysis (but not for the apps used in DMZ) – DoS protec;on (but the Science DMZ assets are filtered already)
62 – ESnet Science Engagement ([email protected]) - 06.07.16 © 2015, Energy Sciences Network