Computing at FNAL" Hans Wenzel Fermilab

PASTA ReviewTechnology for the LHC Era Michael Ernst, FNAL Large Scale Cluster Computing Workshop October 21, 200221 October 2002Michael Ernst, FNALErnst@fnal.gov 089225

Approach to Pasta IIITechnology Review of what was expected from Pasta II and what might be expected in 2005 and beyond.Understand technology drivers which might be market and business driven. In particular the suppliers of basic technologies have undergone in many cases major business changes with divestment, mergers and acquisitions.Try to translate where possible into costs that will enable us to predict how things are evolving.Try to extract emerging best practices and use case studies wherever possible.Involve a wider number of people than CERN in major institutions in at least Europe and the US.Michael Ernst, FNAL Large Scale Cluster Computing Workshop October 21, 2002ParticipantsA: Semiconductor TechnologyIan Fisk (UCSD) Alessandro Machioro (CERN) Don Petravik (Fermilab)B:Secondary Storage Gordon Lee (CERN) Fabien Collin (CERN) Alberto Pace(CERN) C:Mass StorageCharles Curran (CERN) Jean-Philippe Baud(CERN)D:Networking TechnologiesHarvey Newman (Caltech) Olivier Martin (CERN) Simon Leinen(Switch)E:Data Management TechnologiesAndrei Maslennikov (Caspur) David Foster (CERN)F:Storage Management SolutionsMichael Ernst (Fermilab) Nick Sinanis (CERN/CMS) Martin Gasthuber (DESY)G:High Performance Computing SolutionsBernd Panzer (CERN) Ben Segal (CERN) Arie Van Praag(CERN)

Michael Ernst, FNAL Large Scale Cluster Computing Workshop October 21, 2002Current StatusMost reports in the final stages.Networking is the last to complete.Some cosmetic treatments needed.Draft Reports can be found at:

http://david.web.cern.ch/david/pasta/pasta2002.htmMichael Ernst, FNAL Large Scale Cluster Computing Workshop October 21, 2002Year

1997

1999

2001

2003

2006

2009

2012

Technology requirements

Dram pitch (um)

.25

.18

.15

.13

.10

.07

.05

uP channel length

.20

.14

.12

.10

0.7

.05

.035

Tox equivalent (nm)

4-5

3-4

2-3

2-3

1.5-2

Serial ATA (150MB/s 600 MB/s)Serial ATA is expected to dominate the commodity disk connectivity market by end 2003. Fiber channel products still expensive.DVD solutions still 2-3x as expensive as disks. No industry experience managing large DVD libraries. Michael Ernst, FNAL Large Scale Cluster Computing Workshop October 21, 2002

Tape Storage Technology Michael Ernst, FNAL Large Scale Cluster Computing Workshop October 21, 2002Tape Drive Target ApplicationsPerformance NeedsTape CapacityTape AccessAccessThroughputCapacity Application Segments Batch Processing

Tape Transaction Processing

Hierarchical Storage Management

Active Archive (Check, Medical)

Disk Extension

Scientific / Extremely Large Files

Backup and Restore

Deep ArchiveMichael Ernst, FNAL Large Scale Cluster Computing Workshop October 21, 2002Tape Path and Cartridge TypesDual HubInternal tape pathSTK 9840, IBM 3570, QIC

Single HubExternal tape pathDLT, LTO, 3490, 4490,9490, 3590, SD-3, 9940

Dual Hub - CassetteExternal tape pathAIT, Mammoth and other 4/8mm Helical scan

Michael Ernst, FNAL Large Scale Cluster Computing Workshop October 21, 2002

LTO Ultrium RoadmapGA 18-24 Months after Gen1GA 18-24 Months after Gen2GA Start of Q3-00Media SwapMedia SwapMedia SwapMichael Ernst, FNAL Large Scale Cluster Computing Workshop October 21, 2002

Recent Tape Drives (STK)99409840Michael Ernst, FNAL Large Scale Cluster Computing Workshop October 21, 2002Track Layout (STK 9840/9940)

Michael Ernst, FNAL Large Scale Cluster Computing Workshop October 21, 2002Ferrofluid/MFM Images of Data Tracks and Amplitude Based Servo Tracks (STK 9840/9940) Drives

6 servo tracks with bursts of servo signal. Servo head positions between tracks. 5 positions Direction of tape54321Michael Ernst, FNAL Large Scale Cluster Computing Workshop October 21, 20029940 Operational Principle

Single HubExternal tape pathMichael Ernst, FNAL Large Scale Cluster Computing Workshop October 21, 2002Data Access ComparisonData Access: Time from cartridge insertion in drive to first byte of data - timed to mid-point and does not include robotic time Access Seconds9840 12 Timberline 25 CapacitySeconds9940 59 RedWood 66 Magstar 3590 64Magstar 3590E 90+IBM LTO 3580 110 Michael Ernst, FNAL Large Scale Cluster Computing Workshop October 21, 2002

9940 Load TimeMichael Ernst, FNAL Large Scale Cluster Computing Workshop October 21, 2002FuturesFutures200120012003200320002000 GA=1999 20 GB (unc.)10 MB/secAccess = 8.0 secMP MediaSCSI, Fibre Ch.9840A200220029840 and 9940 Tape Drive Roadmap9840BQ301 volume ship20 GB (unc.)19 MB/secAccess = 8.0 secMP MediaSCSI, Fibre Ch.TCP/IP Service Port200370 GB (unc.)30 MB/secAccess = 8.0 secMP MediaESCON, Fibre Ch.TCP/IP Service Port9840CFuture GenerationAccess Drives100 - 300 GB (unc.)45 - 90 MB/sec9940AGA=200060 GB (unc.)10 MB/secMP MediaSCSI, Fibre Ch.1H029940B200 GB (unc.)30 MB/secMP MediaFibre ChannelTCP/IP Service PortAccessCapacityFuture GenerationCapacity Drives300 GB - 1 TB (unc.)60 -120 MB/secMichael Ernst, FNAL Large Scale Cluster Computing Workshop October 21, 2002Tapes - 1Recent tape drive technology (9840, 9940A/B, LTO) is installed at CERN.Current Installation are 10 STK silos capable of taking 800 new format tape drives. Today tape performance is 15MB/sec so theoretical aggregate is 12GB/sec (no way in reality!)Cartridge capacities expected to increase to 1TB before LHC startup but its market demand and not technical limitations driving it.Using tapes as a random access device is no longer a viable optionNeed to consider a much larger, persistent disk cache for LHC reducing tape activity for analysis.Michael Ernst, FNAL Large Scale Cluster Computing Workshop October 21, 2002Random Access Sequential Access FileTransfer ProductionPersonalAnalysis Disk CacheHSM(Hierarchical Storage Manager)

Experiment-specificMichael Ernst, FNAL Large Scale Cluster Computing Workshop October 21, 2002Tapes - 2Current costs are about $33/slot for a tape in the Powderhorn robot.Current tape cartridge (9940A/B, 60GB) costs $86 with a slow decrease over time.Media dominates the overall cost and a move to higher capacity cartridges and tape units sometimes require a complete media change.Current storage costs 0.4-0.7 USD/GB in 2000 could drop to 0.2 USD/GB in 2005 but probably would require a complete media change.Conclusions: No major challenges for tapes for LHC startup but the architecture has to be such that random access is avoidedMichael Ernst, FNAL Large Scale Cluster Computing Workshop October 21, 2002

Interregional Connectivity is the key .NetworkingMichael Ernst, FNAL Large Scale Cluster Computing Workshop October 21, 2002Tier2 Center

Online SystemOffline Farm,CERN Computer CenterUS Center @ FNALFrance Center Italy Center UK Center InstituteInstituteInstituteInstitute~100 MBytes/sec~2.4 Gbits/sec~PBytes/secTier2 CenterTier2 CenterTier2 CenterTier 0 +1Tier 1Tier 3Tier2 Center Tier 2ExperimentCMS has adopted a distributed computing model to perform data analysis, event simulation, and event reconstruction in which two-thirds of the total computing resources are located at regional centers. The unprecedented size of the LHC collaborations and complexity of the computing task requires that new approaches be developed to allow physicists spread globally to efficiently participate. CMS as an example Michael Ernst, FNAL Large Scale Cluster Computing Workshop October 21, 2002 Transatlantic Net WG (HN, L. Price) Bandwidth Requirements [*]

[*] Installed BW. Maximum Link Occupancy 50% AssumedMichael Ernst, FNAL Large Scale Cluster Computing Workshop October 21, 2002Network Progress andIssues for Major Experiments Network backbones are advancing rapidly to the 10 Gbps rangeGbps end-to-end throughput data flows will be in production soon (in 1-2 years)Network advances are changing the view of the nets rolesThis is likely to have a profound impact on the experiments Computing Models, and bandwidth requirements Advanced integrated applications, such as Data Grids, rely onseamless transparent operation of our LANs and WANsWith reliable, quantifiable (monitored), high performanceNetworks need to be integral parts of the Grid(s) designNeed new paradigms of real network and system monitoring, and of new of managed global systems for HENP analysisThese are starting to be developed for LHCMichael Ernst, FNAL Large Scale Cluster Computing Workshop October 21, 2002Signs of the Times: UncertaintyBut No Change in Outlook Key Providers in BankruptcyKPNQwest, Teleglobe, Global Crossing, FLAG; Worldcom Switching to Others, Where Needed and PossibleE.g. T-Systems (Deutsche Telecom) for US-CERNStrong Telecom Market OutlookGood pricing from DTMCI/Worldcom network will continue to operate (?):20 M customers in US; UK academic & research networkAggressive plans by major and startup network equipment providers Strong Outlook in R&E Nets for Rapid Progress Abilene (US) Upgrade On Schedule; GEANT (Europe), and SuperSINET(Japan) Plans Continuing ESNet Backbone Upgrade: 2.5 Gbps Now; 10 Gbps in 2 Yrs. Regional Progress, and Visions; E.g. CALREN: 1 Gbps to Every Californian by 2010 Michael Ernst, FNAL Large Scale Cluster Computing Workshop October 21, 2002LHCnet Network : Late 2002Development and testsAbileneMRENESnetSTARTAPNASALinux PC forPerformance tests & MonitoringCERN -GenevaGEANTSwitchIN2P3WHOLinux PC forPerformance tests & MonitoringCaltech/DoE PoP StarLight Chicago 622 Mbps (Prod.)Cisco 7609CERNCisco 7609Caltech(DoE)Alcatel 7770DataTAG(CERN)Cisco 7606DataTAG(CERN)Juniper M10DataTAG(CERN)Cisco 7606Caltech(DoE)Juniper M10Caltech(DoE)Alcatel 7770DataTAG(CERN)2.5 Gbps (R&D)Optical Mux/DmuxAlcatel 1670Optical Mux/DmuxAlcatel 1670Michael Ernst, FNAL Large Scale Cluster Computing Workshop October 21, 2002The Rapid Pace of Network Technology Advances ContinuesWithin the Next One to Two Years 10 Gbps Ethernet on Switches and Servers; LAN/WAN integration at 10 Gbps 40 Gbps Wavelengths Being Shown HFR: 100 Mpps forwarding engines, 4 and more 10 Gbps ports per Slot; Terabit/sec backplanes etc. Broadband Wireless [Multiple 3G/4G alternatives]: the drive to defeat the last mile problem 802.11 ab, UWB, etc.

Michael Ernst, FNAL Large Scale Cluster Computing Workshop October 21, 2002EU-Solicited Project. CERN, PPARC (UK), Amsterdam (NL), and INFN (IT);and US (DOE/NSF: UIC, NWU and Caltech) partnersMain Aims: Ensure maximum interoperability between US and EU Grid ProjectsTransatlantic Testbed for advanced network research2.5 Gbps Wavelength Triangle 7/02 (10 Gbps Triangle in 2003)

FrRenaterDataTAG Project

NLSURFnet

GENEVAUKSuperJANET4ABILENEESNETCALRENItGARR-B

GEANTNewYorkSTAR-TAPSTARLIGHT Wave TriangleMichael Ernst, FNAL Large Scale Cluster Computing Workshop October 21, 2002HENP Major Links: Bandwidth Roadmap (Scenario) in Gbps

Michael Ernst, FNAL Large Scale Cluster Computing Workshop October 21, 2002HENP Lambda Grids:Fibers for PhysicsProblem: Extract Small Data Subsets of 1 to 100 Terabytes from 1 to 1000 Petabyte Data StoresSurvivability of the HENP Global Grid System, with hundreds of such transactions per day (circa 2007)requires that each transaction be completed in a relatively short time. Example: Take 800 secs to complete the transaction. Then Transaction Size (TB) Net Throughput (Gbps) 1 10 10 100 100 1000 (Capacity of Fiber Today)Summary: Providing Switching of 10 Gbps wavelengthswithin ~3 years; and Terabit Switching within ~6-10 yearswould enable Petascale Grids with Terabyte transactionswithin this decade, as required to fully realize the discovery potential of major HENP programs, as well as other data-intensive fields.Michael Ernst, FNAL Large Scale Cluster Computing Workshop October 21, 2002Time to recover from a single lossTCP reactivity: Due to the Basic Multiplicative-Decrease Additive-Increase Algorithm to Handle Packet LossTime to increase the throughput by 120 Mbit/s is larger than 6 min for a connection between Chicago and CERN.A single loss is disastrousA TCP connection reduces bandwidth use by half after a loss is detected (Multiplicative decrease)A TCP connection increases slowly its bandwidth use (Additive increase)TCP is much more sensitive to packet loss in WANs than in LANs

6 minNetwork Protocol IssuesMichael Ernst, FNAL Large Scale Cluster Computing Workshop October 21, 2002TCP ResponsivenessCaseCapacityRTT (ms) MSS (Byte)ResponsivenessTypical LAN in 198810 Mbps[ 2 ; 20 ] 1460[ 1.5 ms ; 154 ms ]Typical WAN in 19889.6 Kbps40 14600.006 secTypical LAN today100 Mbps5 (worst case)14600.096 secCurrent WAN link CERN Starlight622 Mbps12014606 minutesFuture WAN link CERN Starlight10 Gbit/s120146092 minutesFuture WAN link CERN Starlight10 Gbit/s1208960 (Jumbo Frame)15 minutesMichael Ernst, FNAL Large Scale Cluster Computing Workshop October 21, 2002National Research Networks in Japan Proposal: Two TransPacific 2.5 Gbps Wavelengths, and Japan-CERN Grid Testbed by ~2003 TokyoNagoyaInternetKyoto UICRKyoto-UNagoya UNIFSNIGKEKTohoku UIMSU-TokyoNAOU TokyoNII Hitot.NII ChibaIPWDM pathIP routerOXCISASMichael Ernst, FNAL Large Scale Cluster Computing Workshop October 21, 2002NetworkingMajor cost reductions have taken place in wide-area bandwidth costs.2.5 Gbps common for providers but not academic in 1999. Now, 10Gbps common for providers and 2.5Gbps common for academic.Wide area data migration/replication now feasible and affordable.Tests of multiple streams to the US running at the full capacity of 2Gbps were successful.Local Area Networking moving to 10 Gbps and this is expected to increase. First10Gbps NICs available for end systems.Michael Ernst, FNAL Large Scale Cluster Computing Workshop October 21, 2002Networking TrendsTransitioning from 10Gbit to 20-30 Gbit seems likely.MPLS (Multiprotocol Label Switching) has gained momentum. It provides secure VPN capability over public networks. A possibility for tier-1 center connectivity.Lambda networks based on dark fiber are also becoming very popular. It is a build-yourself network and may also be relevant for the grid and center connectivity.Michael Ernst, FNAL Large Scale Cluster Computing Workshop October 21, 2002Storage - ArchitecturePossibly the biggest challenge for LHCStorage architecture design (seamless integration from CPU caches to deep archive required)Data management. Currently very poor tools and facilities for managing data and storage systems.SAN vs. NAS debate still aliveSAN, scalable and high availability, but costly NAS, cheaper and easier to manageObject storage technologies appearingIntelligent storage system able to manage the objects it is storingAllowing light-weight Filesystems

Michael Ernst, FNAL Large Scale Cluster Computing Workshop October 21, 2002OSD IntelligenceStorage DeviceObject ManagerApplication File Manager

Meta OperationLAN/SANData TransferSecurityObject Storage Device ArchitectureMichael Ernst, FNAL Large Scale Cluster Computing Workshop October 21, 2002Storage ManagementVery little movement in the HSM space since the last PASTA report.HPSS still for large scale systemsA number of mid-range products (make tape look like a big disk) but limited scaling possibleHEP still a leader in tape and data managementCASTOR, Enstore, JASMineWill remain crucial technologies for LHC.Cluster file systems appearing (StorageTank, Lustre)Provide unlimited (PB) file system (e.g. through LAN, SAN)Scale to many 1000s of clients (CPU servers).Need to be interfaced to tertiary storage systems (e.g. Enstore)

Michael Ernst, FNAL Large Scale Cluster Computing Workshop October 21, 2002Storage - ConnectivityFiberChannel market growing at 36%/year from now to 2006 (Gartner). This is the current technology for SAN implementation.iSCSI or equivalent over Gigabit Ethernet is an alternative (and cheaper) but less performant implementation of SAN gaining in popularity.It is expected that GigE will become a popular transport for storage networks.InfiniBand (up to 30 Gbps) is a full-fledged network technology that could change the landscape of cluster architectures and has much, but varying, industry support. Broad adoption could drive costs down significantlyFIO (Compaq, IBM, HP) and NGIO (Intel, MS, Sun) merged to IBExpect bridges between IB and legacy Ethernet and FC netsUses IPv6Supports RDMA and multicast Michael Ernst, FNAL Large Scale Cluster Computing Workshop October 21, 2002Local FSNFSDAFSApplicationApplicationApplicationBuffersBuffersBuffersFS SwitchFS SwitchFileSystemSCSIDriverHBA DriverNFSTCP/IPNIC DriverBufferCacheBufferCachePacketBuffersNICHBANICVI NICDriverDAFSVIPLUSERKernelHardwareData Transfer OverheadMemoryDataControlMichael Ernst, FNAL Large Scale Cluster Computing Workshop October 21, 2002Storage Cost

Cost of managing storage and data are the predominate costsMichael Ernst, FNAL Large Scale Cluster Computing Workshop October 21, 2002Storage Scenario - Today

Michael Ernst, FNAL Large Scale Cluster Computing Workshop October 21, 2002Storage Scenario - Future

Michael Ernst, FNAL Large Scale Cluster Computing Workshop October 21, 2002Some Overall ConclusionsTape and Network trends match or exceed our initial needs.Need to continue to leverage economies of scale to drive down long term costs.CPU trends need to be carefully interpretedThe need for new performance measures are indicated. Change in the desktop market might effect the server strategy.Cost of manageability is an issue.Disk trends continue to make a large (multi PB) disk cache technically feasible, but .The true cost of such an object a bit unclear, given the issues of reliability, manageability and the disk fabric chosen (NAS/SAN, iSCSI/FC etc etc)File system access for a large disk cache (RFIO, DAFS, ) under investigation (urgent !)More architectural work is needed in the next 2 years for the processing and handling of LHC data.NAS/SAN models are converging, many options for system interconnects, new High Performance NAS products are (about to be) rolled out (Zambeel, Panasas, Maximum Throughput, Exanet etc) Michael Ernst, FNAL Large Scale Cluster Computing Workshop October 21, 2002 Sounds like we are in pretty good shape ..Michael Ernst, FNAL Large Scale Cluster Computing Workshop October 21, 2002 but lets be careful ...Michael Ernst, FNAL Large Scale Cluster Computing Workshop October 21, 2002PASTA has addressed issues exclusively on the Fabric level

It is likely that we will get the required technology (Processors, Memory, Secondary and Tertiary Storage Devices, Networking, Basic Storage Management) Missing: Solutions allowing truly distributed Computing on a Global Scale Will the Grid Projects meet our Expectations (in time) ?Michael Ernst, FNAL Large Scale Cluster Computing Workshop October 21, 2002Chart120.517.412.511.63.71.913.55.911.61.4

Disk Vendors Market Share in units - 1998

Sheet114.917.7822.65.64.672.23108.4121.1145.1Seagate20.5Quantum17.4IBM12.5Maxstore11.6Toshiba3.7Hitachi1.9Western Digital13.5Samsung5.9Fujitsu11.6Others1.437000412700204520032244220104254241446004262.23.158132031500.2030.1840.230.2750.3115.01818.24621.3424.7128.45109.65127.94148.34171.65197.275.636.236.816.636.460.3460.6570.9171.1021.2340.4930.0980.0460.0310.0286.6870.5820.0990.0340.04210.115.0821.4350.3440.1178.711.976.261.8220.4724.9611.28314.478.2013.110.2154.11411.9316.73511.4770.2280.2411.6759.23616.04100.1990.2641.1136.151

Sheet1000000000

2.5/3.0"5.25"3.5"Millions of unitsDisk Shipments by Form Factor - 1998

Sheet20000000000

Disk Vendors Market Share in units - 1998

Sheet300000000

Capacity per 3.5 inch platterGB per disk platterAreal Density Projected Growth

0000000000000000000000000000

1 - 1.92 - 2.93 - 4.95 - 6.97 - 9.910 - 19.9> 20units - millionsPC/Desktop Disks - Capacity Trends Sizes in GB

00000000000000000000

1.8 inch2.5 inch3.5 inch5,25 inchUnits - millionsDisk Shipments by Form Factor

000000000000000000000000000000000000000000000

Cartridges< 1 GB1 - 2 GB2 - 3 GB3 - 5 GB5 - 10 GB10 -20 GB20 - 40 GB> 40 GBRevenue US$ - billionDisk Drive Revenues by Drive Capacity

Chart2243497816

HDD Vendor Market Share in Units - 2001Desktop PC/ATA drivesQuantum / Maxtor36%

Sheet1Seagate20Quantum16IBM13Maxtor12Toshiba4Hitachi2Western Digital14Samsung6Fujitsu12Others1Seagate47Quantum/Maxtor9IBM22Fujitsu19Hitachi3Seagate24Quantum/Maxtor34IBM9Fujitsu7Sansung8Western Digital16

Sheet10000000000

HDD Vendor Market Share in Units - 1998

Sheet200000

HDD Vendor Market Share in Units - 2001Enterprise StorageQuantum / Maxtor9%

Sheet3000000

HDD Vendor Market Share in Units - 2001Desktop PC/ATA drivesQuantum / Maxtor36%

Chart147922193

HDD Vendor Market Share in Units - 2001Enterprise StorageQuantum / Maxtor9%

Sheet1Seagate20Quantum16IBM13Maxtor12Toshiba4Hitachi2Western Digital14Samsung6Fujitsu12Others1Seagate47Quantum/Maxtor9IBM22Fujitsu19Hitachi3Seagate24Quantum/Maxtor34IBM9Fujitsu7Sansung8Western Digital16

Sheet10000000000

HDD Vendor Market Share in Units - 1998

Sheet200000

HDD Vendor Market Share in Units - 2001Enterprise StorageQuantum / Maxtor9%

Sheet3000000

HDD Vendor Market Share in Units - 2001Desktop PC/ATA drivesQuantum / Maxtor36%

Sheet1Generation 1Generation 2Generation 3Generation 4Capacity100 GB200 GB400 GB800 GBTransfer Rate10-20 MB/s20-40 MB/s40-80 MB/s80-160 MB/sEnabling Technology????Number of Channels881616Recording MethodRLL 1,7PRMLPRMLPRMLMedia TypeMP2MPMPThin FilmTape Length580 m580 m800 m800 mNumber of Tracks3845127681024

Sheet2

Sheet3

Chart10.50.520.520.59412.5

9940 Load Time To First Data(Total: 58.5 Seconds)41s Locate2.5s Low Speed Position0.5s Load0.5s Un-slack0.5s One Wrap on Reel2s Thread0.5s Initialize2s Wrap Tape Path9s Cover Leader Block

Sheet10.5load0.5un-slack2thread0.51 wrap on mach reel2wrap tape path0.5initialization9cover leader block41Locate2.5low speed posn58.5Totaltotal sec

Sheet1000000000

TC40 Load Time To First Data(Total: 58.5 Seconds)9s Cover Leader Block2s Wrap Tape Path0.5s Initialize2s Thread0.5s One Wrap on Reel0.5s Un-slack0.5s Load2.5s Low Speed Position41s Locate

Sheet2

Sheet3

200120022003200420052006

CMS1002003006008002500

ATLAS501003006008002500

BaBar3006001100160023003000

CDF100300400200030006000

D040016002400320064008000

BTeV2040100200300500

DESY100180210240270300

CERN

BW155-310622250050001000020000

YearProductionExperimentalRemarks

20010.155 0.622-2.5SONET/SDH

20020.6222.5SONET/SDHDWDM; GigE Integ.

20032.510 DWDM; 1 + 10 GigEIntegration

2005102-4 X 10 ( Switch;( Provisioning

20072-4 X 10~10 X 10; 40 Gbps1st Gen. ( Grids

2009~10 X 10or 1-2 X 40 ~5 X 40 or~20-50 X 1040 Gbps (Switching

2011~5 X 40 or

~20 X 10~25 X 40 or ~100 X 10 2nd Gen ( GridsTerabit Networks

2013~Terabit~MultiTerabit~Fill One Fiber

Chart343.1085676.83749676.9012483.62785677.03208877.15428877.25268882.42515278.56898478.42561683.20769678.80190484.06904878.37620884.30739280.95095282.86209684.25716885.62226481.83685684.8961284.88576885.23635285.78266485.85755286.5120486.98097687.25773687.91410486.70230489.91384887.34567289.76596888.82750489.3485690.52990.0996489.61073692.32478490.57992892.85961691.5504493.52308891.41994494.61823292.85496893.63644895.26231295.0333293.05955296.52966496.51245695.75105694.02646497.24512896.89469697.7340497.67180898.57403297.88294498.72801698.81832899.88061698.47701699.83402488.57517676.76773676.84093677.37047277.92012879.08240877.46262479.7557679.61776879.23126480.49762481.27910480.68646481.37325681.9537281.37291283.1849283.58249681.74454485.2616484.50849684.45848884.95702485.63956884.16989686.34722486.41482487.13881688.15516887.75601687.79514487.88730488.1872888.89184890.41854488.92702489.78797691.80734489.33076893.28462490.762691.541292.14774493.32177691.27919295.08813692.48211294.19133694.55601693.448895.73028895.69382496.10337695.37787296.08416896.79652896.75753698.4587697.77948897.78820898.674298.4556898.9053699.01883298.962424100.06324899.80088101.41297699.995384104.402176100.187888101.254928102.193208101.921808103.707944102.131248104.017072104.699128102.328336105.175872103.936192104.217496107.968104104.170136106.513896105.756424106.698168106.154616106.860904107.69216106.602448110.412096107.987424106.642016109.733304109.793544107.986168111.085504110.837688110.001256108.123152111.555392113.171208112.14864110.357336111.505424112.317856112.245264114.674416113.543264114.805768113.592464111.732592115.0058114.041976115.200528115.220304115.626376116.047168117.620048115.553912115.49664117.357936117.09028115.71984117.6754118.471832120.182128117.530248118.578336117.338192120.673784120.64064119.777632118.112224121.350624122.929768118.371696121.525504123.370856119.49888123.14684122.998464119.044776125.642616122.834368120.500272127.206168120.25448127.032656122.53492127.150664122.534392127.861288123.460016127.707144122.540648129.733608121.379512131.651856121.640168129.331112127.440184127.992776129.066712125.268704129.485504130.91472125.796176130.50232131.519768126.337168131.26936130.949552128.67892129.989712133.312544130.721664130.633392131.868112132.385752133.621512129.368496133.322888132.016904131.658264134.881736134.695296132.63828135.44328132.117672134.112048135.508576135.77308134.758248135.9002135.519296133.368096137.645232136.312472137.756656137.823872134.113984137.975752135.379344138.20056136.671784138.358672140.381816140.610048138.57584138.827184136.9118140.908344138.772296142.52036139.387472139.015904139.05108140.862904142.596736142.293128141.661128143.119704140.107888142.78308141.261872142.464688145.039832142.985528141.999656144.884016145.03456143.90284142.86744143.85584145.31184147.962232142.370736145.850784144.583408147.921512144.408424146.021472146.295128148.974344146.41772145.47016149.962392147.492936145.047608148.97328150.627376146.849512150.289816148.132312146.985592154.780472144.616032154.383424148.69656147.157832156.312752145.663264155.191832148.676264154.127576148.407472150.885856151.001264153.424296150.702904155.936768147.072952158.14028146.972984158.744984147.508232159.640968150.366064155.801904149.326496158.968488151.466392152.59572160.141344151.126856158.281704152.333328160.277184155.724408153.99008156.392296157.696688157.869512157.634768152.379248162.365592158.424512153.754984162.71728157.316096153.27392166.647456159.179032153.181016163.018163.159064153.26508162.68036164.027408161.445904154.796144164.229872166.322344153.944856162.868592164.810544166.14316155.259144163.950488163.129232166.76008157.787752163.02684165.517064163.414592163.007184162.0708162.207592165.729936167.061952165.01892163.733856159.973688161.959736169.392904168.288816167.749456166.59964162.86636163.258128166.264328165.335376165.685408170.760736167.578872169.20208164.70692169.48256163.896344165.999816171.363816171.408984169.096528171.529888168.727872166.442344171.723376171.781184169.736656169.71984171.054144170.490016171.66116169.347768168.026952170.15464172.450968172.543872170.931472172.72856172.816816170.781232172.901576172.945064173.029608176.099856173.66072175.942144173.448528173.364656176.121672174.352608170.563872173.596408173.735408177.472656173.87272176.162248173.367816174.134736176.930672176.898968181.961856176.73908174.430544179.022784174.528168177.684672177.17056177.502328177.45164182.379984177.467032176.5728178.13856178.001392181.034376176.201936184.68016179.573264177.990032181.229112176.423408179.460736185.583856178.698184180.741112180.374608177.485904184.274488184.097768180.00436181.820656176.834096189.688656179.942968182.45972180.013488182.901016187.043248182.85544180.17316183.002624188.1168184.421952180.892688184.835168187.67592181.506752180.909048189.946784186.498528179.51236189.072184188.7172181.455704181.20016195.671752183.474544181.711472193.772144182.02384184.906872195.131048184.802472181.830128190.96708189.303936181.90228194.605512187.039504181.47292198.283592186.9428160.09276893.33652895.02912896.1898897.32725694.97350495.53137695.64603297.62646497.64054498.54913697.97642498.53692898.87854498.98495298.8707698.970008100.614096101.63188101.91772101.39594499.335168101.253704103.128888103.366101.132688101.794296106.160264101.081024102.9596108.836768100.993056105.800888105.03708104.02804107.779736104.0286107.663632105.169832107.802512107.210184108.125368110.00136105.694128108.997904111.278976106.978864109.19056111.686936108.074728110.609928109.621592114.104488108.315352112.090936109.77896113.30144112.994848113.090504110.431272113.652168113.737688114.865968113.600768113.999672113.522984115.14212115.185176115.232336116.945184116.016544114.930144116.269632117.127232117.780736116.72568116.835416115.897288119.19972119.403472117.925928120.260848117.44364120.666672118.516856119.535824120.675168122.21944120.502352121.228584120.647576121.788872123.041176120.571288122.56724122.33836124.294712122.317144122.683296124.02408122.503888126.371104123.095008126.637976122.882992128.506456123.02408128.600824122.099008129.844088123.738272128.30344126.091704129.316048126.790976126.115936130.335656126.235848129.72624130.24824126.925376131.034952130.76392126.950344130.314408133.042784126.299376132.11836132.926456131.297456129.970784131.089488134.689672132.387624129.375392132.54092132.313608136.001768134.395832133.578112131.99508134.107344131.576704137.18964135.450376136.458144135.082232135.756135.778784135.922488134.71844135.873128136.808184137.916784136.36764138.090904138.191216136.812128138.28456136.833168138.396096138.477184138.563856141.720472142.11716140.576344139.522856139.695472142.170968137.468168140.325216141.072784141.524488140.831688144.138312142.64516142.908328119.816168107.925648106.520424109.663016106.589168109.350392109.740616107.30316109.107912111.263336108.271184110.981808109.4704112.56228110.5936111.247264111.135832111.80084113.372968112.208784112.96084111.582816113.704392114.8422112.808104114.943992114.169552115.113368114.73024115.48296115.333568115.40264116.209232114.594056116.721464119.782296117.189872116.578296116.90492120.04952116.240896117.047496122.535632117.560392119.458056118.433504123.193688117.240176120.687672120.61392122.601984120.943408120.293672122.68696122.808656120.867992123.321728122.613184122.659088124.968088123.194952123.803192124.199544125.307528121.876096127.820576122.06528129.603336122.011504130.088688123.064832127.778648126.776528125.333168127.159728128.899176128.202072127.038968128.576608124.735936130.735504131.3972126.398136129.312056133.434656124.977584131.102816132.043504133.282472126.557648132.847528131.81984132.851064129.746496132.61592132.726688134.818688130.440632133.993936132.94856133.440592136.80172133.066656131.489672137.16428133.7018135.602424134.303112135.533312137.612496136.10836136.27572137.737888137.044856136.107672137.558112137.162264136.278592138.263408138.408048137.537816138.41192138.466752138.703472138.733752138.30308140.433752139.329608139.106952143.229392142.23556140.649504139.223936140.751592141.26664141.525168141.111216142.9502143.493752141.167968143.542048143.681832142.04208145.092248143.124832142.032456147.545592142.813232145.27644143.380704148.487592144.485264144.088952145.093768149.138816144.487296145.835424146.04764149.47044144.869752146.966096150.28284146.924344146.027736151.311128145.915984148.936150.15616147.707544150.181424149.643048149.126608150.163392149.89936150.182472150.017032152.913296149.05084148.577155.64064148.921456151.996032152.206216152.174984154.19528150.499576154.53016152.451512154.725952150.73484154.700816152.951544154.913096153.152032155.119344155.893296152.477016158.531128150.674984158.091936157.101344155.09144156.40192153.235688158.623424156.504328156.198656160.745232151.821456161.253504159.67864154.048888161.580232157.267144156.611264160.6658158.838456156.462016163.6872158.743784154.243784165.647768159.811112154.802472162.813264165.797672157.515184157.311392163.581232165.759408157.841158.123512163.682408166.555144162.310512157.355232162.705656166.7508165.850816158.994096162.360472165.247096165.411656169.229376158.62492162.807576167.605166.39228169.9688162.11464163.69192164.572512167.08456169.42164167.640488167.302832165.191144164.02356166.256096168.167984170.469592168.566144169.521048168.862904167.36392104.22746485.12971285.54726485.81372886.35246486.11529686.4437686.50828886.58445687.30295288.16875290.34466489.69908886.96467289.10665692.82846487.91416890.88117692.22131289.17768894.35671288.98887295.63491290.03950494.34472893.15959292.75232895.5697694.35817693.20026494.65816896.38249697.1675293.90529697.9702495.1007697.55508898.54566497.34510497.84716898.71012898.79959298.84222499.39731299.5740499.143696101.049296101.034568100.891912102.279432100.600336101.304128102.577456104.280336101.149816103.376312105.246208101.911608104.244312105.34704103.428824106.97208103.325272105.664784105.895504106.947224105.817976107.805832106.428928109.050472106.843296107.771296109.197832108.153664108.465624110.217848109.77416109.15264111.50468109.165984111.512376110.361256111.874936111.202856111.46468114.403448111.171032112.526656113.591376112.800024114.833872114.663592110.754448113.94764115.137704115.186536115.23044115.322216120.14512115.563976115.43244115.575752116.871216117.999568115.821608117.023976119.284816120.701736117.369104116.910648118.66432118.860672124.955616116.925952118.7878120.500976122.996808119.53056121.614944123.170728121.149352121.874312123.351824121.219912122.211288124.487792122.25528125.402976122.65432126.586288122.893032123.542152125.488944125.116448126.33988125.303888125.393792126.468832126.089592126.31796127.3748127.70732126.216104130.427448127.228376126.911416129.567304127.931392129.463824129.708192129.485712129.208288130.434504131.102128128.06312132.184752131.556888131.627064130.91492129.948656133.559832132.847488131.680168132.661784130.278784136.551392133.782608133.94516133.598472133.503504133.449096133.855936135.702872137.275624134.360488136.013016135.968488136.169112135.746704137.73292134.938784136.485672136.46036136.964216138.184168137.948768138.649144138.4098138.49464139.235296140.222248140.8648138.80928138.861064139.047168139.025816139.000856142.7140.72516141.298984142.441312142.469768139.461624139.523096144.443488144.361768142.654144141.181048141.457184141.74616149.066736144.052096139.990592142.345032147.131488147.664904141.501528142.07744149.021968148.211872140.046608145.631512151.159904144.681656140.889408154.930184142.233784146.919768151.404768144.233968148.824456150.55090489.77029675.3352876.34339275.9091275.98263275.91965677.66968877.53902476.51270477.05475278.59798480.45316879.0644479.12574480.65295279.95932881.9751680.63874482.11236881.33717683.96042482.8342882.05912884.13236884.7323684.38201684.20118485.76249684.55585.84042486.4386887.0488.0310887.20983287.62550488.31586488.1642488.64810490.43094488.88497689.4188492.1486889.09340891.85266490.83640892.605291.63834492.767492.62749694.31131291.21289695.1591295.1907692.5074895.4121697.107896.18193693.81079295.71892896.79496897.59249697.59273697.36690498.33427298.01155298.73010498.77830498.854312100.29833699.05985699.62512899.88228100.7314102.976664101.176592101.641248101.038056103.197896103.072168102.529312103.578496103.955896103.55024104.109496105.858968103.453312105.871312104.671107.024344105.3124107.264472105.735016108.485688106.4904109.13148106.323312109.276472108.954024108.04584109.35724109.208048109.09248110.506896110.818864110.163752110.834032110.894904112.251304113.070384110.416744111.075696114.576376113.077312110.853504114.776544113.832416114.870192114.702952115.1504114.21112115.22704115.343712115.179648114.56528115.505856116.696136119.087232115.749376116.886032119.717976118.66052115.124016118.904296120.231904118.370496120.72536117.576704120.872168118.61724122.754928118.874736121.080936120.385976121.424552122.721592120.54756122.779528122.917512122.52024123.12588122.492512123.33296122.763904124.91928125.675208123.778168124.405296124.277008126.852312124.495126.96672125.777504125.908136127.30384126.29552127.230688127.687448129.258072125.092192127.754976130.257744128.493864129.379968128.2644129.540384129.221832130.089288131.1552128.163288131.71344131.362624131.509248130.056416131.964936133.173936130.867944130.190432134.317456133.44128132.02584133.367872134.1452134.614888133.854016133.286296135.944904135.33064133.65636134.644064135.856408135.902608135.752408135.5642136.190112135.924856137.84064137.93244136.084936136.447232138.152528138.208064138.323376138.53764140.225296138.892704138.56808140.10656140.431624140.382752139.034032141.186512139.022752142.171344142.56056140.063984139.635016142.627248143.265816141.191248142.762672144.362752139.86872142.837504145.54172141.142112146.090872144.222968140.037344150.44128142.390936143.24556143.72228148.852488143.932968145.919376145.621408147.247512147.603736144.507168149.02436146.1562147.776904146.699312147.587784149.097472147.28708148.289688149.973592146.8878148.480768150.140264148.147736149.305656151.40216148.759816148.423488151.141656151.983112151.60408150.149456153.822624150.11528150.7048151.099376155.48072151.0292153.978768148.909032158.522408148.835064158.791544148.869096156.771048153.962656153.954168154.723672151.480472154.798184155.539816157.886344154.835328155.38844155.241888155.55308153.81192160.50564157.76192153.395816158.727264155.690496160.003736158.778936154.028816159.885592160.082688153.215544162.560872162.887248153.505160.060064164.697216152.922544159.466984165.594832158.68072158.342888160.839232163.9992159.04108159.899608164.662936161.610328158.055888163.929016163.951768164.923872159.333512160.9358165.197688166.715163.509128159.238248165.03028165.897672169.400424161.417544159.490736167.873984166.171128167.732328170.05256159.224952151.50606484.17505683.93371282.89378484.6060884.37396885.42748885.60569686.32287285.80337687.14171287.0992887.24183287.42313688.85714488.22413688.38518489.18423290.95432889.38720890.05647291.37215291.013491.47895291.10571293.14117691.25426493.66728892.50931294.5196493.19198493.60677695.4577693.89776894.51755296.85258496.9585294.7121296.32129697.59413696.1362497.760497.52605698.71344898.8069298.90630498.85315299.68895299.525376101.043576100.406496101.63312100.034496101.152008101.095152103.408048102.855896101.75392103.163504103.959128103.18584103.586168104.575144101.365192108.14084104.044976107.129544104.496312107.158152105.516248108.396672104.23316108.978968104.637064110.686792108.303608107.82184109.952248107.752328109.76532110.149712110.6302107.34928112.989064111.432696112.181032108.111424111.496216113.351984112.977048113.39856112.496032112.525592113.769776112.708952111.019832114.970024115.057224114.060888115.253384115.263048115.32548115.460448119.679248117.205944117.207424118.036768115.6286115.950624117.776328116.898424118.142984122.758464117.482288117.443976117.596728123.280216119.699752119.005648118.993952125.986792119.138096117.790824125.426472121.375368118.003912127.402456120.89576122.64724123.586416120.118016129.410704118.729808129.185208121.67156126.7704122.712904125.694952124.246456128.717632124.392176126.666096124.39328126.308456126.81072127.118496130.765944126.466664126.042824128.746072129.644184128.288912130.306704128.523608129.090992131.26684128.775912129.281376132.02308130.842224130.49136130.279176132.95044131.633888131.971784131.643184133.941133.554872132.9688131.117904132.051608135.409064135.828096135.526856132.922216136.096424131.782752135.160784136.012128135.796144136.4668136.120488136.359216136.165784136.198952137.919032137.985296138.107504136.461888138.226736138.350752138.8612139.289816136.87516138.578048140.28172138.714016140.808528138.825140.394032144.251232140.298232141.318496140.116312139.2778141.308032142.679128141.097248144.516672142.28564139.315488144.337608144.727312143.106752144.827248141.40664142.587456148.534112144.18256143.428984142.192064150.38816142.180016146.997608143.697768149.617576142.120528149.325168145.870832146.9352147.041768149.484816147.602456145.224888152.081216145.917504148.842456150.95536145.825936146.756048153.643016148.293904150.223096147.982576150.93716152.331128150.43428155.042496147.46856153.705112147.796184152.16836152.836128151.62556153.635032152.931688151.322144153.764153.15344154.13664153.014248153.191656154.535856151.299904156.656016151.497984157.9782156.632296155.139704153.461752155.113888157.987424151.401752161.65828154.719672155.11208160.082064152.1568159.738408158.367688154.18508162.166184159.286016153.873704160.378624162.21636153.79528160.02692165.425032151.944016159.68472165.421768155.527688161.595608160.638112158.84084157.652672164.427168163.649784160.689504158.279856163.455016165.193752160.58308160.34163.686544164.966672162.796576160.054576164.073488166.597656163.66708163.85184161.656512165.027032166.178704163.9242167.66964163.679472160.320984167.59464168.88072165.191504167.226456166.954576166.036872165.863544164.739904168.380344167.676408168.892488170.503528168.381424167.037328166.258936166.779472170.337376166.37744170.29208169.258592168.958952169.070816171.822184169.54708170.882392170.925296169.451768169.611408172.300456170.249576172.441688172.142328170.929472169.742592169.692032170.236096172.812184172.792264172.94792173.018528173.0828173.158168173.231832173.733016175.916688173.466736170.541656173.612472176.748032175.394424174.001472179.836144174.529752174.01844176.755296174.66528174.134216176.35728176.751752174.9252180.21884178.577248179.5208172.596768177.996456176.756784179.064048175.263232181.187232179.304784178.229016178.09508178.660968181.274952178.393872182.751064178.227112180.927784176.915752180.876704182.993832179.746376181.429424180.212248181.769064183.39708179.308608182.840672182.521248182.80284180.368016183.431216182.705688183.17444182.52244183.10008184.345344184.988832181.054296183.943968184.488048185.57236180.59244188.162968182.984248183.915736186.710624185.761392183.3542187.755312185.268688186.879488183.484576189.747904187.457016184.021456189.275232186.957376185.349936190.205184186.16992185.921936190.598424187.040328186.996528190.225016187.750984189.089344188.608592190.299672187.603216189.69856190.25264186.870752193.21828189.41028190.263376190.9868187.444456193.578112190.268592187.698592196.720184187.774216194.424488191.280736189.766344193.446424188.606296198.54808189.229984194.542344192.291784191.13892196.721576189.211624199.19016189.908096197.603376191.116197.806512190.762704197.098264192.117968196.271193.168456192.617624197.164968193.388408200.119576192.979624197.465232193.427184197.701624194.56356200.084264194.555872199.046952196.237488195.537298.84476898.42229698.99604899.77583299.72344102.419288100.822912102.952648100.13551299.8098103.152216103.427808102.780624101.763304103.694456104.651584102.212256107.395752103.088208104.228648105.972232105.848824106.604344105.480168106.422048106.544368106.446088107.686488107.93852109.625272106.381816108.492696111.543528106.445816109.594392112.599168107.529272109.547168112.52696111.91136109.780544111.432832110.547112112.896248114.501936112.427296112.381648112.445368112.501904114.854312112.594168115.026096114.051584115.133016115.224112115.285856116.465016116.958384116.366328116.968824116.691096117.24428117.40864115.83856115.915072122.004944117.071976121.542116.794096116.527936122.877544121.02208118.201744117.632096126.436432118.717112118.080464123.290176123.042456119.554416123.496536122.092656122.160032126.43236118.512168125.622904123.367224124.490936124.542496121.564544128.795496122.379936128.620728122.81908128.305256123.376496128.784424126.90772125.717928127.667432125.481992130.711144124.918728130.231864130.574432125.497336129.384864129.300696128.81092129.197288132.527016127.53764130.608984133.019592130.208008129.258464130.913288134.836872131.740128130.842344131.923688132.842016135.091512131.116672132.364544132.62872134.432135.423608135.94028135.45256131.376216134.196064134.718016137.459135.964736135.4248136.124904136.15384137.874168137.937424136.436392136.610752136.67708138.099544138.235128136.687592138.374936138.499888139.521232138.656832140.291424139.352512141.757112141.852736138.9318140.343064139.352512144.108752139.30392141.315408140.91284142.946816142.3948141.518512142.189656144.021424143.217672144.623528141.691328143.874328144.744784143.852752144.721328142.220512143.925064146.893264147.10292143.731504144.621016150.123064144.686376144.75528147.13692145.983528144.832232148.057856150.7302145.257576148.290736148.91136147.557032151.771376144.830752150.353128145.826816149.535968152.960128147.56208151.171784149.973592152.431856148.915576148.790672154.812424148.949488153.044472151.115512154.9812150.382232154.492576150.511736152.055544154.266168152.84644154.583392152.791328154.229296155.41936151.800856158.313312149.908736158.55716150.073408161.30956151.748688154.875344161.864248150.421064159.69872152.247064159.347344160.210872153.873424158.927544155.959096156.386488162.447248154.45816156.50784162.496112159.078512156.659512160.392784159.428296157.451624162.675936161.19336158.182032160.512408161.200168162.674752161.493768158.16592161.999576160.777888161.798248163.158736164.34844159.058163.973424165.905592160.080232163.547376167.240216159.696264163.591128166.791248162.849032165.293016164.292592163.557624166.311456164.9178169.408984163.088656165.744984162.121704167.944888168.066016167.9728167.23756166.21064169.536162.00428168.186608168.714296165.887168171.085984171.169768168.324328167.601016167.759168171.472952165.724296170.098376169.234344169.394608169.632768171.908456168.991968172.090424169.612096170.943048172.355248172.448784169.976424172.535170.394112172.631168172.66816172.853128172.94008172.066424172.80364172.703312173.161170.659592174.820576173.49336174.27092176.31784173.79636178.573872175.915216174.066456173.826248171.753952176.17456179.24564174.43672175.485048173.416968175.350456179.40372180.269952175.612872175.918904180.192128176.2038175.342376174.066096177.654264185.3352177.78708178.613032176.368624176.561656177.159344183.153248180.783936182.274184174.77916180.2538177.650048184.097768181.30316179.947216179.032608177.897168188.185768178.854472182.750424179.47856180.66672186.33828181.229128180.926608182.225872184.352576185.352392182.13884178.609232187.613392187.431816181.321184180.548128188.664832185.07356181.145392182.77256193.894856180.07416182.531576186.847856188.680704183.703504183.022984192.300264181.963832185.799232140.00688893.31792896.58534492.34525694.12454495.99604896.27863294.33665697.70575294.01813696.69732898.44516897.69220898.54515298.6569697.93801698.76504897.98568899.752698.97992899.88893699.954032102.396672100.223192101.439808101.820496100.35556103.900416102.53332102.925928102.205416104.765024103.826808103.696664104.606432103.749368106.787536103.461856105.99312106.490672106.348648105.407272108.503088106.740184108.048896107.255696108.984456109.2052107.814784110.111232108.223696109.221832109.406111.3688110.500952110.014632110.60844111.614576113.251110.861632112.293824112.597648111.058784114.67896112.056952114.824288113.75496114.996464113.918088115.10048114.000136115.228144115.311688116.62588115.467272117.300176116.968168115.802104118.56928117.879696117.581808115.968888119.91492118.005544118.609848120.34936117.550592121.270384119.50928119.7308119.346056122.582544121.698392119.387504124.252344119.581672122.223576123.94456119.055592125.36812122.164304121.909904126.97436118.724168128.718424120.875992128.773344121.984888127.75724122.449176129.18972124.002184126.207976124.608712126.920184127.423904127.572352129.757888126.138872126.98128128.1038129.281488129.31684127.593896130.295984129.429768129.2274130.779129.116224132.326344129.789131.468872129.352736134.726672129.680728133.474832127.86948134.960968133.588248133.570184133.541904128.261456132.762544136.300736135.612408135.434424134.896248137.355936130.327968132.857424136.044984136.149135.898136.51292137.76436137.810296135.888752136.829096138.043424136.61544138.2128138.25044136.86056139.940816138.48984140.467016138.604376140.37044138.75992138.823048141.140424141.041984142.53616140.639984139.382144139.230168145.599408141.90008141.177232140.58908144.139312139.687816146.055952145.489832138.574936144.193216145.258328144.45144142.75587272.50065672.6210873.57043272.48900876.26699274.19973674.25759274.2560876.54527276.3484876.7921276.39882477.41916879.51088879.48288877.86894478.21422481.30552878.99353679.19695283.24675279.4626482.71789680.24203282.82806484.41081681.01967285.15598484.17770484.50977685.75389682.62091285.62183285.70685686.35805686.44963287.12523287.81655288.2015688.57563288.15617688.63804888.0902491.44465688.80005689.53616892.86010489.69197691.64699291.13214492.85867291.52891292.989292.1994.8179692.87367294.0359295.11056895.15844893.70847296.13228896.39645695.92316895.5287696.81373696.94518498.46823296.99192898.60809698.7054898.75482498.82345699.66941698.98074499.872056100.679176101.108432102.34956100.095488100.943728101.890736102.25612103.780944101.616248103.4588104.51104102.216784104.99452104.927384104.17336106.271592103.148664108.314936102.371064109.407296103.865168109.193136106.4466107.661107.583288107.258576110.139568107.6588109.578312109.23264110.717168108.059368109.282304114.020736109.053168109.252136113.76404111.083152113.316784111.39112111.588784111.470888113.750776114.749704112.041392112.929584114.95128115.02836114.929392114.252168116.344928115.32976115.395968116.6408117.102216115.38512116.769384118.141496116.140656119.72776117.067416118.395592118.71056118.21408119.403472119.749432119.860848119.036328121.159088120.842192120.84552119.199872124.102168120.349184120.55328125.068696119.12372122.197728125.340776119.717126.725016122.271848126.018584121.705816125.21748124.455112125.272072125.871864125.502024123.684272127.541328122.931792130.47228125.295472127.95256125.832784128.153168127.843096126.1814132.303592125.36388126.49572132.376872128.809928127.67468132.771856128.743504128.122368133.018312130.87652130.838464130.45276132.417504131.48328131.064104133.494144133.413688132.977656131.69444132.90116132.80836133.794984136.528312134.973512131.231328136.447264133.671344134.598488134.496216135.858376137.56064136.039936135.991624137.060096137.085512134.68384137.97984138.025232138.093624137.154296138.312424138.418248139.32384138.463328138.411016136.7058138.716544138.889488139.014904143.224032140.13184142.626968140.358096141.194344138.89316142.221112141.927216141.500832144.390032143.288328139.582048142.341016145.769936143.284456142.95556140.926608145.238952148.4038142.525688142.787184142.377168150.989312143.528856144.049376142.968151.794264142.6748146.602672149.10984147.472608143.440496150.574408148.077392145.753216150.827248147.988592145.885528151.239624148.177312147.588296151.012344146.41772150.591984149.061704152.258936151.769146.512048155.246216147.340096152.89636150.929768153.240736153.00336150.856264151.809832150.728608156.003752148.439248157.033112150.500392154.707952153.152608156.908216154.818144152.649312155.791784151.593112156.752888154.607608156.389984152.319768161.076704155.268048152.805159.869736153.659216158.12836156.938168154.83984162.00944155.029048155.188784163.530216154.659488157.130592162.932488158.13464157.480888160.501656160.604608158.115184159.794688161.541488155.566064161.214408164.454112159.024544159.043528164.094784164.352296156.065264161.175128166.099736165.626856155.590704163.746856164.968704165.825592159.27808162.74572166.596832162.646672165.941704162.067968164.495376163.701248169.02908163.500888165.510856164.707576165.840344166.710048165.965016167.742752165.98308165.21684167.362248167.190472166.797216168.465872166.09208165.744328170.429784168.878672168.781048167.248544168.906904168.677544167.297504169.332184168.987376169.772392171.294184169.745672169.636096172.042544170.102168168.44328172.202768170.04528172.354248172.191096172.616264169.817832172.669576172.856344172.9472173.045672173.137096173.016736175.406528170.297952173.23828170.677688173.353904176.62484173.815456176.237736176.508016173.709984174.211408176.593608175.76628174.73792174.059592174.09644180.069144176.98928174.149392177.197888177.107144180.202968176.806872171.66064180.743408178.539672178.247952175.088456177.633952182.399608175.435144180.9438178.779296175.272768184.6712178.249472176.875472182.725592178.08184181.213872182.667264178.778608181.078144181.155144178.563184181.9958185.599472180.21044180.857968178.662488189.040656180.952768180.926232181.502032184.622032185.386184181.421952182.19936182.316392186.951144183.666496179.53464187.652424189.002752180.599888183.750184186.321656186.399544183.462472183.825872191.096688181.978688184.090488192.330232183.5982184.563048192.191472184.037968183.170936191.347048183.887128189.611064191.020264184.7558189.98808190.92984185.372672188.359424192.369488185.850408189.036392192.321328186.064624189.875190.498032190.14436194.21536186.939456190.078488193.336296189.186032191.713048188.455456196.3252186.91156193.6952193.694656189.705816192.922816189.183472197.0602190.196128193.022248193.010704195.497232193.034672192.869624195.873688189.500984195.721048193.350968194.209376195.1928191.886248198.392672194.729064198.293248191.741264198.279656192.240672201.981016192.384392198.6948192.525672198.389096192.790736202.081544192.945016200.521392194.682528200.875736191.30864205.461408197.77108195.047576201.317064195.144216204.576392193.718376201.433048195.411504201.80156198.965768199.26964197.49808203.077048200.672344195.087656206.402144195.188264202.007592195.632048205.37308204.19416192.074212.149264193.042672109.137856101.076816101.658608104.048544101.065101.323048104.504664103.788816101.44876106.20392103.056096105.686105.098552103.13248106.500112104.787736107.775168104.982264107.447744106.361448108.17192107.612032107.165408107.679576109.021704108.794288108.838392107.811656111.990984110.94556108.040024110.627152111.81404111.151736111.389496111.074912111.433114.494896111.732592113.615696111.800496114.007416113.73816114.441008113.137232115.05612115.137704114.167912115.273032116.390696117.385312114.98956115.68776118.369864116.024568116.82744116.025208116.855104122.312616118.306936117.110224119.413504117.235432122.467664118.192256118.984336119.81412124.31016118.548504120.731608120.811424123.713824121.193816121.049888123.072056121.479416125.706448120.768112124.59872121.870424124.730424123.478912124.511256124.712464125.099776123.068288128.154272121.878912129.921696124.885488126.972448126.640272125.438288131.371296123.123352130.012072129.062184123.958768132.11892126.990888128.188168131.46208129.091192128.385504129.69696133.612376127.480848128.822392132.326344134.54108129.543496128.826336132.202688134.432608133.825184131.972672131.722144130.564264134.447264135.083528132.81472130.373336132.252132.569112136.663984133.983768137.078888132.460688134.32972132.947312135.752704135.720344136.0428134.983504137.166528136.884672137.679832136.103048137.941952137.922216136.970488136.478768138.193504138.324736138.371512138.44528138.509264138.675624137.497232140.415424142.148216140.704376139.429168139.602496142.023456142.69636141.116216139.578112141.003592142.524952143.754296137.715984147.261144143.470624141.401872142.496144142.356144146.246424147.274048140.346016143.995968141.725096148.345264145.34816144.430312143.852376148.727904146.09284144.153672147.576096148.726096143.4878147.694184150.780888145.332296146.960936147.807904147.593232147.958656148.338184150.776512149.251216147.251312149.17584149.548512148.986512153.76484148.38336150.813704149.876856152.27028151.305752150.118544154.114872151.157456152.432968149.82708153.327264153.328672152.747688152.426872154.148512151.138032156.187656151.391488156.331968153.902784154.959096154.28944151.629392161.11136150.9608158.28044154.85128154.189312160.94816152.636232157.71684156.570048155.493112161.987704152.662376155.417624163.266048154.422344157.354936162.564968155.541816157.726184165.16028154.338528157.170704162.929159.472544157.577472160.266872163.113376157.095456160.038392163.635408164.179248156.896544162.462016163.976752162.104472160.041456164.030168159.481512166.872544161.232128162.089128163.707656162.13892167.258128161.187512164.73128165.359472164.226168165.907904162.953392163.464576167.672032165.339264167.80616165.658576163.439096168.369184168.14616165.435904

6000000 1448Time (s)Throughput (Mbit/s)TCP Throughput CERN-Chicago over the 622 Mbit/s link

a1-30minSetsockoptrcv=6000000MaxsegTCP:1448sink4startingonlxusa(reclen=6000000TCPhost=lxusa-ge)-TueJun46:44:262002#descriptionhostsample_KBtotal_MBsample_KB/savge_KB/scpu_secuser_secsys_secsec/MBcpu_pct1sinklxusa10486.41610.4865388.575388.570.1100.110.0161.94643.108561sinklxusa10486.41620.9739604.6877496.6280.0900.090.00983.03876.8374961sinklxusa10486.41631.4599612.6558201.9710.1200.120.011114.12976.901241sinklxusa10487.86441.94710453.4828764.8490.0900.090.00995.13383.6278561sinklxusa10486.41652.4349629.0118937.6810.0900.090.00986.22277.0320881sinklxusa10486.41662.929644.2869055.4480.0700.070.00767.30977.1542881sinklxusa10489.31273.4099656.5869141.3250.0900.090.00988.39677.2526881sinklxusa10486.41683.89610303.1449286.5530.1400.140.013149.41482.4251521sinklxusa10486.41694.3829821.1239345.9490.100.10.01910.48178.5689841sinklxusa10486.416104.8699803.2029391.6750.0900.090.009811.55178.4256161sinklxusa10486.416115.35510400.9629483.4280.1100.110.011112.5683.2076961sinklxusa10486.416125.8419850.2389889.0340.0700.070.007713.62478.8019041sinklxusa10486.416136.32810508.6319971.2110.0900.090.009914.62284.0690481sinklxusa10487.864146.8169797.0269987.9720.10.010.090.01915.69378.3762081sinklxusa10486.416157.30210538.4249995.6940.1200.120.0111216.68884.3073921sinklxusa10487.864167.7910118.86910040.2260.0900.090.009917.72580.9509521sinklxusa10486.416178.27610357.76210105.0880.0900.090.009918.73782.8620961sinklxusa10486.416188.76310532.14610184.6840.0900.090.009919.73384.2571681sinklxusa10486.416199.24910702.78310221.0150.0700.070.007720.71385.6222641sinklxusa10487.864209.73710229.60710258.150.1200.120.0111221.73881.8368561sinklxusa10487.864220.22510612.01510331.6790.0800.080.008822.72784.896121sinklxusa10486.416230.71110610.72110350.7480.1200.120.0111223.71584.8857681sinklxusa10486.416241.19810654.54410423.8660.0900.090.009924.785.2363521sinklxusa10486.416251.68410722.83310443.3390.0800.080.008825.67885.7826641sinklxusa10486.416262.17110732.19410528.3540.0700.070.007726.65585.8575521sinklxusa10486.416272.65710814.00510553.4070.0900.090.009927.62586.512041sinklxusa10486.416283.14310872.62210621.930.1100.110.011128.58986.9809761sinklxusa10486.416293.6310907.21710671.8810.0900.090.009929.55187.2577361sinklxusa10486.416304.11610989.26310713.4370.0500.050.005530.50587.9141041sinklxusa10486.416314.60310837.78810725.710.0700.070.007731.47386.7023041sinklxusa10486.416325.08911239.23110817.4940.0900.090.0091032.40689.9138481sinklxusa10486.416335.57510918.20910845.330.140.010.130.0131533.36787.3456721sinklxusa10486.416346.06211220.74610900.7870.0900.090.0091034.30189.7659681sinklxusa10487.864356.5511103.43810941.5950.1200.120.0111335.24688.8275041sinklxusa10486.416367.03611168.5710982.1170.0400.040.004436.18589.348561sinklxusa10486.416377.52311316.12511035.2010.0800.080.008937.11290.5291sinklxusa10486.416388.00911262.45511075.970.0900.090.0091038.04390.099641sinklxusa10486.416398.49511201.34211105.8530.1100.110.011238.97989.6107361sinklxusa10487.864408.98311540.59811163.4330.0800.080.008939.88892.3247841sinklxusa10486.416419.4711322.49111193.7270.1100.110.011240.81590.5799281sinklxusa10486.416429.95611607.45211263.6960.1300.130.0121441.71892.8596161sinklxusa10486.416440.44311443.80511282.2940.0900.090.0091042.63591.550441sinklxusa10486.416450.92911690.38611352.4920.1500.150.0141743.53293.5230881sinklxusa10486.416461.41511427.49311371.2870.0600.060.006744.44991.4199441sinklxusa10487.864471.90311827.27911437.0910.100.10.011145.33694.6182321sinklxusa10487.864482.39111606.87111476.9360.0700.070.007846.2492.8549681sinklxusa10486.416492.87711704.55611512.2480.1200.120.0111347.13693.6364481sinklxusa10486.416503.36411907.78911570.9150.1100.110.011248.01795.2623121sinklxusa10486.416513.8511879.16511632.5350.100.10.011148.995.033321sinklxusa10486.416524.33711632.44411640.8850.0800.080.008949.80193.0595521sinklxusa10487.864534.82512066.20811708.4950.0800.080.008950.67196.5296641sinklxusa10486.416545.31112064.05711750.0050.0800.080.008951.5496.5124561sinklxusa10487.864555.79911968.88211797.7390.0800.080.008952.41695.7510561sinklxusa10487.864566.28711753.30811803.4590.1100.110.011253.30994.0264641sinklxusa10486.416576.77312155.64111869.6540.1200.120.0111454.17297.2451281sinklxusa10486.416587.2612111.83711895.5230.100.10.011255.03896.8946961sinklxusa10487.864597.74712216.75511950.9670.1300.130.0121555.89697.734041sinklxusa10487.864608.23512208.97611996.8240.1100.110.011356.75597.6718081sinklxusa10486.416618.72212321.75412034.4570.1200.120.0111457.60798.5740321sinklxusa10487.864629.2112235.36812066.8390.0700.070.007858.46497.8829441sinklxusa10487.864639.69712341.00212131.2530.1100.110.011359.31498.7280161sinklxusa10486.416650.18412352.29112157.2610.1100.110.011360.16398.8183281sinklxusa10486.416660.6712485.07712195.5350.100.10.011261.00399.8806161sinklxusa10487.864671.15812309.62712226.5120.0800.080.008961.85598.4770161sinklxusa10487.864681.64612479.25312292.5070.0700.070.007862.69699.8340241sinklxusa10486.416692.13211071.89712193.9850.0700.070.007763.64388.5751761sinklxusa10487.864702.629595.96711965.270.110.010.10.011064.73676.7677361sinklxusa10486.416713.1079605.11711727.8480.1200.120.0111165.82876.8409361sinklxusa10486.416723.5939671.30911497.1510.050.010.040.005566.91277.3704721sinklxusa10487.864734.0819740.01611262.4480.0900.090.009867.98977.9201281sinklxusa10486.416744.5679885.30111048.8050.0700.070.007769.0579.0824081sinklxusa10486.416755.0549682.82810807.1530.0700.070.007670.13377.4626241sinklxusa10486.416765.549969.4710590.5330.0800.080.008871.18579.755761sinklxusa10486.416776.0279952.22110360.2730.100.10.01972.23979.6177681sinklxusa10486.416786.5139903.90810141.5710.0500.050.005573.29879.2312641sinklxusa10487.864797.00110062.2039921.840.0600.060.006674.3480.4976241sinklxusa10486.416807.48710159.8889838.930.040.010.030.004475.37381.2791041sinklxusa10486.416817.97410085.8089883.4610.0700.070.007776.41280.6864641sinklxusa10487.864828.46210171.6579934.9640.140.010.130.0131477.44481.3732561sinklxusa10487.864838.94910244.2159987.0470.1200.120.0111278.46881.953721sinklxusa10486.416849.43610171.61410026.2830.0500.050.005579.49981.3729121sinklxusa10486.416859.92210398.11510072.9020.100.10.011080.50783.184921sinklxusa10486.416870.40910447.81210142.4460.0700.070.007781.51183.5824961sinklxusa10486.416880.89510218.06810165.0460.1500.150.0141582.53881.7445441sinklxusa10487.864891.38310657.70510229.1810.0600.060.006683.52285.261641sinklxusa10486.416901.86910563.56210289.150.130.010.120.0121384.51584.5084961sinklxusa10487.864912.35710557.31110334.1590.0700.070.007785.50884.4584881sinklxusa10486.416922.84410619.62810375.9540.050.010.040.005586.49684.9570241sinklxusa10486.416933.3310704.94610432.2390.0500.050.005587.47585.6395681sinklxusa10487.864943.81810521.23710464.0190.0900.090.009988.47284.1698961sinklxusa10487.864954.30610793.40310513.9460.0700.070.007789.44486.3472241sinklxusa10487.864964.79410801.85310571.240.0600.060.006690.41586.4148241sinklxusa10486.416975.2810892.35210616.1710.0900.090.009991.37887.1388161sinklxusa10487.864985.76811019.39610668.1330.0600.060.006692.3388.1551681sinklxusa10486.416996.25410969.50210736.4450.110.010.10.011293.28687.7560161sinklxusa10486.4161006.74110974.39310765.2350.1100.110.011294.24287.7951441sinklxusa10486.4161017.22710985.91310803.630.0700.070.007795.19687.8873041sinklxusa10486.4161027.71411023.4110846.0030.0800.080.008896.14888.187281sinklxusa10486.4161038.211111.48110890.7170.1200.120.0111397.09288.8918481sinklxusa10486.4161048.68711302.31810945.0230.0700.070.007898.0290.4185441sinklxusa10487.8641059.17411115.87810999.0820.0800.080.008898.96388.9270241sinklxusa10487.8641069.66211223.49711038.1810.1500.150.0141699.89889.7879761sinklxusa10487.8641080.1511475.91811099.460.1100.110.0112100.81291.8073441sinklxusa10486.4161090.63611166.34611124.3680.0900.090.00910101.75189.3307681sinklxusa10486.4161101.12311660.57811182.6580.120.010.110.01113102.6593.2846241sinklxusa10486.4161111.60911345.32511216.8230.100.10.0111103.57590.76261sinklxusa10487.8641122.09711442.6511259.3920.0900.090.00910104.49291.54121sinklxusa10487.8641132.58511518.46811307.8060.0400.040.0044105.40292.1477441sinklxusa10486.4161143.07111665.22211366.1530.100.10.0111106.30193.3217761sinklxusa10486.4161153.55811409.89911393.2820.1100.110.0112107.2291.2791921sinklxusa10487.8641164.04611886.01711446.3450.100.10.0111108.10395.0881361sinklxusa10487.8641174.53411560.26411486.7440.100.10.0111109.0192.4821121sinklxusa10486.4161185.0211773.91711536.7820.0800.080.0089109.90194.1913361sinklxusa10486.4161195.50611819.50211568.0170.1100.110.0112110.78894.5560161sinklxusa10487.8641205.99411681.111614.8130.0500.050.0056111.68693.44881sinklxusa10486.4161216.48111966.28611642.6040.1100.110.0113112.56395.7302881sinklxusa10486.4161226.96711961.72811698.6410.1700.170.01619113.4495.6938241sinklxusa10486.4161237.45412012.92211750.4840.0800.080.0089114.31396.1033761sinklxusa10486.4161247.9411922.23411787.190.0900.090.00910115.19295.3778721sinklxusa10487.8641258.42812010.52111818.5810.0800.080.0089116.06696.0841681sinklxusa10486.4161268.91412099.56611881.2780.0900.090.00910116.93396.7965281sinklxusa10487.8641279.40212094.69211900.2480.100.10.0112117.896.7575361sinklxusa10487.8641289.8912307.34511968.1650.0600.060.0067118.65298.458761sinklxusa10487.8641300.37812222.43612008.9390.10.010.090.0112119.5197.7794881sinklxusa10486.4161310.86412223.52612045.6690.1400.140.01316120.36897.7882081sinklxusa10486.4161321.35112334.27512105.0480.1100.110.0113121.21998.67421sinklxusa10486.4161331.83712306.9612136.0190.0800.080.0089122.07198.455681sinklxusa10486.4161342.32312363.1712172.5130.0900.090.00911122.91998.905361sinklxusa10486.4161352.8112377.35412205.6440.1200.120.01114123.76799.0188321sinklxusa10487.8641363.29812370.30312246.3770.0500.050.0056124.61498.9624241sinklxusa10486.4161373.78412507.90612291.5940.1100.110.0113125.453100.0632481sinklxusa10486.4161384.27112475.1112325.7340.090.010.080.00911126.29499.800881sinklxusa10487.8641394.75912676.62212378.6370.0900.090.00911127.121101.4129761sinklxusa10486.4161405.24512499.42312396.0990.0900.090.00911127.9699.9953841sinklxusa10487.8641415.73313050.27212471.3570.0900.090.00911128.764104.4021761sinklxusa10486.4161426.21912523.48612498.6260.1100.110.0113129.602100.1878881sinklxusa10486.4161436.70612656.86612527.9520.0900.090.00911130.43101.2549281sinklxusa10486.4161447.19212774.15112570.4240.100.10.0112131.251102.1932081sinklxusa10486.4161457.67812740.22612604.7020.1300.130.01216132.074101.9218081sinklxusa10486.4161468.16512963.49312657.9870.1300.130.01216132.883103.7079441sinklxusa10486.4161478.65112766.40612693.9970.100.10.0112133.705102.1312481sinklxusa10486.4161489.13813002.13412738.9260.0900.090.00911134.512104.0170721sinklxusa10486.4161499.62413087.39112794.5880.1100.110.0114135.313104.6991281sinklxusa10486.4161510.1112791.04212804.990.0900.090.00911136.133102.3283361sinklxusa10486.4161520.59713146.98412863.8590.0900.090.00911136.931105.1758721sinklxusa10487.8641531.08512992.02412858.5640.0400.040.0045137.738103.9361921sinklxusa10486.4161541.57113027.18712904.3550.100.10.0112138.543104.2174961sinklxusa10486.4161552.05813496.01312980.6410.110.010.10.0114139.32107.9681041sinklxusa10486.4161562.54413021.26713003.1060.0700.070.0079140.126104.1701361sinklxusa10487.8641573.03213314.23713055.2890.100.10.0113140.914106.5138961sinklxusa10487.8641583.5213219.55313078.5670.0600.060.0068141.707105.7564241sinklxusa10486.4161594.00613337.27113130.4640.0300.030.0034142.494106.6981681sinklxusa10487.8641604.49413269.32713154.7540.100.10.0113143.284106.1546161sinklxusa10487.8641614.98213357.61313179.320.1100.110.0114144.069106.8609041sinklxusa10486.4161625.46813461.5213240.2720.0800.080.00810144.848107.692161sinklxusa10486.4161635.95513325.30613256.4830.100.10.0113145.635106.6024481sinklxusa10486.4161646.44113801.51213330.0730.0700.070.0079146.395110.4120961sinklxusa10486.4161656.92813498.42813372.9130.0700.070.0079147.172107.9874241sinklxusa10486.4161667.41413330.25213357.8440.1100.110.0114147.959106.6420161sinklxusa10486.4161677.913716.66313421.0620.10.010.090.0113148.724109.7333041sinklxusa10487.8641688.38813724.19313458.3310.1100.110.0114149.488109.7935441sinklxusa10486.4161698.87513498.27113483.6690.0900.090.00912150.265107.9861681sinklxusa10486.4161709.36113885.68813533.5250.0700.070.0079151.02111.0855041sinklxusa10487.8641719.84913854.71113586.7420.1300.130.01217151.778110.8376881sinklxusa10487.8641730.33713750.15713622.4270.110.020.090.0114152.54110.0012561sinklxusa10487.8641740.82513515.39413627.3250.0800.080.00810153.317108.1231521sinklxusa10486.4161751.31113944.42413683.6080.0900.090.00912154.069111.5553921sinklxusa10486.4161761.79714146.40113714.9620.0700.070.0079154.81113.1712081sinklxusa10486.4161772.28414018.5813762.2490.0900.090.00912155.558112.148641sinklxusa10486.4161782.7713794.66713804.4680.100.10.0113156.319110.3573361sinklxusa10486.4161793.25713938.17813824.6060.1200.120.01116157.071111.5054241sinklxusa10486.4161803.74314039.73213853.2910.0600.060.0068157.818112.3178561sinklxusa10486.4161814.2314030.65813901.690.0800.080.00811158.566112.2452641sinklxusa10486.4161824.71614334.30213942.4730.0700.070.00710159.297114.6744161sinklxusa10486.4161835.20214192.90813973.2180.0900.090.00912160.036113.5432641sinklxusa10486.4161845.68914350.72114027.8150.0900.090.00912160.767114.8057681sinklxusa10486.4161856.17514199.05814089.9660.1200.120.01116161.506113.5924641sinklxusa10487.8641866.66313966.57414091.980.110.010.10.0115162.257111.7325921sinklxusa10486.4161877.1514375.72514112.8280.1200.120.01116162.986115.00581sinklxusa10486.4161887.63614255.24714134.3430.0700.070.00710163.722114.0419761sinklxusa10486.4161898.12214400.06614189.3790.0700.070.00710164.451115.2005281sinklxusa10486.4161908.60914402.53814231.5940.060.010.050.0068165.179115.2203041sinklxusa10486.4161919.09514453.29714269.190.1900.190.01826165.904115.6263761sinklxusa10487.8641929.58314505.89614312.3940.0600.060.0068166.628116.0471681sinklxusa10486.4161940.06914702.50614345.8670.1300.130.01218167.341117.6200481sinklxusa10486.4161950.55614444.23914368.7150.0700.070.00710168.067115.5539121sinklxusa10486.4161961.04214437.0814376.5660.0900.090.00912168.794115.496641sinklxusa10486.4161971.52914669.74214419.3550.1200.120.01117169.508117.3579361sinklxusa10486.4161982.01514636.28514480.2380.0800.080.00811170.225117.090281sinklxusa10486.4161992.50214464.9814488.3520.0800.080.00811170.95115.719841sinklxusa10487.8642002.98914709.42514529.6410.150.010.140.01421171.663117.67541sinklxusa10486.4162013.47614808.97914566.8150.0700.070.00710172.371118.4718321sinklxusa10486.4162023.96215022.76614623.20.1100.110.0116173.07120.1821281sinklxusa10486.4162034.44914691.28114644.8350.1200.120.01117173.784117.5302481sinklxusa10486.4162044.93514822.29214673.5980.1100.110.0116174.491118.5783361sinklxusa10487.8642055.42314667.27414670.3950.1700.170.01624175.206117.3381921sinklxusa10486.4162065.90915084.22314728.5750.0800.080.00812175.902120.6737841sinklxusa10487.8642076.39715080.0814787.030.0700.070.00710176.597120.640641sinklxusa10487.8642086.88514972.20414814.5260.0700.070.00710177.298119.7776321sinklxusa10487.8642097.37314764.02814826.1390.0700.070.00710178.008118.1122241sinklxusa10487.8642107.86115168.82814890.1260.1200.120.01117178.7121.3506241sinklxusa10486.4162118.34715366.22114949.8340.1100.110.0116179.382122.9297681sinklxusa10486.4162128.83414796.46214948.6960.0800.080.00811180.091118.3716961sinklxusa10487.8642139.32215190.68814963.9620.0800.080.00812180.782121.5255041sinklxusa10487.864-2145.15815421.35715030.3330.100.10.0115181.462123.3708561sinklxusa10487.864-2134.6714937.3615040.7930.100.10.0114182.164119.498881sinklxusa10486.416-2124.18415393.35515106.8010.1300.130.01219182.846123.146841sinklxusa10487.864-2113.69615374.80815133.2180.1100.110.0116183.528122.9984641sinklxusa10486.416-2103.20914880.59715115.0830.0800.080.00811184.233119.0447761sinklxusa10486.416-2092.72315705.32715181.730.0900.090.00913184.9125.6426161sinklxusa10486.416-2082.23715354.29615235.3910.0700.070.00710185.584122.8343681sinklxusa10486.416-2071.7515062.53415225.7280.100.10.0114186.28120.5002721sinklxusa10486.416-2061.26415900.77115274.3230.0900.090.00914186.94127.2061681sinklxusa10486.416-2050.77715031.8115295.7190.0800.080.00811187.637120.254481sinklxusa10487.864-2040.28915879.08215358.30.0700.070.00711188.298127.0326561sinklxusa10486.416-2029.80315316.86515348.8010.1100.110.0116188.983122.534921sinklxusa10487.864-2019.31515893.83315435.7530.090.010.080.00914189.643127.1506641sinklxusa10486.416-2008.82915316.79915428.7930.1100.110.0116190.327122.5343921sinklxusa10486.416-1998.34215982.66115484.0520.0800.080.00812190.984127.8612881sinklxusa10486.416-1987.85615432.50215534.2250.1100.110.0116191.663123.4600161sinklxusa10486.416-1977.3715963.39315557.6860.0800.080.00812192.32127.7071441sinklxusa10486.416-1966.88315317.58115554.3480.1200.120.01118193.005122.5406481sinklxusa10486.416-1956.39716216.70115659.2730.100.10.0115193.652129.7336081sinklxusa10486.416-1945.9115172.43915593.0610.0400.040.0046194.343121.3795121sinklxusa10486.416-1935.42416456.48215722.5760.1200.120.01119194.98131.6518561sinklxusa10486.416-1924.93715205.02115661.2980.110.010.10.0116195.67121.6401681sinklxusa10487.864-1914.44916166.38915738.5270.0900.090.00914196.319129.3311121sinklxusa10486.416-1903.96315930.02315741.8170.0900.090.00914196.977127.4401841sinklxusa10486.416-1893.47715999.09715803.8440.0800.080.00812197.633127.9927761sinklxusa10487.864-1882.98916133.33915817.5420.1200.120.01118198.283129.0667121sinklxusa10486.416-1872.50215658.58815838.0960.0700.070.00710198.953125.2687041sinklxusa10486.416-1862.01616185.68815858.3040.0800.080.00812199.601129.4855041sinklxusa10486.416-1851.5316364.3415953.4640.1300.130.01220200.242130.914721sinklxusa10487.864-1841.04215724.52215908.7210.0700.070.00710200.909125.7961761sinklxusa10486.416-1830.55516312.7916012.3890.0800.080.00812201.552130.502321sinklxusa10486.416-1820.06916439.97116010.8880.1300.130.01220202.19131.5197681sinklxusa10487.864-1809.58115792.14616064.2630.0700.070.00711202.854126.3371681sinklxusa10487.864-1799.09316408.6716086.2890.100.10.0116203.493131.269361sinklxusa10487.864-1788.60516368.69416126.1680.0700.070.00711204.134130.9495521sinklxusa10486.416-1778.11916084.86516133.9650.0900.090.00914204.786128.678921sinklxusa10486.416-1767.63216248.71416144.4530.0900.090.00914205.432129.9897121sinklxusa10487.864-1757.14516664.06816235.8610.1400.140.01322206.061133.3125441sinklxusa10487.864-1746.65716340.20816249.9080.0700.070.00711206.703130.7216641sinklxusa10486.416-1736.1716329.17416246.7110.1100.110.0117207.346130.6333921sinklxusa10486.416-1725.68416483.51416315.710.110.010.10.0117207.982131.8681121sinklxusa10487.864-1715.19616548.21916337.1130.0800.080.00813208.616132.3857521sinklxusa10486.416-1704.7116702.68916360.9960.0900.090.00914209.244133.6215121sinklxusa10486.416-1694.22316171.06216395.4430.1300.130.01220209.892129.3684961sinklxusa10489.312-1683.73416665.36116418.7790.0700.070.00711210.522133.3228881sinklxusa10486.416-1673.24716502.11316430.9080.0800.080.00813211.158132.0169041sinklxusa10486.416-1662.76116457.28316464.7640.0700.070.00711211.795131.6582641sinklxusa10487.864-1652.27316860.21716520.3550.100.10.0116212.417134.8817361sinklxusa10486.416-1641.78716836.91216536.0680.0600.060.00610213.04134.6952961sinklxusa10486.416-1631.316579.78516557.8480.0800.080.00813213.673132.638281sinklxusa10487.864-1620.81216930.4116612.5060.100.10.0116214.292135.443281sinklxusa10487.864-1610.32516514.70916615.3420.1300.130.01220214.927132.1176721sinklxusa10487.864-1599.83716764.00616634.9590.060.010.050.00610215.553134.1120481sinklxusa10486.416-1589.3516938.57216656.4030.0900.090.00915216.172135.5085761sinklxusa10486.416-1578.86416971.63516729.1820.1300.130.01221216.79135.773081sinklxusa10486.416-1568.37816844.78116745.4930.0300.030.0035217.413134.7582481sinklxusa10486.416-1557.89116987.52516789.6210.0800.080.00813218.03135.90021sinklxusa10486.416-1547.40516939.91216833.4970.120.010.110.01119218.65135.5192961sinklxusa10486.416-1536.91816671.01216816.2960.0400.040.0046219.279133.3680961sinklxusa10486.416-1526.43217205.65416849.8180.0900.090.00915219.888137.6452321sinklxusa10487.864-1515.94417039.05916891.570.140.010.130.01323220.504136.3124721sinklxusa10486.416-1505.45817219.58216917.8590.0500.050.0058221.113137.7566561sinklxusa10486.416-1494.97117227.98416982.7020.090.010.080.00915221.722137.8238721sinklxusa10487.864-1484.48316764.24816982.7240.0700.070.00711222.348134.1139841sinklxusa10486.416-1473.99717246.96917010.760.100.10.0116222.956137.9757521sinklxusa10486.416-1463.5116922.41817006.2860.100.10.0116223.576135.3793441sinklxusa10486.416-1453.02417275.0717045.4030.0800.080.00813224.183138.200561sinklxusa10486.416-1442.53817083.97317054.1710.0800.080.00813224.797136.6717841sinklxusa10487.864-1432.0517294.83417086.4370.100.10.0116225.403138.3586721sinklxusa10486.416-1421.56317547.72717166.1380.0900.090.00915226.001140.3818161sinklxusa10486.416-1411.07717576.25617199.8290.0600.060.00610226.598140.6100481sinklxusa10486.416-1400.5917321.9817225.5490.100.10.0117227.203138.575841sinklxusa10486.416-1390.10417353.39817237.7140.110.020.090.0118227.808138.8271841sinklxusa10486.416-1379.61817113.97517227.350.0500.050.0058228.42136.91181sinklxusa10486.416-1369.13117613.54317304.5580.1200.120.01120229.016140.9083441sinklxusa10486.416-1358.64517346.53717313.610.0500.050.0058229.621138.7722961sinklxusa10486.416-1348.15817815.04517394.7580.100.10.0117230.209142.520361sinklxusa10486.416-1337.67217423.43417408.2460.0900.090.00915230.811139.3874721sinklxusa10487.864-1327.18417376.98817434.8830.0500.050.0058231.415139.0159041sinklxusa10486.416-1316.69817381.38517442.7520.1500.150.01425232.018139.051081sinklxusa10486.416-1306.21117607.86317448.2190.1100.110.0118232.614140.8629041sinklxusa10487.864-1295.72317824.59217470.7950.0600.060.00610233.203142.5967361sinklxusa10486.416-1285.23717786.64117513.0360.1200.120.01120233.792142.2931281sinklxusa10487.864-1274.74917707.64117545.240.1700.170.01629234.385141.6611281sinklxusa10486.416-1264.26317889.96317615.7850.100.10.0117234.971143.1197041sinklxusa10487.864-1253.77517513.48617606.6890.100.10.0117235.57140.1078881sinklxusa10487.864-1243.28717847.88517652.2660.100.10.0117236.158142.783081sinklxusa10486.416-1232.80117657.73417637.9650.0500.050.0058236.752141.2618721sinklxusa10486.416-1222.31417808.08617672.9330.1100.110.0119237.341142.4646881sinklxusa10486.416-1211.82818129.97917741.3870.1300.130.01222237.919145.0398321sinklxusa10486.416-1201.34117873.19117786.0960.0300.030.0035238.506142.9855281sinklxusa10487.864-1190.85417749.95717799.0140.0900.090.00915239.097141.9996561sinklxusa10487.864-1180.36618110.50217825.0060.130.010.120.01222239.676144.8840161sinklxusa10486.416-1169.87918129.3217856.1590.0700.070.00712240.255145.034561sinklxusa10486.416-1159.39317987.85517881.6330.1100.110.0119240.838143.902841sinklxusa10486.416-1148.90617858.4317878.7660.1200.120.01120241.425142.867441sinklxusa10487.864-1138.41917981.9817921.3560.0900.090.00915242.009143.855841sinklxusa10487.864-1127.93118163.9817950.0920.1200.120.01121242.586145.311841sinklxusa10486.416-1117.44418495.27918026.2330.100.10.0118243.153147.9622321sinklxusa10487.864-1106.95617796.34218025.1650.0700.070.00712243.743142.3707361sinklxusa10486.416-1096.4718231.34818034.3810.0800.080.00814244.318145.8507841sinklxusa10487.864-1085.98218072.92618052.5380.1700.170.01629244.898144.5834081sinklxusa10486.416-1075.49618490.18918119.8320.0800.080.00814245.466147.9215121sinklxusa10486.416-1065.00918051.05318114.4280.1100.110.0119246.047144.4084241sinklxusa10487.864-1054.52118252.68418125.6420.0800.080.00814246.621146.0214721sinklxusa10486.416-1044.03518286.89118152.8270.1300.130.01223247.195146.2951281sinklxusa10486.416-1033.54918621.79318222.2240.1100.110.0120247.758148.9743441sinklxusa10487.864-1023.06118302.21518251.3360.0600.060.00610248.331146.417721sinklxusa10486.416-1012.57418183.7718253.1350.0900.090.00916248.908145.470161sinklxusa10487.864-1002.08618745.29918275.8640.0900.090.00916249.468149.9623921sinklxusa10486.416-991.618436.61718334.0710.100.10.0118250.037147.4929361sinklxusa10486.416-981.11418130.95118324.9440.1400.140.01324250.615145.0476081sinklxusa10486.416-970.62718621.6618374.8290.100.10.0118251.179148.973281sinklxusa10487.864-960.13918828.42218405.5780.0900.090.00916251.736150.6273761sinklxusa10487.864-949.65118356.18918433.3170.1200.120.01121252.307146.8495121sinklxusa10486.416-939.16518786.22718481.8210.0800.080.00814252.865150.2898161sinklxusa10486.416-928.67918516.53918502.6980.1200.120.01121253.432148.1323121sinklxusa10487.864-918.19118373.19918480.0990.1300.130.01223254.003146.9855921sinklxusa10486.416-907.70419347.55918575.130.1500.150.01428254.545154.7804721sinklxusa10486.416-897.21818077.00418565.4240.110.020.090.0119255.125144.6160321sinklxusa10487.864-886.7319297.92818615.6630.10.010.090.0118255.669154.3834241sinklxusa10486.416-876.24418587.0718629.3410.0700.070.00712256.233148.696561sinklxusa10487.864-865.75618394.72918653.320.0700.070.00712256.803147.1578321sinklxusa10486.416-855.26919539.09418736.7240.0800.080.00815257.34156.3127521sinklxusa10487.864-844.78118207.90818680.3130.0700.070.00712257.916145.6632641sinklxusa10487.864-834.29419398.97918775.1120.100.10.0118258.457155.1918321sinklxusa10486.416-823.80718584.53318756.7760.1100.110.0119259.021148.6762641sinklxusa10486.416-813.32119265.94718824.9040.1100.110.0120259.566154.1275761sinklxusa10486.416-802.83418550.93418841.0620.0900.090.00916260.131148.4074721sinklxusa10486.416-792.34818860.73218796.8050.0800.080.00814260.687150.8858561sinklxusa10486.416-781.86218875.15818869.3650.100.10.0118261.243151.0012641sinklxusa10486.416-771.37519178.03718858.4660.0600.060.00611261.79153.4242961sinklxusa10486.416-760.88918837.86318881.2650.1400.140.01325262.347150.7029041sinklxusa10486.416-750.40219492.09618981.0260.0700.070.00713262.885155.9367681sinklxusa10487.864-739.91418384.11918876.0280.0900.090.00916263.456147.0729521sinklxusa10487.864-729.42719767.53519017.8120.0900.090.00917263.986158.140281sinklxusa10487.864-718.93918371.62318924.4160.070.010.060.00712264.557146.9729841sinklxusa10486.416-708.45219843.12319038.8330.100.10.0119265.086158.7449841sinklxusa10486.416-697.96618438.52918963.6140.0500.050.0059265.655147.5082321sinklxusa10486.416-687.47919955.12119091.2670.0900.090.00917266.18159.6409681sinklxusa10487.864-676.99218795.75819085.360.10.010.090.0118266.738150.3660641sinklxusa10487.864-666.50419475.23819139.9130.1200.120.01122267.277155.8019041sinklxusa10486.416-656.01718665.81219093.3470.1500.150.01427267.839149.3264961sinklxusa10486.416-645.53119871.06119187.2740.0900.090.00917268.367158.9684881sinklxusa10486.416-635.04418933.29919136.4740.0900.090.00916268.921151.4663921sinklxusa10486.416-624.55819074.46519199.2330.070.010.060.00713269.471152.595721sinklxusa10486.416-614.07220017.66819221.9720.1300.130.01225269.995160.1413441sinklxusa10487.864-603.58418890.85719269.1760.0900.090.00916270.55151.1268561sinklxusa10487.864-593.09619785.21319263.9110.1400.140.01326271.08158.2817041sinklxusa10487.864-582.60819041.66619318.7420.0700.070.00713271.631152.3333281sinklxusa10486.416-572.12220034.64819325.9710.0700.070.00713272.155160.2771841sinklxusa10487.864-561.63419465.55119386.8620.100.10.0119272.694155.7244081sinklxusa10486.416-551.14719248.7619366.2730.1100.110.0120273.238153.990081sinklxusa10486.416-540.66119549.03719446.5660.100.10.0119273.775156.3922961sinklxusa10486.416-530.17519712.08619432.1140.100.10.0119274.307157.6966881sinklxusa10487.864-519.68719733.68919504.8760.1100.110.0121274.839157.8695121sinklxusa10486.416-509.219704.34619562.1380.1200.120.01123275.371157.6347681sinklxusa10486.416-498.71419047.40619473.9330.0600.060.00611275.922152.3792481sinklxusa10487.864-488.22620295.69919601.6460.1500.150.01429276.439162.3655921sinklxusa10486.416-477.7419803.06419603.2680.0700.070.00713276.968158.4245121sinklxusa10486.416-467.25319219.37319619.4240.1200.120.01122277.514153.7549841sinklxusa10486.416-456.76720339.6619647.1520.0800.080.00816278.03162.717281sinklxusa10486.416-446.2819664.51219665.2390.0900.090.00917278.563157.3160961sinklxusa10487.864-435.79219159.2419657.1010.1100.110.0120279.111153.273921sinklxusa10486.416-425.30620830.93219773.6370.0900.090.00918279.614166.6474561sinklxusa10486.416-414.8219897.37919790.4820.0400.040.0048280.141159.1790321sinklxusa10487.864-404.33219147.62719737.2030.0900.090.00916280.689153.1810161sinklxusa10487.864-393.84420377.2519798.3770.1300.130.01225281.204163.0181sinklxusa10486.416-383.35720394.88319920.8740.0800.080.00816281.718163.1590641sinklxusa10486.416-372.87119158.13519817.460.090.010.080.00916282.266153.265081sinklxusa10486.416-362.38520335.04519865.8210.1400.140.01327282.782162.680361sinklxusa10486.416-351.89820503.42619982.5530.120.010.110.01123283.293164.0274081sinklxusa10486.416-341.41220180.73819968.1060.100.10.0119283.813161.4459041sinklxusa10487.864-330.92419349.51819939.470.0600.060.00611284.355154.7961441sinklxusa10487.864-320.43620528.73420063.970.1500.150.01429284.866164.2298721sinklxusa10486.416-309.9520790.29320060.2750.0500.050.00510285.371166.3223441sinklxusa10486.416-299.46319243.10720000.7960.1200.120.01122285.916153.9448561sinklxusa10486.416-288.97720358.57420110.8820.0700.070.00714286.431162.8685921sinklxusa10487.864-278.48920601.31820131.2520.0800.080.00816286.94164.8105441sinklxusa10486.416-268.00320767.89520165.1620.0900.090.00918287.445166.143161sinklxusa10486.416-257.51619407.39320187.8220.0800.080.00815287.986155.2591441sinklxusa10486.416-247.0320493.81120202.2550.0500.050.00510288.497163.9504881sinklxusa10486.416-236.54320391.15420192.0490.0900.090.00918289.012163.1292321sinklxusa10486.416-226.05720845.0120252.4370.100.10.0120289.515166.760081sinklxusa10486.416-215.5719723.46920286.4330.1500.150.01428290.047157.7877521sinklxusa10486.416-205.08420378.35520272.7620.160.010.150.01531290.562163.026841sinklxusa10486.416-194.59820689.63320263.6110.0400.040.0048291.069165.5170641sinklxusa10486.416-184.11120426.82420371.2210.0900.090.00918291.582163.4145921sinklxusa10486.416-173.62520375.89820372.7960.0800.080.00816292.097163.0071841sinklxusa10487.864-163.13720258.8520341.6630.0800.080.00815292.615162.07081sinklxusa10486.416-152.65120275.94920296.9410.100.10.0119293.132162.2075921sinklxusa10486.416-142.16420716.24220415.9270.1100.110.0122293.638165.7299361sinklxusa10486.416-131.67820882.74420451.2840.0800.080.00816294.141167.0619521sinklxusa10486.416-121.19120627.36520472.7580.0900.090.00918294.649165.018921sinklxusa10486.416-110.70520466.73220438.3690.1200.120.01123295.162163.7338561sinklxusa10486.416-100.21819996.71120463.210.0500.050.00510295.686159.9736881sinklxusa10487.864-89.73120244.96720451.0830.1200.120.01123296.204161.9597361sinklxusa10486.416-79.24421174.11320495.1270.1300.130.01226296.7169.3929041sinklxusa10487.864-68.75621036.10220550.5160.0900.090.00918297.198168.2888161sinklxusa10487.864-58.26820968.68220604.4050.1300.130.01226297.699167.7494561sinklxusa10487.864-47.78120824.95520655.8690.100.10.0120298.202166.599641sinklxusa10486.416-37.29420358.29520663.3550.120.010.110.01123298.718162.866361sinklxusa10486.416-26.80820407.26620635.2670.0900.090.00918299.232163.2581281sinklxusa10486.416-16.32120783.04120626.2030.0800.080.00816299.736166.2643281sinklxusa10487.864-5.83320666.92220629.7990.0900.090.00918300.244165.3353761sinklxusa10486.4164.65320710.67620651.9750.0600.060.00612300.75165.6854081sinklxusa10486.41615.13921345.09220774.5550.0700.070.00714301.242170.7607361sinklxusa10486.41625.62620947.35920838.4090.1100.110.0122301.742167.5788721sinklxusa10487.86436.11421150.2620836.2410.0800.080.00816302.238169.202081sinklxusa10486.41646.620588.36520795.5370.1100.110.0122302.748164.706921sinklxusa10486.41657.08621185.3220815.2320.110.010.10.0122303.243169.482561sinklxusa10486.41667.57320487.04320784.5130.160.010.150.01531303.755163.8963441sinklxusa10486.41678.05920749.97720820.120.0900.090.00918304.26165.9998161sinklxusa10486.41688.54621420.47720912.230.1100.110.0122304.75171.3638161sinklxusa10486.41699.03221426.12320970.6920.090.010.080.00918305.24171.4089841sinklxusa10486.416109.51921137.06621013.4330.100.10.0120305.736169.0965281s