improving cms data transfers among its distributed ... · 12/04/2011 j. flix –improving cms data...

30
J. Flix – Improving CMS Data Transfers – EGI User Forum (Vilnius April 2011) 12/04/2011 1 Improving CMS data transfers among its distributed Computing Facilities J. Flix Port d'Informació Científica (PIC), Barcelona, Spain N. Magini CERN IT-ES-VOS, Geneva, Switzerland A. Sartirana École Polytechnique, Palaiseau, France [On behalf of CMS experiment] EGI User Forum – 11-14 April 2011, Vilnius, Lithuania

Upload: others

Post on 06-Jan-2020

4 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Improving CMS data transfers among its distributed ... · 12/04/2011 J. Flix –Improving CMS Data Transfers –EGI User Forum (Vilnius April 2011) 1 Improving CMS data transfers

J. Flix – Improving CMS Data Transfers – EGI User Forum (Vilnius April 2011)12/04/2011 1

Improving CMS data transfers among its

distributed Computing Facilities

J. FlixPort d'Informació Científica (PIC), Barcelona, Spain

N. MaginiCERN IT-ES-VOS, Geneva, Switzerland

A. SartiranaÉcole Polytechnique, Palaiseau, France

[On behalf of CMS experiment]

EGI User Forum – 11-14 April 2011, Vilnius, Lithuania

Page 2: Improving CMS data transfers among its distributed ... · 12/04/2011 J. Flix –Improving CMS Data Transfers –EGI User Forum (Vilnius April 2011) 1 Improving CMS data transfers

J. Flix – Improving CMS Data Transfers – EGI User Forum (Vilnius April 2011)12/04/2011 2

Outline

‣ CMS data transfer workflows and Management System

‣ Measuring the performance of CMS data transfers

‣ Using data transfer statistics to improve transfer operations

Page 3: Improving CMS data transfers among its distributed ... · 12/04/2011 J. Flix –Improving CMS Data Transfers –EGI User Forum (Vilnius April 2011) 1 Improving CMS data transfers

J. Flix – Improving CMS Data Transfers – EGI User Forum (Vilnius April 2011)12/04/2011 3

CMS Computing Model

CAF

900 MB/s agg.

50-500MB/s10-20MB/s

CMSdetector

WLCG Computing Grid InfrastructureTIER-0

CERN

TIER-1 TIER-1

TIER-2

7 Tier-1s(“online” to the DAQ)

High availability centres

Custodial mass storage of share of data

Data reconstruction and reprocessing

Data skimming & selection

Distribute analysis data ���� Tier-2s

~50 Tier-2sin ~20 countries

End-user physics analyses

Detector Studies

Monte Carlo Simulation ���� Tier-1

Distribute analysis data ���� Tier-2s

Tier-0(the accelerator centre)

Data acquisition & initial processing

Long-term mass data storage

CMS CERN Analysis Facility (latency critical data processing, high priority analysis)

Distribution of data ���� Tier-1 centres

TIER-2 TIER-2

10-50MB/s agg.

Page 4: Improving CMS data transfers among its distributed ... · 12/04/2011 J. Flix –Improving CMS Data Transfers –EGI User Forum (Vilnius April 2011) 1 Improving CMS data transfers

J. Flix – Improving CMS Data Transfers – EGI User Forum (Vilnius April 2011)12/04/2011 4

PhEDEx: CMS transfer management system

‣ PhEDEx - Physics Experiment Data Export (reliable, scalable dataset replication system)

‣ Each CMS site runs a set of software agents

‣ Coupled & highly-specialized components which fulfill a specific “simple” task in a reliable way‣ Central agents: routing, task assignment, …‣ Run at CERN‣ Site-specific agents: download, export, mass storage staging and migration …‣ Run at every CMS site on a gLite VOBOX (to simplify proxy management) or UI

Page 5: Improving CMS data transfers among its distributed ... · 12/04/2011 J. Flix –Improving CMS Data Transfers –EGI User Forum (Vilnius April 2011) 1 Improving CMS data transfers

J. Flix – Improving CMS Data Transfers – EGI User Forum (Vilnius April 2011)12/04/2011 5

PhEDEx: CMS transfer management system

PhEDEx - Physics Experiment Data Export (reliable, scalable dataset replication system)

‣ Agents inter-communicatewith a central blackboard

‣ transfer Management DB (TMDB) @ Oracle RAC backend‣ Block replica location & file mapping‣ Block subscription and allocation‣ File metadata information (filesize, cksum, etc)‣ Transfer state (at node; in transfer; wanted; available)

‣ Transfer status monitored from PhEDEx webpage‣ DataService to retrieve information from the database through simple http access‣ 1 T0, 7 T1’s, 50 T2’s + T3’s in current PhEDEx transfer topology (~100 Sites)

Page 6: Improving CMS data transfers among its distributed ... · 12/04/2011 J. Flix –Improving CMS Data Transfers –EGI User Forum (Vilnius April 2011) 1 Improving CMS data transfers

J. Flix – Improving CMS Data Transfers – EGI User Forum (Vilnius April 2011)12/04/2011 6

CMS Transfer Workflow

PhEDExCMS transfer management system

WLCG

MiddlewareFTS

SRM

gridFTP

Page 7: Improving CMS data transfers among its distributed ... · 12/04/2011 J. Flix –Improving CMS Data Transfers –EGI User Forum (Vilnius April 2011) 1 Improving CMS data transfers

J. Flix – Improving CMS Data Transfers – EGI User Forum (Vilnius April 2011)12/04/2011 7

CMS Transfer Workflow

Transfer request is placed through

PhEDEx web interface

Page 8: Improving CMS data transfers among its distributed ... · 12/04/2011 J. Flix –Improving CMS Data Transfers –EGI User Forum (Vilnius April 2011) 1 Improving CMS data transfers

J. Flix – Improving CMS Data Transfers – EGI User Forum (Vilnius April 2011)12/04/2011 8

CMS Transfer Workflow

PhEDEx central agents create and distribute transfer tasks to site agents

It takes into account the

aggregated throughput

and errors on the links

in the routing algorithm

Page 9: Improving CMS data transfers among its distributed ... · 12/04/2011 J. Flix –Improving CMS Data Transfers –EGI User Forum (Vilnius April 2011) 1 Improving CMS data transfers

J. Flix – Improving CMS Data Transfers – EGI User Forum (Vilnius April 2011)12/04/2011 9

CMS Transfer Workflow

PhEDEx Download agent submits

transfer batch job to FTS server

FTS chosen according

to CMS policy

Page 10: Improving CMS data transfers among its distributed ... · 12/04/2011 J. Flix –Improving CMS Data Transfers –EGI User Forum (Vilnius April 2011) 1 Improving CMS data transfers

J. Flix – Improving CMS Data Transfers – EGI User Forum (Vilnius April 2011)12/04/2011 10

CMS Transfer Workflow

FTS contacts source and destination SRMs to get transfer URLs

Page 11: Improving CMS data transfers among its distributed ... · 12/04/2011 J. Flix –Improving CMS Data Transfers –EGI User Forum (Vilnius April 2011) 1 Improving CMS data transfers

J. Flix – Improving CMS Data Transfers – EGI User Forum (Vilnius April 2011)12/04/2011 11

CMS Transfer Workflow

FTS executes the transfer as third-party with gridFTP

FTP extension

GSI security

Can run parallel streams

Flexible: other configurations can be

also used

a) srmCopy started by FTS server

b) srmCopy started directly by

PhEDEx Download agent with

SRM client (not encouraged)

Page 12: Improving CMS data transfers among its distributed ... · 12/04/2011 J. Flix –Improving CMS Data Transfers –EGI User Forum (Vilnius April 2011) 1 Improving CMS data transfers

J. Flix – Improving CMS Data Transfers – EGI User Forum (Vilnius April 2011)12/04/2011 12

gLite File Transfer Service (FTS)

‣ Provides scheduling of multiple asynchronous file transfers on CHANNELS‣ Single direction transfer queue between two endpoints- Not tied to a physical network path

‣ Each endpoint (source and destination) can be:

‣ A single site, e.g.‣ CERN-RAL

‣ IN2P3-BELGIUMULB

‣ A group of sites (“cloud”), e.g.‣ RALLCG2-CLOUDCMSITALY

‣ CLOUDCMSFRANCE-RALLCG2

‣ All sites (“star”), e.g.‣ CNAF-STAR

‣ STAR-FNAL

Page 13: Improving CMS data transfers among its distributed ... · 12/04/2011 J. Flix –Improving CMS Data Transfers –EGI User Forum (Vilnius April 2011) 1 Improving CMS data transfers

J. Flix – Improving CMS Data Transfers – EGI User Forum (Vilnius April 2011)12/04/2011 13

Typical FTS server deployment (CMS)

Tier-0FTS

Tier-1

Tier-1

.

.

.

At Tier-0Dedicated channel to each of the Tier-1s

Tier-1FTS

Tier-1

Tier-1

.

.

.

At each Tier-1Dedicated channel from each of the other Tier-1s

Tier-1FTS

Tier-2assoc.

Tier-2assoc.

.

.

.Tier-2others

At each Tier-1Dedicated channels to and from each of the associated Tier-2s

CLOUD and/or STAR channels to/from other Tier-2sSTAR-T2 channels for each associated Tier-2

Tier-1...

Tier-2others

.

.

.

Page 14: Improving CMS data transfers among its distributed ... · 12/04/2011 J. Flix –Improving CMS Data Transfers –EGI User Forum (Vilnius April 2011) 1 Improving CMS data transfers

J. Flix – Improving CMS Data Transfers – EGI User Forum (Vilnius April 2011)12/04/2011 14

FTS channels :: configuration

‣ FTS channel configuration defines:‣ Transfer limits

‣ Maximum number of concurrent active transfers‣ Protect network, storage

‣ Shared among VOs according to policy

‣ Transfer priorities

‣ Between users in the same VO on a channel

‣ Transfer parameters

‣ Number of parallel TCP streams, buffer size

‣ Timeouts (SRM Gets/Puts; No_Tx_activity; …)

Page 15: Improving CMS data transfers among its distributed ... · 12/04/2011 J. Flix –Improving CMS Data Transfers –EGI User Forum (Vilnius April 2011) 1 Improving CMS data transfers

J. Flix – Improving CMS Data Transfers – EGI User Forum (Vilnius April 2011)12/04/2011 15

FTS channels :: throughput

‣ Overall throughput for a link in a channel

Link throughput = rate/stream * streams/file * active transfers/link

‣ In a dedicated channel:

‣ Expect ~constant rate/stream up to saturation (in absence of source/destination contention)

‣ Fixed number of available active transfer slots/link

‣ In cloud or star channel:

‣ Rate/stream can be significantly different for links in same channel

‣ Available active transfer slots/link depends on overall channel occupancy‣ Slow links keep transfer slots busy for longer (no ‘fair-share’ concept in STAR/CLOUD channels)

� Low performing Tier-2/Tier-3 sites can block those channels

+overheads

Page 16: Improving CMS data transfers among its distributed ... · 12/04/2011 J. Flix –Improving CMS Data Transfers –EGI User Forum (Vilnius April 2011) 1 Improving CMS data transfers

J. Flix – Improving CMS Data Transfers – EGI User Forum (Vilnius April 2011)12/04/2011 16

Evolution of transfer workflows (CMS)

Scale and complexity of CMS data transfers has been steadily increasing thanks to focused effort on improving transfer quality and throughput (200 TBs/day)

T0 →→→→ T1

T1 ↔↔↔↔ T1

T1 ↔↔↔↔ all T2s

T2 ↔↔↔↔ all T2s

* As more and more data transfer links are commissioned, the sites start competing for the same slots in the FTS channels [CMS has ~2300 data transfer links commissioned]

* Making optimal use of bandwidth requires identifying and isolating the problematic links

Page 17: Improving CMS data transfers among its distributed ... · 12/04/2011 J. Flix –Improving CMS Data Transfers –EGI User Forum (Vilnius April 2011) 1 Improving CMS data transfers

J. Flix – Improving CMS Data Transfers – EGI User Forum (Vilnius April 2011)12/04/2011 17

FTSMonitor

‣ FTS server database contains detailed transfer information

‣ Wealth of knowledge that can be used to spot issues

‣ Information is exposed through FTS Monitor (CCIN2P3)

-Transfer Sum

mary -

https://forge.in2p3.fr/projects/ftsmonitor/files

Page 18: Improving CMS data transfers among its distributed ... · 12/04/2011 J. Flix –Improving CMS Data Transfers –EGI User Forum (Vilnius April 2011) 1 Improving CMS data transfers

J. Flix – Improving CMS Data Transfers – EGI User Forum (Vilnius April 2011)12/04/2011 18

FTSMonitor

‣ FTS server database contains detailed transfer information

‣ Wealth of knowledge that can be used to spot issues

‣ Information is exposed through FTS Monitor (CCIN2P3)

-Channel Config details -

https://forge.in2p3.fr/projects/ftsmonitor/files

Page 19: Improving CMS data transfers among its distributed ... · 12/04/2011 J. Flix –Improving CMS Data Transfers –EGI User Forum (Vilnius April 2011) 1 Improving CMS data transfers

J. Flix – Improving CMS Data Transfers – EGI User Forum (Vilnius April 2011)12/04/2011 19

FTSMonitor

‣ FTS server database contains detailed transfer information

‣ Wealth of knowledge that can be used to spot issues

‣ Information is exposed through FTS Monitor (CCIN2P3)

-Ind. Transfer details -

https://forge.in2p3.fr/projects/ftsmonitor/files

Page 20: Improving CMS data transfers among its distributed ... · 12/04/2011 J. Flix –Improving CMS Data Transfers –EGI User Forum (Vilnius April 2011) 1 Improving CMS data transfers

J. Flix – Improving CMS Data Transfers – EGI User Forum (Vilnius April 2011)12/04/2011 20

FTSMonitor

- Ind. Transfer details -

- Transfer Summary -

- Channel Config details -

We can trace down transfer problems/issues and details of those failed attemps

We can trace down if channel is properly configured for CMSAdditionally, if timeouts need to be adjusted

The most important informationwe can gather from individual successful file transfersPhEDEx aggregated plots (result of heterogeneus configs) is not enough to evaluate link throughputs

https://forge.in2p3.fr/projects/ftsmonitor/files

Page 21: Improving CMS data transfers among its distributed ... · 12/04/2011 J. Flix –Improving CMS Data Transfers –EGI User Forum (Vilnius April 2011) 1 Improving CMS data transfers

J. Flix – Improving CMS Data Transfers – EGI User Forum (Vilnius April 2011)12/04/2011 21

FTS monitor parser

‣ In CMSwe developed a tool to extract data from FTSMonitors worldwide‣ Full statistics about successful transfers are extracted daily, and summary reports

are produced (the tool treat thrughputs and overheads separately)

‣ Several views available

‣ Monthly averages of transfer rate per file and per stream on all channels, for each

point-to-point (we continuously transfer test files in most of the ~2300 links in CMS)

‣ Historical evolution of rate per stream and for each point-to-point transfer link

‣ Comparions of SRM preparation/release times for all of the transfers, at origins

and destination (available since the most recent FTM version)

‣ Identification of congested channels (in preparation)

Page 22: Improving CMS data transfers among its distributed ... · 12/04/2011 J. Flix –Improving CMS Data Transfers –EGI User Forum (Vilnius April 2011) 1 Improving CMS data transfers

J. Flix – Improving CMS Data Transfers – EGI User Forum (Vilnius April 2011)12/04/2011 22

e.g. Channel STAR-CIEMAT @ PIC FTS

~x45

Large fluctuationsMaybe due to theDestination site

Page 23: Improving CMS data transfers among its distributed ... · 12/04/2011 J. Flix –Improving CMS Data Transfers –EGI User Forum (Vilnius April 2011) 1 Improving CMS data transfers

J. Flix – Improving CMS Data Transfers – EGI User Forum (Vilnius April 2011)12/04/2011 23

e.g. Transfers from T1_IT_CNAF ���� T1_ES_PIC

Of course, point-to-point

throughputs on 1-month period

are not constant

Page 24: Improving CMS data transfers among its distributed ... · 12/04/2011 J. Flix –Improving CMS Data Transfers –EGI User Forum (Vilnius April 2011) 1 Improving CMS data transfers

J. Flix – Improving CMS Data Transfers – EGI User Forum (Vilnius April 2011)12/04/2011 24

e.g. Tier-2 exports/imports from/to PIC Tier-1

~x40

~x100

Page 25: Improving CMS data transfers among its distributed ... · 12/04/2011 J. Flix –Improving CMS Data Transfers –EGI User Forum (Vilnius April 2011) 1 Improving CMS data transfers

J. Flix – Improving CMS Data Transfers – EGI User Forum (Vilnius April 2011)12/04/2011 25

e.g. Tier-1 ↔ ↔ ↔ ↔ Tier-1 transfer rates

~200!!~x200!!

All these transfers go

Through LHCOPN

All these transfers go

Through LHCOPN

Page 26: Improving CMS data transfers among its distributed ... · 12/04/2011 J. Flix –Improving CMS Data Transfers –EGI User Forum (Vilnius April 2011) 1 Improving CMS data transfers

J. Flix – Improving CMS Data Transfers – EGI User Forum (Vilnius April 2011)12/04/2011 26

Improving Channel Configuration

‣ Using data extracted from FTS Monitoring to improve transfer operations

‣ Example: PIC����T2 exports

‣ Massive PIC�T2s transfers in early October ’10 following processing campaign

‣ Clogged by slow links on PIC-STAR FTS channel

‣ Links with low rate-per-stream identified

Page 27: Improving CMS data transfers among its distributed ... · 12/04/2011 J. Flix –Improving CMS Data Transfers –EGI User Forum (Vilnius April 2011) 1 Improving CMS data transfers

J. Flix – Improving CMS Data Transfers – EGI User Forum (Vilnius April 2011)12/04/2011 27

Improving Channel Configuration

‣ Using data extracted from FTS Monitoring to improve transfer operations

‣ Example: PIC����T2 exports

‣ Created “cloud” FTS channels for “fast” and “slow” links

‣ Improved FTS channel occupancy

‣ Increased number of transfer attempts

‣ Improved overall export throughput

Acknowledgment to M. Caubet

(PIC FTS admin)

Page 28: Improving CMS data transfers among its distributed ... · 12/04/2011 J. Flix –Improving CMS Data Transfers –EGI User Forum (Vilnius April 2011) 1 Improving CMS data transfers

J. Flix – Improving CMS Data Transfers – EGI User Forum (Vilnius April 2011)12/04/2011 28

Identifying infrastructure issues

‣ The wealth of data available allows to spot potential issues in the site or network infrastructure

‣ Example: PIC import/export asymmetry

‣ Rate-per-stream lower for exports than for imports on most links‣ Doesn’t seem to depend on distance‣ Potential site issue?‣ One possible explanation could be a known

limitation of the Solaris kernel used on the disk servers at PIC which penalizes the outgoing transfers when the other endpoint runs a Scientific Linux 5 kernel.

‣ Dedicated testing to reveal more…

Page 29: Improving CMS data transfers among its distributed ... · 12/04/2011 J. Flix –Improving CMS Data Transfers –EGI User Forum (Vilnius April 2011) 1 Improving CMS data transfers

J. Flix – Improving CMS Data Transfers – EGI User Forum (Vilnius April 2011)12/04/2011 29

Expanding the scope

‣ Gathering data for most of FTS servers included. Working on the missing atm (2 out of 8)‣ Lot of potential to improve data transfers. Now focused on T0����T1s and T1s ↔↔↔↔T1s

‣ Identify “reference” statistics and publish corresponding plots to monitor regularly ‣ For central shifters & site administrators‣ For data transfer experts – Task Force to improve data transfers in CMS

‣ Spot problems in sites and network and assist site administrators with troubleshooting‣ Include more statistics and FTS servers‣ Distributions by file size‣ Transfer preparation/release times‣ Channel occupancy‣ …

‣ PhEDEx to provide “Latency Analysis” – For T1s ↔T1s, use LHCOPN dashboard info as well

‣ The tools can be easily used by other VOs

Page 30: Improving CMS data transfers among its distributed ... · 12/04/2011 J. Flix –Improving CMS Data Transfers –EGI User Forum (Vilnius April 2011) 1 Improving CMS data transfers

J. Flix – Improving CMS Data Transfers – EGI User Forum (Vilnius April 2011)12/04/2011 30

Summary & Outlook

‣ PhEDEx ensures reliable data transfers with FTS

‣ The scale and complexity of CMS transfers has constantly increased over the years

‣ The FTS Monitor offers detailed information on transfers

‣ Extracting and analyzing transfer statistics (CMS work started by CHEP’10) provides useful insight to improve transfer operations

‣ Either re-configuring FTS channels(getting rid of them, see: gLite File Transfer Service in the EMI Data Area, MOLNAR, Zsolt, in this Conf.)

‣ Identifying performance problems @ the network level or significant OverHeads(T1↔↔↔↔T1 LHC transfers differ x200! and this is FTS-independent...)

‣ Discussion with FTS developers about transfer monitoring in future FTS versions

Acknowledgments

Lionel Schwarz from CC-IN2P3 for prompt support in fixing issues and including new features in the FTS Monitor. Local contacts and FTS server administrators at all Tier-1 s for deployment of local FTS Monitors & feedback on operating the service.