Harvey B. Newman, CaltechHarvey B. Newman, Caltech Data Analysis for Global HEP CollaborationsData Analysis for Global HEP Collaborations

LCG Launch Workshop, CERNLCG Launch Workshop, CERNl3www.cern.ch/~newman/LHCCMPerspective_hbn031102.pptl3www.cern.ch/~newman/LHCCMPerspective_hbn031102.ppt

LHC Computing Model LHC Computing Model PerspectivePerspective

To Solve: the LHC “Data Problem”To Solve: the LHC “Data Problem”

While the proposed LHC computing and data handling While the proposed LHC computing and data handling facilities are large by present-day standards,facilities are large by present-day standards,They will not support FREE access, transport or They will not support FREE access, transport or

processing for more than a minute part of the dataprocessing for more than a minute part of the data Technical Goals: Technical Goals: Ensure that the system is dimensioned, Ensure that the system is dimensioned,

configured, managed and used “optimally”configured, managed and used “optimally” Specific Problems to be Explored. How to Specific Problems to be Explored. How to

Prioritise many hundreds of requests of local and remote Prioritise many hundreds of requests of local and remote communities, consistent with Collaboration policiescommunities, consistent with Collaboration policies

Develop Strategies to Simultaneously ensure:Develop Strategies to Simultaneously ensure:Acceptable turnaround times; Efficient resource use Acceptable turnaround times; Efficient resource use

Balance proximity to large computational and data Balance proximity to large computational and data handling facilities, against proximity to end users and handling facilities, against proximity to end users and more local resources (for frequently-accessed datasets)more local resources (for frequently-accessed datasets)

MONARC: CMS Analysis MONARC: CMS Analysis ProcessProcess

Hierarchy of Processes (Experiment, Analysis Groups,Individuals)Hierarchy of Processes (Experiment, Analysis Groups,Individuals)

SelectionSelection

Iterative selectionIterative selectionOnce per monthOnce per month

~20 Groups’~20 Groups’ActivityActivity

(10(109 9 101077 events) events)

Trigger based andTrigger based andPhysics basedPhysics basedrefinementsrefinements

25 25 SI95sec/eventSI95sec/event~20 jobs per ~20 jobs per

monthmonth

25 25 SI95sec/eventSI95sec/event~20 jobs per ~20 jobs per

monthmonth

AnalysisAnalysisDifferent Physics cutsDifferent Physics cuts

& MC comparison& MC comparison~Once per day~Once per day

~25 Individual~25 Individualper Groupper GroupActivityActivity

(10(1066 –10 –1077 events) events)

Algorithms Algorithms applied to applied to

datadatato get to get resultsresults

10 SI95sec/event10 SI95sec/event~500 jobs per ~500 jobs per

dayday

10 SI95sec/event10 SI95sec/event~500 jobs per ~500 jobs per

dayday

Monte CarloMonte Carlo5000 5000

SI95sec/eventSI95sec/event5000 5000

SI95sec/eventSI95sec/event

RAW DataRAW Data

ReconstructionReconstruction Re-processingRe-processing3 Times per year3 Times per year

Experiment-Experiment-Wide ActivityWide Activity(10(1099 events) events)

New detector New detector calibrationscalibrations

Or understandingOr understanding

3000 3000 SI95sec/eventSI95sec/event

1 job year1 job year

3000 3000 SI95sec/eventSI95sec/event

1 job year1 job year

3000 3000 SI95sec/eventSI95sec/event3 jobs per year3 jobs per year

3000 3000 SI95sec/eventSI95sec/event3 jobs per year3 jobs per year

Requirements IssuesRequirements Issues

Some significant aspects of the LHC Computing Models Some significant aspects of the LHC Computing Models Need further studyNeed further study

A highly ordered analysis process: assumed relatively A highly ordered analysis process: assumed relatively little re-reconstruction and event selection on demandlittle re-reconstruction and event selection on demand

Restricted direct data flows from Tiers 0 and 1 to Tiers 3 and 4Restricted direct data flows from Tiers 0 and 1 to Tiers 3 and 4 Efficiency of use of CPU and storage with a real workloadEfficiency of use of CPU and storage with a real workload Pressure to store more dataPressure to store more data

More data per Reconstructed Event More data per Reconstructed Event Higher DAQ recording rateHigher DAQ recording rate Simulated data: produced at many remote sites;Simulated data: produced at many remote sites;

eventually stored and accessed at CERNeventually stored and accessed at CERN Tendency to greater CPU (as code and computers progress)Tendency to greater CPU (as code and computers progress)

~3000 SI95-sec to fully reconstruct (CMS ORCA Production)~3000 SI95-sec to fully reconstruct (CMS ORCA Production) To 20 SI95-sec to analyze To 20 SI95-sec to analyze

B Physics: Samples of 1 to Several X 10B Physics: Samples of 1 to Several X 108 8 Events;Events; MONARC CMS/ATLAS Studies assume typically 10 MONARC CMS/ATLAS Studies assume typically 1077

(aimed at high p(aimed at high pTT physics) physics)

Role of SimulationRole of Simulationfor Distributed Systemsfor Distributed Systems

SIMULATIONS: Widely recognized as essential toolsSIMULATIONS: Widely recognized as essential tools for the design, performance evaluation and optimisation for the design, performance evaluation and optimisation

of complex distributed systemsof complex distributed systems From battlefields to agriculture; from the factory floor From battlefields to agriculture; from the factory floor

to telecommunications systemsto telecommunications systems Very different from HEP “Monte Carlos”Very different from HEP “Monte Carlos”

““Time” intervals and interrupts are the essentialsTime” intervals and interrupts are the essentials Simulations with an appropriate high level of abstraction Simulations with an appropriate high level of abstraction

are required to represent large systems with complex are required to represent large systems with complex behaviorbehavior

Just started to be part of the HEP cultureJust started to be part of the HEP culture Experience in trigger, online and tightly coupledExperience in trigger, online and tightly coupled

computing systems: CERN CS2 modelscomputing systems: CERN CS2 models MONARC (Process-Oriented; Java Threads) ExperienceMONARC (Process-Oriented; Java Threads) Experience

Simulation is vital to evaluate and optimize the LHC CMSimulation is vital to evaluate and optimize the LHC CM And to design & optimise the Grid services themselvesAnd to design & optimise the Grid services themselves

Some “Large” Grid Issues:Some “Large” Grid Issues: to be Simulated and Studied to be Simulated and Studied

Consistent transaction managementConsistent transaction management Query (task completion time) estimationQuery (task completion time) estimation Queueing and co-scheduling strategiesQueueing and co-scheduling strategies Load balancing (e.g. Self Organizing Neural Network)Load balancing (e.g. Self Organizing Neural Network) Error Recovery: Fallback and Redirection StrategiesError Recovery: Fallback and Redirection Strategies Strategy for use of tapesStrategy for use of tapes Extraction, transport and caching of physicists’ Extraction, transport and caching of physicists’

object-collections; Grid/Database Integrationobject-collections; Grid/Database Integration Policy-driven strategies for resource sharing Policy-driven strategies for resource sharing

among sites and activities; policy/capability tradeoffsamong sites and activities; policy/capability tradeoffs Network Peformance and Problem Handling Network Peformance and Problem Handling

Monitoring and Response to BottlenecksMonitoring and Response to Bottlenecks Configuration and Use of New-Technology Networks Configuration and Use of New-Technology Networks

e.g. Dynamic Wavelength Scheduling or Switchinge.g. Dynamic Wavelength Scheduling or Switching Fault-Tolerance, Performance of the Grid Services Fault-Tolerance, Performance of the Grid Services

Architecture Architecture

Transatlantic Net WG (HN, L. Price)Transatlantic Net WG (HN, L. Price) Bandwidth Requirements [*] Bandwidth Requirements [*]

2001 2002 2003 2004 2005 2006

CMS 100 200 300 600 800 2500

ATLAS 50 100 300 600 800 2500

BaBar 300 600 1100 1600 2300 3000

CDF 100 300 400 2000 3000 6000

D0 400 1600 2400 3200 6400 8000

BTeV 20 40 100 200 300 500

DESY 100 180 210 240 270 300

US-CERN 310 622 1250 2500 5000 10000 [*] [*] Installed BW. Maximum Link Occupancy 50% Installed BW. Maximum Link Occupancy 50%

AssumedAssumedThe Network Challenge is Shared by Both Next- The Network Challenge is Shared by Both Next-

and Present Generation Experimentsand Present Generation ExperimentsSee http://gate.hep.anl.gov/lprice/TAN

Gbps Network Issues & ChallengesGbps Network Issues & Challenges

Requirements for High ThroughputRequirements for High Throughput Packet Loss must be ~Zero (10Packet Loss must be ~Zero (10-6 -6 and Below for Large Flows)and Below for Large Flows)

I.e. No “Commodity” networksI.e. No “Commodity” networks Need to track down packet lossNeed to track down packet loss

No Local infrastructure bottlenecksNo Local infrastructure bottlenecks Gigabit Ethernet “clear paths” between selected host pairs Gigabit Ethernet “clear paths” between selected host pairs

needed now; To 10 Gbps Ethernet by ~2003 or 2004needed now; To 10 Gbps Ethernet by ~2003 or 2004 TCP/IP stack configuration and tuning Absolutely RequiredTCP/IP stack configuration and tuning Absolutely Required

Large Windows; Possibly Multiple StreamsLarge Windows; Possibly Multiple Streams New Concepts of New Concepts of Fair UseFair Use Must then be Developed Must then be Developed

Careful Router configuration; monitoring Careful Router configuration; monitoring Server and Client CPU, I/O and NIC throughput sufficientServer and Client CPU, I/O and NIC throughput sufficient

End-to-endEnd-to-end monitoring and tracking of performance monitoring and tracking of performance Close collaboration with local and “regional” network staffsClose collaboration with local and “regional” network staffs

TCP Does Not Scale to the 1-10 Gbps RangeTCP Does Not Scale to the 1-10 Gbps Range

New Technologies: Lambdas, MPLS, Lambda Switching New Technologies: Lambdas, MPLS, Lambda Switching Security and Firewall PerformanceSecurity and Firewall Performance

Tier0-Tier1 Link Requirements Estimate: Hoffmann Report 2001

Tier0-Tier1 Link Requirements Estimate: Hoffmann Report 2001

1) Tier1 1) Tier1 Tier0 Data Flow for Analysis Tier0 Data Flow for Analysis 0.5 - 1.0 Gbps0.5 - 1.0 Gbps

2) Tier2 2) Tier2 Tier0 Data Flow for Analysis Tier0 Data Flow for Analysis 0.2 - 0.5 Gbps0.2 - 0.5 Gbps

3) Interactive Collaborative Sessions (30 Peak) 3) Interactive Collaborative Sessions (30 Peak) 0.1 - 0.3 Gbps0.1 - 0.3 Gbps

4) Remote Interactive Sessions (30 Flows Peak) 4) Remote Interactive Sessions (30 Flows Peak) 0.1 - 0.2 Gbps0.1 - 0.2 Gbps5) Individual (Tier3 or Tier4) data transfers 5) Individual (Tier3 or Tier4) data transfers 0.8 Gbps0.8 Gbps (Limit to 10 Flows of 5 MBytes/sec each) (Limit to 10 Flows of 5 MBytes/sec each)

TOTAL Per Tier0 - Tier1 Link TOTAL Per Tier0 - Tier1 Link 1.7 - 2.8 Gbps1.7 - 2.8 Gbps

NOTE:NOTE: Adopted Baseline by the LHC Experiments; Adopted Baseline by the LHC Experiments;

Given in the Hoffmann Steering Committee Report: Given in the Hoffmann Steering Committee Report:

““1.5 - 3 Gbps per experiment”1.5 - 3 Gbps per experiment”

Tier0-Tier1 BW Requirements Estimate: Hoffmann Report 2001

Tier0-Tier1 BW Requirements Estimate: Hoffmann Report 2001

Scoped for 100Hz X 1 MB Data Recording Scoped for 100Hz X 1 MB Data Recording (CMS and ATLAS) (CMS and ATLAS)

Does Not Allow Fast Download to Tier3+4 Does Not Allow Fast Download to Tier3+4 of “Small” Object Collectionsof “Small” Object Collections Example: Download 10Example: Download 1077 Events of AODs (10 Events of AODs (104 4 Bytes Each) Bytes Each)

100 GB; At 5 Mbytes/sec per person that’s 6 Hours ! 100 GB; At 5 Mbytes/sec per person that’s 6 Hours ! Still a bottoms-up, static, and hence Conservative Model. Still a bottoms-up, static, and hence Conservative Model.

A Dynamic Grid system with Caching, Co-scheduling, A Dynamic Grid system with Caching, Co-scheduling, and Pre-Emptive data movement may require greater and Pre-Emptive data movement may require greater bandwidthbandwidth

Does Not Include “Virtual Data” operations: Derived Does Not Include “Virtual Data” operations: Derived Data Copies; DB and Data-description overheadsData Copies; DB and Data-description overheads

Network Requirements will evolve as network Network Requirements will evolve as network technologies and prices advancetechnologies and prices advance

HENP Related Data Grid HENP Related Data Grid ProjectsProjects

ProjectsProjects PPDG IPPDG I USAUSA DOEDOE $2M$2M 1999-20011999-2001 GriPhyNGriPhyN USAUSA NSFNSF $11.9M + $1.6M$11.9M + $1.6M 2000-20052000-2005 EU DataGridEU DataGrid EUEU ECEC €10M€10M 2001-20042001-2004 PPDG II (CP)PPDG II (CP) USAUSA DOEDOE $9.5M$9.5M 2001-20042001-2004 iVDGLiVDGL USAUSA NSFNSF $13.7M + $2M$13.7M + $2M 2001-20062001-2006 DataTAGDataTAG EUEU ECEC €4M€4M 2002-20042002-2004 GridPP GridPP UKUK PPARCPPARC >$15M>$15M 2001-20042001-2004 LCG Phase1LCG Phase1 CERN MSCERN MS 30 MCHF30 MCHF 2002-20042002-2004

Many Other Projects of interest to HENPMany Other Projects of interest to HENP Initiatives in US, UK, Italy, France, NL, Germany, Japan, …Initiatives in US, UK, Italy, France, NL, Germany, Japan, … US and EU networking initiatives: US and EU networking initiatives: AMPATH, I2, DataTAG AMPATH, I2, DataTAG US Distributed Terascale Facility: US Distributed Terascale Facility:

($53M, 12 TeraFlops, 40 Gb/s network)($53M, 12 TeraFlops, 40 Gb/s network)

CMS Milestones: In Depth Design & CMS Milestones: In Depth Design & Data Challenges 1999-2007Data Challenges 1999-2007

Trigger (Filter) Studies: 1999-2001Trigger (Filter) Studies: 1999-2001 November 2000: Level 1 Trigger TDR (Completed)November 2000: Level 1 Trigger TDR (Completed)

Large-scale productions for L1 trigger studiesLarge-scale productions for L1 trigger studies Dec 2002: DAQ TDRDec 2002: DAQ TDR

Continue High Level Trigger studies; Production Continue High Level Trigger studies; Production at Prototype Tier0, Tier1s and Tier2sat Prototype Tier0, Tier1s and Tier2s

Dec 2003: Core Software and Computing TDR Dec 2003: Core Software and Computing TDR First large-scale Data Challenge (5%)First large-scale Data Challenge (5%) Use full chain from online farms to production Use full chain from online farms to production

in Tier0, 1, 2 centersin Tier0, 1, 2 centers Dec 2004: Physics TDR Dec 2004: Physics TDR

Test physics performance, with large amount of dataTest physics performance, with large amount of data Verify technology choices with distributed analysisVerify technology choices with distributed analysis

Dec 2004: Second large-scale Data Challenge (20%)Dec 2004: Second large-scale Data Challenge (20%) Final test of scalability of the fully distributed CMS Final test of scalability of the fully distributed CMS

computing system before production system purchasecomputing system before production system purchase Fall 2006: Computing, database and Grid systems in Fall 2006: Computing, database and Grid systems in

place. Commission for LHC Startupplace. Commission for LHC Startup Apr. 2007: All Systems Ready for First LHC RunsApr. 2007: All Systems Ready for First LHC Runs

Ongoing Study of the Model: Evolving with Ongoing Study of the Model: Evolving with Experience and Advancing Technologies Experience and Advancing Technologies

Requirements Requirements Site components and architecturesSite components and architectures Networks: technology, scale, operationsNetworks: technology, scale, operations High Level Software Services architecture: High Level Software Services architecture:

Scalable and resilient Scalable and resilient loosely coupled, adaptive, loosely coupled, adaptive, partly autonomous, e.g. agent-based partly autonomous, e.g. agent-based

Operational Modes (Develop a Common Understanding ?)Operational Modes (Develop a Common Understanding ?) What are the technical goals + emphasis of the What are the technical goals + emphasis of the

systemsystem How is it intended to be used by the Collaboration ?How is it intended to be used by the Collaboration ?

e.g. What are guidelines and steps that make up the e.g. What are guidelines and steps that make up the data access/processing/analysis policy and data access/processing/analysis policy and

strategystrategy

Note: Common services imply somewhat similar op. modesNote: Common services imply somewhat similar op. modes

The LHC Distributed ComputingThe LHC Distributed ComputingModel: from Here ForwardModel: from Here Forward

Agent-Based Distributed Services: JINI Prototype (Caltech/Pakistan)

Agent-Based Distributed Services: JINI Prototype (Caltech/Pakistan)

Includes “Station Servers” (static) that Includes “Station Servers” (static) that host mobile “Dynamic Services”host mobile “Dynamic Services”

Servers are interconnected dynamically Servers are interconnected dynamically to form a fabric in which mobile agents to form a fabric in which mobile agents travel, with a payload of physics travel, with a payload of physics analysis tasksanalysis tasks

Prototype is highly flexible and Prototype is highly flexible and robust against network outagesrobust against network outages

Adaptable to WSDL-based services: Adaptable to WSDL-based services: OGSA; and to many platformsOGSA; and to many platforms

The Design and Studies with this The Design and Studies with this prototype use the MONARC prototype use the MONARC Simulator, and build on SONN Simulator, and build on SONN studies. Seestudies. See

http://home.cern.ch/clegrand/lia/http://home.cern.ch/clegrand/lia/

StationStationServerServer

StationStationServerServer

StationStationServerServer

LookupLookupServiceService

LookupLookupServiceService

Proxy ExchangeProxy Exchange

Registration

Registration

Service Listener

Service Listener

Lookup Lookup Discovery Discovery

ServiceService

Remote Notification

Remote Notification

Grid projects have been a step forward for HEP and Grid projects have been a step forward for HEP and LHC: a path to meet the “LHC Computing” challengesLHC: a path to meet the “LHC Computing” challenges

But: the differences between HENP Grids and But: the differences between HENP Grids and classical Grids are not yet fully appreciated classical Grids are not yet fully appreciated

The original Computational and Data Grid concepts are The original Computational and Data Grid concepts are largely stateless, open systems: known to be scalablelargely stateless, open systems: known to be scalable

Analogous to the WebAnalogous to the Web The classical Grid architecture has a number of implicit The classical Grid architecture has a number of implicit

assumptionsassumptions The ability to locate and schedule suitable resources,The ability to locate and schedule suitable resources,

within a tolerably short time (i.e. resource within a tolerably short time (i.e. resource richness)richness) Short transactions; Relatively simple failure modesShort transactions; Relatively simple failure modes

HEP Grids are data-intensive and resource constrainedHEP Grids are data-intensive and resource constrained Long transactions; some long queuesLong transactions; some long queues Schedule conflicts; policy decisions; task redirectionSchedule conflicts; policy decisions; task redirection A Lot of global system state to be monitored+trackedA Lot of global system state to be monitored+tracked

LHC Distributed CM: HENP Data LHC Distributed CM: HENP Data Grids Versus Classical GridsGrids Versus Classical Grids

Upcoming Grid Challenges: SecureWorkflow Management and Optimization

Upcoming Grid Challenges: SecureWorkflow Management and Optimization

Maintaining a Maintaining a Global ViewGlobal View of Resources and System State of Resources and System State End-to-end System MonitoringEnd-to-end System Monitoring Adaptive Learning: new paradigms for execution Adaptive Learning: new paradigms for execution

optimization (eventually automated)optimization (eventually automated) Workflow Management,Workflow Management, Balancing Policy Versus Balancing Policy Versus

Moment-to-moment Capability to Complete TasksMoment-to-moment Capability to Complete Tasks Balance High Levels of Usage of Limited Resources Balance High Levels of Usage of Limited Resources

Against Better Turnaround Times for Priority JobsAgainst Better Turnaround Times for Priority Jobs Goal-Oriented; SteeringGoal-Oriented; Steering Requests According to Requests According to

(Yet to be Developed) Metrics(Yet to be Developed) Metrics Robust Grid Transactions In a Multi-User EnvironmentRobust Grid Transactions In a Multi-User Environment Realtime Error Detection, RecoveryRealtime Error Detection, Recovery

Handling User-Grid Interactions: Guidelines; AgentsHandling User-Grid Interactions: Guidelines; Agents Building Higher Level Services, and an IntegratedBuilding Higher Level Services, and an Integrated

User Environment for the AboveUser Environment for the Above

foster@mcs.anl.gov ARGONNE CHICAGO

Grid Architecture

“Talking to things”: Communication (Internet protocols) & security

“Sharing single resources”: Negotiating access, controlling use

“Coordinating multiple resources”: ubiquitous infrastructure services, app-specific distributed services

“Controlling things locally”: Access to, & control of resources

Connectivity

Resource

Collective

Application

Fabric

Internet

Transport

Appli-cation

Link

Inte

rnet P

roto

col

Arc

hite

ctu

re

More info: www.globus.org/research/papers/anatomy.pdf

Physicists’ Application CodesPhysicists’ Application Codes Reconstruction, Calibration, AnalysisReconstruction, Calibration, Analysis

Experiments’ Software Framework LayerExperiments’ Software Framework Layer Modular and Grid-aware: Architecture able to interact Modular and Grid-aware: Architecture able to interact

effectively with the lower layers (above) effectively with the lower layers (above) Grid Applications LayerGrid Applications Layer

(Parameters and algorithms that govern system operations)(Parameters and algorithms that govern system operations) Policy and priority metricsPolicy and priority metrics Workflow evaluation metricsWorkflow evaluation metrics Task-Site Coupling proximity metricsTask-Site Coupling proximity metrics

Global End-to-End System Services LayerGlobal End-to-End System Services Layer Monitoring and Tracking Component performanceMonitoring and Tracking Component performance Workflow monitoring and evaluation mechanismsWorkflow monitoring and evaluation mechanisms Error recovery and redirection mechanismsError recovery and redirection mechanisms System self-monitoring, evaluation and System self-monitoring, evaluation and

optimisationoptimisation mechanisms mechanisms

HENP Grid Architecture: HENP Grid Architecture: Layers Above the Collective Layer Layers Above the Collective Layer

GGF4 (Feb. 2002): Presentation of the OGSA (Draft)GGF4 (Feb. 2002): Presentation of the OGSA (Draft) See http://www.globus.org/research/papers/ogsa.pdf See http://www.globus.org/research/papers/ogsa.pdf

Uniform Grid Services are definedUniform Grid Services are defined Defines standard mechanisms for creating, naming Defines standard mechanisms for creating, naming and discovering transient Grid services and discovering transient Grid services Defines Web-service (WSDL) interfaces, conventions Defines Web-service (WSDL) interfaces, conventions

and mechanisms to build the basic servicesand mechanisms to build the basic services As required for composing sophisticated As required for composing sophisticated

distributed systems distributed systems Expresses the intent to provide higher level standard Expresses the intent to provide higher level standard

services: for distributed data management; workflow; services: for distributed data management; workflow; auditing; instrumentation and monitoring; problem auditing; instrumentation and monitoring; problem

determination for distributed computing, security determination for distributed computing, security protocol mapping protocol mapping

Adoption of the Web-services approach by a broad Adoption of the Web-services approach by a broad range of major industrial players, most notably IBMrange of major industrial players, most notably IBM

The Evolution of Global Grid The Evolution of Global Grid StandardsStandards

The emergence of a standard Web-services based The emergence of a standard Web-services based architecture (OGSA) is a major step forwardarchitecture (OGSA) is a major step forward

But we have to consider a number of practical factors:But we have to consider a number of practical factors: Schedule of Emerging Standards relative to the Schedule of Emerging Standards relative to the

LHC Experiments’ Schedule and MilestonesLHC Experiments’ Schedule and Milestones Availability and functionality of standard servicesAvailability and functionality of standard services as a function of time as a function of time Extent and scope of the standard servicesExtent and scope of the standard services

Basic services will be standardizedBasic services will be standardized Industry will compete over tools and higherIndustry will compete over tools and higher

level services built on top of the basic serviceslevel services built on top of the basic services Major vendors are not in the business of vertically Major vendors are not in the business of vertically integrated applications (for the community)integrated applications (for the community)

Question at GGF4: Who builds the distributed system, Question at GGF4: Who builds the distributed system, with sufficient intelligence and functionality to meet our with sufficient intelligence and functionality to meet our needs ? needs ?

Answer: Answer: You Do.You Do.

The Evolution of Grid StandardsThe Evolution of Grid Standardsand the LHC/HENP Grid Taskand the LHC/HENP Grid Task

Focus on End-to-End integration and deployment of experiment Focus on End-to-End integration and deployment of experiment applications with existing and emerging Grid servicesapplications with existing and emerging Grid services

Including the E2E and Grid Applications LayersIncluding the E2E and Grid Applications Layers Collaborative development of Grid middleware and extensions Collaborative development of Grid middleware and extensions

between application and middleware groupsbetween application and middleware groups Leading to pragmatic and acceptable-risk solutionsLeading to pragmatic and acceptable-risk solutions

Grid technologies and services need to be deployedGrid technologies and services need to be deployedin production (24x7) environments in production (24x7) environments

Meeting experiments’ Milestones Meeting experiments’ Milestones With stressful performance needsWith stressful performance needs Services that work; increasing functionality at each stage Services that work; increasing functionality at each stage

as an integral part of the development process as an integral part of the development process We need toWe need to adopt common basic security and information adopt common basic security and information

infrastructures, and basic components sooninfrastructures, and basic components soon Move on to tackle the LHC “Computing Problem” as a wholeMove on to tackle the LHC “Computing Problem” as a whole

Develop the network-distributed data analysis and Develop the network-distributed data analysis and collaborative collaborative systems systems To meet the needs of the global LHC Collaborations To meet the needs of the global LHC Collaborations

The LHC “Computing Problem” and The LHC “Computing Problem” and Grid R&D/Deployment StrategyGrid R&D/Deployment Strategy

Some Extra Some Extra

Slides FollowSlides Follow

Computing Challenges: Computing Challenges: Petabyes, Petaflops, Global VOsPetabyes, Petaflops, Global VOs

Geographical dispersion:Geographical dispersion: of people and resources of people and resources Complexity:Complexity: the detector and the LHC environment the detector and the LHC environment Scale: Scale: Tens of Petabytes per year of dataTens of Petabytes per year of data

5000+ Physicists 250+ Institutes 60+ Countries

Major challenges associated with:Major challenges associated with:Communication and collaboration at a distanceCommunication and collaboration at a distance

Managing globally distributed computing & data resources Managing globally distributed computing & data resources Remote software development and physics analysisRemote software development and physics analysisR&D: New Forms of Distributed Systems: Data GridsR&D: New Forms of Distributed Systems: Data Grids

LHC Data Grid HierarchyLHC Data Grid Hierarchy

Tier 1

Tier2 Center

Online System

CERN 700k SI95 ~1 PB Disk; Tape Robot

FNAL: 200k SI95; 600 TBIN2P3 Center INFN Center RAL Center

InstituteInstituteInstituteInstitute ~0.25TIPS

Workstations

~100-400 MBytes/sec

2.5 Gbps

100 - 1000

Mbits/sec

Physicists work on analysis “channels”

Each institute has ~10 physicists working on one or more channels

Physics data cache

~PByte/sec

~2.5 Gbits/sec

Tier2 CenterTier2 CenterTier2 Center~2.5 Gbps

Tier 0 +1

Tier 3

Tier 4

Tier2 Center Tier 2

Experiment

CERN/Outside Resource Ratio ~1:2Tier0/( Tier1)/( Tier2) ~1:1:1

Why Worldwide Computing? Why Worldwide Computing? Regional Center Concept Regional Center Concept

Maximize total funding resources to meet the Maximize total funding resources to meet the total computing and data handling needstotal computing and data handling needs

An N-Tiered Model: for fair-shared access An N-Tiered Model: for fair-shared access for Physicists everywherefor Physicists everywhere Smaller size, greater control as N increases Smaller size, greater control as N increases

Utilize all intellectual resources, & expertise in Utilize all intellectual resources, & expertise in all time zonesall time zones Involving students and physicists at home Involving students and physicists at home

universities and labsuniversities and labs Greater flexibility to pursue different physics interests, Greater flexibility to pursue different physics interests,

priorities, and resource allocation strategies by regionpriorities, and resource allocation strategies by region And/or by Common Interest: physics topics, subdetectors,And/or by Common Interest: physics topics, subdetectors,

…… Manage the System’s ComplexityManage the System’s Complexity

Partitioning facility tasks, to manage & focus resourcesPartitioning facility tasks, to manage & focus resources Efficient use of network: higher throughputEfficient use of network: higher throughput

Per Flow: Local > regional > national > internationalPer Flow: Local > regional > national > international

MONARC: Project at CERNMONARC: Project at CERN

MModels odels OOf f NNetworked etworked AAnalysis nalysis At At RRegional egional CCentersenters

Caltech, CERN, Columbia, FNAL, Heidelberg, Caltech, CERN, Columbia, FNAL, Heidelberg, Helsinki, INFN, IN2P3, KEK, Marseilles, MPI Helsinki, INFN, IN2P3, KEK, Marseilles, MPI

Munich, Orsay, Oxford, TuftsMunich, Orsay, Oxford, Tufts

PROJECT GOALS ACHIEVEDPROJECT GOALS ACHIEVED Developed LHC “Baseline Models”Developed LHC “Baseline Models” Specified the main parameters Specified the main parameters

characterizing the Model’s characterizing the Model’s performance: throughputs, latencies performance: throughputs, latencies

Established resource requirement baselines: Established resource requirement baselines: Computing, Data handling, Networks Computing, Data handling, Networks

TECHNICAL GOALSTECHNICAL GOALS Defined the baseline Defined the baseline Analysis ProcessAnalysis Process Defined Defined RC Architectures and ServicesRC Architectures and Services Provided Provided Guidelines for the final ModelsGuidelines for the final Models Provided a Provided a Simulation Toolset Simulation Toolset for Further for Further

Model studiesModel studies

2.5

Gb

ps

2.5 Gbps

Univ 2

CERN~700k SI95 1000+ TB

Disk; Robot

Tier2 Ctr~50k SI95 ~100 TB

Disk Robot

FNAL/BNL~200k SI95650 Tbyte

Disk; Robot

2.5

Gbp

s

N X

2.5

Gb

ps

2.5 Gbps

2.5 Gbps

Univ1

UnivM

Model Circa Model Circa 20062006

MONARC History MONARC History

Spring 1998 Spring 1998 First Distributed Center Models (Bunn; Von Praun)First Distributed Center Models (Bunn; Von Praun) 6/19986/1998 Presentation to LCB; Project Assignment PlanPresentation to LCB; Project Assignment Plan Summer 1998Summer 1998 MONARC Project Startup (ATLAS, CMS, LHCb)MONARC Project Startup (ATLAS, CMS, LHCb) 9 - 10/19989 - 10/1998 Project Execution Plan; Approved by LCBProject Execution Plan; Approved by LCB 1/19991/1999 First Analysis Process to be ModeledFirst Analysis Process to be Modeled 2/19992/1999 First Java Based Simulation Models (I. Legrand)First Java Based Simulation Models (I. Legrand) Spring 1999Spring 1999 Java2 Based Simulations; GUIJava2 Based Simulations; GUI 4/99; 8/99; 12/994/99; 8/99; 12/99 Regional Centre Representative MeetingsRegional Centre Representative Meetings 6/19996/1999 Mid-Project Progress ReportMid-Project Progress Report

Including MONARC Baseline ModelsIncluding MONARC Baseline Models 9/19999/1999 Validation of MONARC Simulation on TestbedsValidation of MONARC Simulation on Testbeds

Reports at LCB Workshop (HN, I. Legrand)Reports at LCB Workshop (HN, I. Legrand) 1/20001/2000 Phase 3 Letter of Intent (4 LHC Experiments)Phase 3 Letter of Intent (4 LHC Experiments) 2/20002/2000 Papers and Presentations at CHEP2000:Papers and Presentations at CHEP2000:

D385, F148, D127, D235, C113, C169 D385, F148, D127, D235, C113, C169 3/20003/2000 Phase 2 ReportPhase 2 Report Spring 2000Spring 2000 New Tools: SNMP-based Monitoring; S.O.M.New Tools: SNMP-based Monitoring; S.O.M. 5/20005/2000 Phase 3 Simulation of ORCA4 Production;Phase 3 Simulation of ORCA4 Production;

Begin Studies with TapesBegin Studies with Tapes Spring 2000Spring 2000 MONARC Model Recognized by Hoffmann WWC Panel;MONARC Model Recognized by Hoffmann WWC Panel;

Basis of Data Grid Efforts in US and Europe Basis of Data Grid Efforts in US and Europe

MONARC Key FeaturesMONARC Key Featuresfor a Successful Projectfor a Successful Project

The broad based nature of the collaboration: LHC experiments, The broad based nature of the collaboration: LHC experiments, regional representatives, covering different local conditions and a regional representatives, covering different local conditions and a range of estimated financial meansrange of estimated financial means

The choice of the process-oriented discrete event simulation The choice of the process-oriented discrete event simulation approach backed up by testbeds, allowing to simulate accuratelyapproach backed up by testbeds, allowing to simulate accurately

a complex set of networked Tier0/Tier1/Tier2 Centresa complex set of networked Tier0/Tier1/Tier2 Centres the analysis process: a dynamic workload of reconstruction the analysis process: a dynamic workload of reconstruction

and analysis jobs submitted to job schedulers, and and analysis jobs submitted to job schedulers, and then to then to multi-tasking compute and data serversmulti-tasking compute and data servers

the behavior of key elements of the system, such as distributed the behavior of key elements of the system, such as distributed database servers and networks database servers and networks

The design of the simulation system, with an appropriate level The design of the simulation system, with an appropriate level of abstraction, allowing it to be CPU and memory-efficientof abstraction, allowing it to be CPU and memory-efficient

The use of prototyping on the testbeds to ensure the The use of prototyping on the testbeds to ensure the simulation is capable of providing accurate resultssimulation is capable of providing accurate results

Organization into four technical working groupsOrganization into four technical working groups Incorporation of the Regional Centres CommitteeIncorporation of the Regional Centres Committee

Major StepsMajor Steps Conceptualize, profile and parameterize workloadsConceptualize, profile and parameterize workloads

and their time-behaviorsand their time-behaviors Develop and parameterize schemes for taskDevelop and parameterize schemes for task

prioritization, coupling tasks to sitesprioritization, coupling tasks to sites Simulate individual Grid services & transaction Simulate individual Grid services & transaction

behaviorbehavior Develop/test error recovery and fallback strategiesDevelop/test error recovery and fallback strategies

Handle an increasingly rich set of “situations”Handle an increasingly rich set of “situations” (failures) as the Grid system and workload scales (failures) as the Grid system and workload scales

Learn from experiments’ Data Challenge MilestonesLearn from experiments’ Data Challenge Milestones Also study: Also study: Grid-Enabled User Analysis EnvironmentsGrid-Enabled User Analysis Environments

““MONARC” Simulations and MONARC” Simulations and LHC CM Development LHC CM Development

Design Considerations of the Design Considerations of the MONARC Simulation System MONARC Simulation System

This simulation project is based on Java2(TM) technology which provides adequate tools for developing a flexible and distributed process oriented simulation. Java has built-in multi-thread support for concurrent processing, which can be used for simulation purposes by providing a dedicated scheduling mechanism.

The distributed objects support (through RMI or CORBA) can be used on distributed simulations, or for an environment in which parts of the system are simulated and interfaced through such a mechanism with other parts which actually are running the real application.

A PROCESS ORIENTED APPROACH for discrete event simulation

is well-suited to describe concurrent running tasks“Active objects” (having an execution thread, a program counter,

stack...) provide an easy way to map the structure of a set of distributed running programs into the simulation environment.

F148

Multitasking Processing ModelMultitasking Processing Model

“Interrupt” driven scheme: For each new task or when one task is finished, an interrupt is

generated and all “times to completion” are recomputed.

It provides:

An easy way to apply different load balancing schemes

An efficient mechanism to simulate multitask processing

Assign active tasks (CPU, I/O, network) to Java threads Concurrent running tasks share resources (CPU, memory, I/O)

Example : Physics Analysis at Example : Physics Analysis at Regional CentresRegional Centres

Similar data processingSimilar data processing

jobs are performed in jobs are performed in each of several RCs each of several RCs

There is profile of jobs,There is profile of jobs,each submitted to a job each submitted to a job schedulerscheduler

Each Centre has “TAG”Each Centre has “TAG”and “AOD” databases and “AOD” databases replicated.replicated.

Main Centre provides Main Centre provides “ESD” and “RAW” data “ESD” and “RAW” data

Each job processes Each job processes AOD data, and also aAOD data, and also aa fraction of ESD and a fraction of ESD and RAW data.RAW data.

Modelling and understanding networked regional center Modelling and understanding networked regional center configurations, their performance and limitations, is essential configurations, their performance and limitations, is essential for the design of large scale distributed systems.for the design of large scale distributed systems.

The simulation system developed in MONARCThe simulation system developed in MONARC ( (MModels odels OOf f NNetworked etworked AAnalysis At nalysis At RRegional egional CCenters), enters), based on a process based on a process oriented approach to discrete event simulation using oriented approach to discrete event simulation using JavaJava(TM)(TM) technology, provides technology, provides a scalable tool for realistic modelling of a scalable tool for realistic modelling of large scale distributed systemslarge scale distributed systems.

Modeling and Simulation:Modeling and Simulation:MONARC SystemMONARC System

SIMULATION of Complex Distributed Systems

MONARC SONN: 3 Regional Centres MONARC SONN: 3 Regional Centres Learning to Export Jobs (Day 9)Learning to Export Jobs (Day 9)

NUST20 CPUs

CERN30 CPUs

CALTECH25 CPUs

1MB/s ; 150 ms RTT

1.2 MB

/s

150 ms R

TT

0

.8 M

B/s

200

ms

RTT

Day = 9

<E> = 0.73

<E> = 0.66

<E> = 0.83

Links Required to US Labs and Transatlantic [*]

Links Required to US Labs and Transatlantic [*]

[*] Maximum Link Occupancy 50% Assumed[*] Maximum Link Occupancy 50% Assumed

May Indicate N X OC192 Required Into CERN By May Indicate N X OC192 Required Into CERN By 20072007

2001 2002 2003 2004 2005 2006

SLAC OC12 2 X OC12 2 X OC12 OC48 OC48 2 X OC48

BNL OC12 2 X OC12 2 X OC12 OC48 OC48 2 X OC48

FNAL OC12 OC48 2 X OC48 OC192 OC192 2 X OC192

US-CERN 2 X OC3 OC12 2 X OC12 OC48 2 X OC48 OC192

US-DESY OC3 2 X OC3 2 X OC3 2 X OC3 2 X OC3 OC12

GriPhyN: PetaScale Virtual Data GridsGriPhyN: PetaScale Virtual Data Grids

Virtual Data Tools

Request Planning &

Scheduling ToolsRequest Execution & Management Tools

Transforms

Distributed resources(code, storage,

computers, and network)

Resource Management

Services

Resource Management

Services

Security and Policy

Services

Security and Policy

Services

Other Grid ServicesOther Grid

Services

Interactive User Tools

Production TeamIndividual

Investigator Workgroups

Raw data source

Particle Physics Data GridParticle Physics Data GridCollaboratory Pilot (2001-2003)Collaboratory Pilot (2001-2003)

Computer Science Program of WorkComputer Science Program of Work CS1: Job Description LanguageCS1: Job Description Language CS2: Schedule and Manage Data CS2: Schedule and Manage Data

Processing and Placement ActivitiesProcessing and Placement Activities CS3 Monitoring and Status ReportingCS3 Monitoring and Status Reporting CS4 Storage Resource ManagementCS4 Storage Resource Management CS5 Reliable Replication ServicesCS5 Reliable Replication Services CS6 High Performance RobustCS6 High Performance Robust

File Transfer Services File Transfer Services CS7 Collect/Document Current CS7 Collect/Document Current

Experiment Practices and Potential Experiment Practices and Potential Generalizations…Generalizations…

CS9 Authent., Authorization, SecurityCS9 Authent., Authorization, Security CS10 End-to-End Apps. & TestbedsCS10 End-to-End Apps. & Testbeds

“The PPDG Collaboratory Pilot will develop, evaluate and deliver vitally needed Grid-enabled tools for data-intensive collaboration in particle and nuclear physics. Novel mechanisms and policies will be vertically integrated with Grid Middleware, experiment-specific applications and computing resources to provide effective end-to-end capability.”

TECHNICAL FOCUS: TECHNICAL FOCUS: End-to-End Applications End-to-End Applications & Integrated Production Systems, & Integrated Production Systems, With With Robust Data ReplicationRobust Data Replication Intelligent Job Placement and SchedulingIntelligent Job Placement and Scheduling Management of Storage ResourcesManagement of Storage Resources Monitoring and Information Global ServicesMonitoring and Information Global Services

METHODOLOGY: Deploy Systems Useful METHODOLOGY: Deploy Systems Useful to the Experiments to the Experiments In 24 X 7 Production Environments,In 24 X 7 Production Environments,

with Stressful Requirementswith Stressful Requirements With Increasing FunctionalityWith Increasing Functionality

at Each Round at Each Round STANDARD Grid Middleware Components STANDARD Grid Middleware Components

Integrated as they Emerge Integrated as they Emerge

PPDG: Focus and FoundationsPPDG: Focus and Foundations

CMS Production: Event Simulation CMS Production: Event Simulation

and Reconstructionand Reconstruction

““Grid-Enabled”Grid-Enabled” AutomatedAutomated

Imperial Imperial CollegeCollege

UFLUFL

Fully operationalFully operationalCaltechCaltech

PUPUNo PUNo PU

In progressIn progress

Common Common Prod. toolsProd. tools

(IMPALA)(IMPALA)GDMPGDMPDigitizationDigitization

SimulationSimulation

HelsinkiHelsinki

IN2P3IN2P3

WisconsinWisconsin

BristolBristol

UCSDUCSD

INFNINFN

MoscowMoscow

FNALFNAL

CERNCERN

Worldwide Productio

n

at 12 Sites

foster@mcs.anl.gov ARGONNE CHICAGO

GriPhyN/PPDGData Grid Architecture

GSI, CAS

MDS

MCAT; GriPhyN catalogs

Application

Planner

Executor

Catalog Services

Info Services

Policy/Security

Monitoring

Repl. Mgmt.

Reliable TransferService

Compute Resource Storage Resource

= initial solution is operational

Ian Foster, Carl Kesselman, Miron Livny, Mike Wilde, others

DAG

Tier0/1 facilityTier2 facility

10 Gbps link2.5 Gbps link622 Mbps linkOther link

Tier3

International Virtual-Data Grid Laboratory Conduct Data Grid tests “at scale” Develop Common Grid infrastructure National, international scale Data Grid

tests, leading to managed ops (iGOC)

Components Tier1, Selected Tier2 and Tier3 Sites Distributed Terascale Facility (DTF) 0.6 - 10 Gbps networks

Planned New Partners

Brazil T1 Russia T1 Pakistan T2 China T2 …

GriPhyN iVDGL Map Circa 2002-2003GriPhyN iVDGL Map Circa 2002-2003 US, UK, Italy, France, Japan, Australia US, UK, Italy, France, Japan, Australia

TeraGrid (www.teragrid.org)TeraGrid (www.teragrid.org)NCSA, ANL, SDSC, CaltechNCSA, ANL, SDSC, Caltech

NCSA/UIUC

ANL

UIC Multiple Carrier Hubs

Starlight / NW UnivStarlight / NW Univ

Ill Inst of Tech

Univ of ChicagoIndianapolis (Abilene NOC)

I-WIRE

Pasadena

San Diego

DTF Backplane(4x): 40 Gbps)

Abilene

Chicago

Indianapolis

Urbana

OC-48 (2.5 Gb/s, Abilene)Multiple 10 GbE (Qwest)Multiple 10 GbE (I-WIRE Dark Fiber)

Solid lines in place and/or available in 2001 Dashed I-WIRE lines planned for Summer 2002

Source: Charlie Catlett, Argonne

A Preview of the Grid Hierarchyand Networks of the LHC Era

Demonstration of the use of Virtual Data Demonstration of the use of Virtual Data technology for interactive CMS physics technology for interactive CMS physics analysis at Supercomputing 2001, Denveranalysis at Supercomputing 2001, Denver Interactive subsetting and analysis of Interactive subsetting and analysis of

144,000 CMS QCD events (105 GB)144,000 CMS QCD events (105 GB) Tier 4 workstation (Denver) gets data from Tier 4 workstation (Denver) gets data from

two tier 2 servers (Caltech and San Diego)two tier 2 servers (Caltech and San Diego) Prototype tool showing feasibility of these Prototype tool showing feasibility of these

CMS computing model concepts:CMS computing model concepts: Navigates from tag data to full event dataNavigates from tag data to full event data Transparently accesses `virtual' objects Transparently accesses `virtual' objects

through Grid-APIthrough Grid-API Reconstructs On-Demand Reconstructs On-Demand

(=Virtual Data materialisation)(=Virtual Data materialisation) Integrates object persistency Integrates object persistency

layer and grid layerlayer and grid layer Peak throughput achieved: 29.1 Mbyte/s;Peak throughput achieved: 29.1 Mbyte/s;

78% efficiency on 3 Fast Ethernet Ports78% efficiency on 3 Fast Ethernet Ports

Grid-enabled Data Analysis: SC2001 Demoby K. Holtman, J. Bunn (CMS/Caltech)

Grid-enabled Data Analysis: SC2001 Demoby K. Holtman, J. Bunn (CMS/Caltech)

The LHC Computing GridProject Structure

The LHC Computing Grid Project

LHCC

Project Overview Board

RTAG

Reports

Reviews

CommonComputing

RRB

Resource Matters

OtherComputing

GridProjects

EUDataGridProject

implementation teams

Other HEPGrid

Projects

OtherLabs

Project Manager

ProjectExecution

Board

Requirements,Monitoring

Software andComputingCommittee

(SC2)

Launch WorkshopMarch 11-15

Grid R&D: Focal AreasGrid R&D: Focal Areas

Development of Grid-Enabled User Analysis EnvironmentsDevelopment of Grid-Enabled User Analysis Environments Web Services (OGSA based) Web Services (OGSA based) for ubiquitous, platform for ubiquitous, platform

and OS-independent data (and code) accessand OS-independent data (and code) access Analysis Portals for Event Visualization, Data Processing Analysis Portals for Event Visualization, Data Processing

and Analysis and Analysis Simulations for Systems Modeling, OptimizationSimulations for Systems Modeling, Optimization

For example: theFor example: the MONARC MONARC System System Globally Scalable Agent-Based Realtime Information Globally Scalable Agent-Based Realtime Information

Marshalling SystemsMarshalling Systems For the next-generation challenge of DynamicFor the next-generation challenge of Dynamic

Grid design and operationsGrid design and operations Self-learning (e.g. SONN) optimization Self-learning (e.g. SONN) optimization Simulation enhanced: to monitor, track and forward Simulation enhanced: to monitor, track and forward

predict site, network and global system statepredict site, network and global system state 1-10 Gbps Networking development and deployment1-10 Gbps Networking development and deployment

Work with DataTAG, the TeraGrid, STARLIGHT, Abilene, the Work with DataTAG, the TeraGrid, STARLIGHT, Abilene, the iVDGL, iGOC, HENP Internet2 WG, Internet2 E2EiVDGL, iGOC, HENP Internet2 WG, Internet2 E2E

Global Collaboratory Development: e.g. VRVS, Virtual Access GridGlobal Collaboratory Development: e.g. VRVS, Virtual Access Grid

9352 Hosts;5369 Registered Users in 63 Countries 42 (7 I2) Reflectors Annual Growth 2.5X