paul avery university of florida [email protected]
DESCRIPTION
Integrating Universities and Laboratories In National Cyberinfrastructure. Paul Avery University of Florida [email protected]. PASI Lecture Mendoza, Argentina May 17, 2005. Outline of Talk. Cyberinfrastructure and Grids Data intensive disciplines and Data Grids - PowerPoint PPT PresentationTRANSCRIPT
PASI: Mendoza, Argentina (May 17, 2005)
Paul Avery 1
Paul AveryUniversity of [email protected]
Integrating Universities and Laboratories In National
Cyberinfrastructure
PASI LectureMendoza, Argentina
May 17, 2005
PASI: Mendoza, Argentina (May 17, 2005)
Paul Avery 2
Outline of Talk Cyberinfrastructure and Grids
Data intensive disciplines and Data Grids
The Trillium Grid collaborationGriPhyN, iVDGL, PPDG
The LHC and its computing challenges Grid3 and the Open Science Grid A bit on networks Education and Outreach Challenges for the future Summary
Presented from a physicist’s perspective!
PASI: Mendoza, Argentina (May 17, 2005)
Paul Avery 3
Cyberinfrastructure (cont)Software programs, services, instruments,
data, information, knowledge, applicable to specific projects, disciplines, and communities.
Cyberinfrastructure layer of enabling hardware, algorithms, software, communications, institutions, and personnel. A platform that empowers researchers to innovate and eventually revolutionize what they do, how they do it, and who participates.
Base technologies: Computation, storage, and communication components that continue to advance in raw capacity at exponential rates.
[Paraphrased from NSF Blue Ribbon Panel report, 2003]
Challenge: Creating and operating advanced cyberinfrastructure andintegrating it in science and engineering applications.
PASI: Mendoza, Argentina (May 17, 2005)
Paul Avery 4
Cyberinfrastructure and Grids Grid: Geographically distributed computing
resources configured for coordinated use
Fabric: Physical resources & networks provide raw capability
Ownership: Resources controlled by owners and shared w/ others
Middleware: Software ties it all together: tools, services, etc.
Enhancing collaboration via transparent resource sharing
US-CMS“Virtual Organization”
PASI: Mendoza, Argentina (May 17, 2005)
Paul Avery 5
Data Grids & Collaborative Research
Team-based 21st century scientific discoveryStrongly dependent on advanced information technologyPeople and resources distributed internationally
Dominant factor: data growth (1 Petabyte = 1000 TB)2000 ~0.5 Petabyte2005 ~10 Petabytes2010 ~100 Petabytes2015-7 ~1000 Petabytes?
Drives need for powerful linked resources: “Data Grids”
Computation Massive, distributed CPUData storage and access Distributed hi-speed disk and
tapeData movement International optical networks
Collaborative research and Data GridsData discovery, resource sharing, distributed analysis, etc.
How to collect, manage,access and interpret thisquantity of data?
PASI: Mendoza, Argentina (May 17, 2005)
Paul Avery 6
Examples of Data Intensive Disciplines
High energy & nuclear physicsBelle, BaBar, Tevatron, RHIC, JLABLarge Hadron Collider (LHC)
AstronomyDigital sky surveys, “Virtual” ObservatoriesVLBI arrays: multiple- Gb/s data streams
Gravity wave searchesLIGO, GEO, VIRGO, TAMA, ACIGA, …
Earth and climate systemsEarth Observation, climate modeling, oceanography, …
Biology, medicine, imagingGenome databasesProteomics (protein structure & interactions, drug delivery,
…)High-resolution brain scans (1-10m, time dependent)
Primary driver
PASI: Mendoza, Argentina (May 17, 2005)
Paul Avery 7
Our Vision & Goals
Develop the technologies & tools needed to exploit a Grid-based cyberinfrastructure
Apply and evaluate those technologies & tools in challenging scientific problems
Develop the technologies & procedures to support a permanent Grid-based cyberinfrastructure
Create and operate a persistent Grid-based cyberinfrastructure in support of discipline-specific research goals
GriPhyN + iVDGL + DOE Particle Physics Data Grid (PPDG) = Trillium
End-to-end
PASI: Mendoza, Argentina (May 17, 2005)
Paul Avery 8
Our Science Drivers Experiments at Large Hadron Collider
New fundamental particles and forces100s of Petabytes 2007 - ?
High Energy & Nuclear Physics exptsTop quark, nuclear matter at extreme
density~1 Petabyte (1000 TB) 1997 – present
LIGO (gravity wave search)Search for gravitational waves100s of Terabytes 2002 – present
Sloan Digital Sky SurveySystematic survey of astronomical objects10s of Terabytes 2001 – present
Data
gro
wth
Com
mu
nit
y g
row
th
2007
2005
2003
2001
2009
PASI: Mendoza, Argentina (May 17, 2005)
Paul Avery 9
Grid Middleware: Virtual Data Toolkit
Sources(CVS)
Patching
GPT srcbundles
NMI
Build & TestCondor pool
22+ Op. Systems
Build
Test
Package
VDT
Build
Many Contributors
Build
Pacman cache
RPMs
Binaries
Binaries
Binaries Test
A unique laboratory for testing, supporting, deploying, packaging, upgrading, & troubleshooting complex sets of software!
PASI: Mendoza, Argentina (May 17, 2005)
Paul Avery 10
VDT Growth Over 3 Years
0
5
10
15
20
25
30
35Ja
n-02
Apr
-02
Jul-0
2
Oct
-02
Jan-
03
Apr
-03
Jul-0
3
Oct
-03
Jan-
04
Apr
-04
Jul-0
4
Oct
-04
Jan-
05
Apr
-05
VDT 1.1.x VDT 1.2.x VDT 1.3.x
# o
f co
mponen
ts
VDT 1.0Globus 2.0bCondor 6.3.1
VDT 1.1.7Switch to Globus 2.2
VDT 1.1.11Grid3
VDT 1.1.8First real use by LCG
www.griphyn.org/vdt/
PASI: Mendoza, Argentina (May 17, 2005)
Paul Avery 11
Components of VDT 1.3.5 Globus 3.2.1 Condor 6.7.6 RLS 3.0 ClassAds 0.9.7 Replica 2.2.4 DOE/EDG CA certs ftsh 2.0.5 EDG mkgridmap EDG CRL Update GLUE Schema 1.0 VDS 1.3.5b Java Netlogger 3.2.4 Gatekeeper-Authz MyProxy1.11 KX509
System Profiler GSI OpenSSH 3.4 Monalisa 1.2.32 PyGlobus 1.0.6 MySQL UberFTP 1.11 DRM 1.2.6a VOMS 1.4.0 VOMS Admin 0.7.5 Tomcat PRIMA 0.2 Certificate Scripts Apache jClarens 0.5.3 New GridFTP Server GUMS 1.0.1
PASI: Mendoza, Argentina (May 17, 2005)
Paul Avery 12
Collaborative Relationships:A CS + VDT Perspective
ComputerScience
Research
VirtualData
Toolkit
Partner science projectsPartner networking projectsPartner outreach projects
LargerScience
Community
Globus, Condor, NMI, iVDGL, PPDGEU DataGrid, LHC Experiments,QuarkNet, CHEPREO, Dig. Divide
ProductionDeployment
Tech
Transfer
Techniques
& software
RequirementsPrototyping
& experiments
Other linkages Work force CS researchers Industry
U.S.GridsInt’l
Outreach
PASI: Mendoza, Argentina (May 17, 2005)
Paul Avery 13
U.S. “Trillium” Grid Partnership Trillium = PPDG + GriPhyN + iVDGL
Particle Physics Data Grid: $12M (DOE) (1999 – 2006)GriPhyN: $12M (NSF) (2000 – 2005) iVDGL: $14M (NSF) (2001 – 2006)
Basic composition (~150 people)PPDG: 4 universities, 6 labsGriPhyN: 12 universities, SDSC, 3 labs iVDGL: 18 universities, SDSC, 4 labs, foreign partnersExpts: BaBar, D0, STAR, Jlab, CMS, ATLAS, LIGO,
SDSS/NVO
Coordinated internally to meet broad goalsGriPhyN: CS research, Virtual Data Toolkit (VDT)
development iVDGL: Grid laboratory deployment using VDT,
applicationsPPDG: “End to end” Grid services, monitoring, analysisCommon use of VDT for underlying Grid middlewareUnified entity when collaborating internationally
PASI: Mendoza, Argentina (May 17, 2005)
Paul Avery 14
Goal: Peta-scale Data Grids forGlobal Science
Virtual Data Tools
Request Planning &Scheduling Tools
Request Execution & Management Tools
Transforms
Distributed resources(code, storage, CPUs,networks)
ResourceManagement
Services
Security andPolicy
Services
Other GridServices
Interactive User Tools
Production TeamSingle Researcher Workgroups
Raw datasource
PetaOps Petabytes Performance
PASI: Mendoza, Argentina (May 17, 2005)
Paul Avery 15
Sloan Digital Sky Survey (SDSS)Using Virtual Data in GriPhyN
1
10
100
1000
10000
100000
1 10 100
Num
ber
of C
lust
ers
Number of Galaxies
Galaxy clustersize distribution
Sloan Data
PASI: Mendoza, Argentina (May 17, 2005)
Paul Avery 16
The LIGO Scientific Collaboration (LSC)and the LIGO Grid
LIGO Grid: 6 US sites
* LHO, LLO: observatory sites* LSC - LIGO Scientific Collaboration - iVDGL supported
iVDGL has enabled LSC to establish a persistent production grid
Cardiff
AEI/Golm •
+ 3 EU sites (Cardiff/UK, AEI/Germany)
Birmingham•
PASI: Mendoza, Argentina (May 17, 2005)
Paul Avery 17
Large Hadron Collider &its Frontier Computing
Challenges
PASI: Mendoza, Argentina (May 17, 2005)
Paul Avery 18
Search for Origin of Mass New fundamental forces Supersymmetry Other new particles 2007 – ?
TOTEM
LHCb
ALICE
27 km Tunnel in Switzerland & France
CMS
ATLAS
Large Hadron Collider (LHC)@ CERN
PASI: Mendoza, Argentina (May 17, 2005)
Paul Avery 19
CMS: “Compact” Muon Solenoid
Inconsequential humans
PASI: Mendoza, Argentina (May 17, 2005)
Paul Avery 20
LHC Data Rates: Detector to Storage
Level 1 Trigger: Special Hardware
40 MHz
75 KHz 75 GB/sec
5 KHz 5 GB/sec
Level 2 Trigger: Commodity CPUs
100 Hz0.15 – 1.5 GB/sec
Level 3 Trigger: Commodity CPUs
Raw Data to storage(+ simulated data)
Physics filtering ~TBytes/sec
PASI: Mendoza, Argentina (May 17, 2005)
Paul Avery 21
All charged tracks with pt > 2 GeV
Reconstructed tracks with pt > 25 GeV
(+30 minimum bias events)
109 collisions/sec, selectivity: 1 in 1013
Complexity: Higgs Decay to 4 Muons
PASI: Mendoza, Argentina (May 17, 2005)
Paul Avery 22
LHC: Petascale Global Science Complexity: Millions of individual detector channels Scale: PetaOps (CPU), 100s of Petabytes (Data) Distribution: Global distribution of people & resources
CMS Example- 20075000+ Physicists 250+ Institutes 60+ Countries
BaBar/D0 Example - 2004700+ Physicists 100+ Institutes 35+ Countries
PASI: Mendoza, Argentina (May 17, 2005)
Paul Avery 23
CMS Experiment
LHC Global Data Grid (2007+)
Online System
CERN Computer Center
USAKorea RussiaUK
Maryland
150 - 1500 MB/s
>10 Gb/s
10-40 Gb/s
2.5-10 Gb/s
Tier 0
Tier 1
Tier 3
Tier 2
Physics caches
PCs
Iowa
UCSDCaltechU Florida
5000 physicists, 60 countries
10s of Petabytes/yr by 2008 1000 Petabytes in < 10 yrs?
FIU
Tier 4
PASI: Mendoza, Argentina (May 17, 2005)
Paul Avery 24
University Tier2 Centers Tier2 facility
Essential university role in extended computing infrastructure
20 – 25% of Tier1 national laboratory, supported by NSFValidated by 3 years of experience (CMS, ATLAS, LIGO)
FunctionsPerform physics analysis, simulationsSupport experiment softwareSupport smaller institutions
Official role in Grid hierarchy (U.S.)Sanctioned by MOU with parent organization (ATLAS, CMS,
LIGO)Selection by collaboration via careful processLocal P.I. with reporting responsibilities
PASI: Mendoza, Argentina (May 17, 2005)
Paul Avery 25
Grids and Globally Distributed Teams
Non-hierarchical: Chaotic analyses + productions Superimpose significant random data flows
PASI: Mendoza, Argentina (May 17, 2005)
Paul Avery 26
Grid3 andOpen Science Grid
PASI: Mendoza, Argentina (May 17, 2005)
Paul Avery 27
Grid3: A National Grid Infrastructure32 sites, 4000 CPUs: Universities + 4 national
labsPart of LHC Grid, Running since October 2003Sites in US, Korea, Brazil, TaiwanApplications in HEP, LIGO, SDSS, Genomics, fMRI,
CS
Brazil
http://www.ivdgl.org/grid3
PASI: Mendoza, Argentina (May 17, 2005)
Paul Avery 28
Grid3 World Map
PASI: Mendoza, Argentina (May 17, 2005)
Paul Avery 29
Grid3 Components Computers & storage at ~30 sites: 4000 CPUs Uniform service environment at each site
Globus Toolkit: Provides basic authentication, execution management, data movement
Pacman: Installs numerous other VDT and application services
Global & virtual organization servicesCertification & registration authorities, VO membership
services, monitoring services
Client-side tools for data access & analysis Virtual data, execution planning, DAG management,
execution management, monitoring
IGOC: iVDGL Grid Operations Center Grid testbed: Grid3dev
Middleware development and testing, new VDT versions, etc.
PASI: Mendoza, Argentina (May 17, 2005)
Paul Avery 30
Grid3 Applications
CMS experiment p-p collision simulations & analysis
ATLAS experiment p-p collision simulations & analysis
BTEV experiment p-p collision simulations & analysis
LIGO Search for gravitational wave sources
SDSS Galaxy cluster findingBio-molecular analysis
Shake n Bake (SnB) (Buffalo)
Genome analysis GADU/GnarefMRI Functional MRI (Dartmouth)CS Demonstrators Job Exerciser, GridFTP,
NetLoggerwww.ivdgl.org/grid3/applications
PASI: Mendoza, Argentina (May 17, 2005)
Paul Avery 31
Grid3 Shared Use Over 6 months
CMS DC04
ATLASDC2
Sep 10
Usa
ge:
CP
Us
PASI: Mendoza, Argentina (May 17, 2005)
Paul Avery 32
Grid3 Production Over 13 Months
PASI: Mendoza, Argentina (May 17, 2005)
Paul Avery 33
U.S. CMS 2003 Production10M p-p collisions; largest ever
2x simulation sample ½ manpower
Multi-VO sharing
PASI: Mendoza, Argentina (May 17, 2005)
Paul Avery 34
Grid3 as CS Research Lab:E.g., Adaptive Scheduling
Adaptive data placementin a realistic environment(K. Ranganathan)
Enables comparisonswith simulations
0
10
20
30
40
50
60
70
80
22-Jan 23-Jan 24-Jan 25-Jan 26-Jan 27-Jan 28-Jan
Aver
age
num
ber o
f Idl
e C
PUs
FNAL_CMS
IU_ATLAS_Tier2
0
500
1000
1500
2000
2500
BN
L_
AT
LA
S
Ca
lTe
ch
-Gri
d3
UFlo
rid
a-P
G
IU_
AT
LA
S_
Tie
r2
UB
uff
alo
-CC
R
UC
Sa
nD
ieg
oP
G
UFlo
rid
a-G
rid
3
UM
_A
TLA
S
Va
nd
erb
ilt
AN
L_
HE
P
Ca
lTe
ch
-PG
Tota
l I/O
tra
ffic
(M
Byte
s)
0
200
400
600
800
Least Loaded At Data Cost Function
Tot
al R
espo
nse
Tim
e (s
econ
ds)
PASI: Mendoza, Argentina (May 17, 2005)
Paul Avery 35
Grid3 Lessons Learned How to operate a Grid as a facility
Tools, services, error recovery, procedures, docs, organization
Delegation of responsibilities (Project, VO, service, site, …)Crucial role of Grid Operations Center (GOC)
How to support people people relationsFace-face meetings, phone cons, 1-1 interactions, mail lists,
etc.
How to test and validate Grid tools and applicationsVital role of testbeds
How to scale algorithms, software, processSome successes, but “interesting” failure modes still occur
How to apply distributed cyberinfrastructureSuccessful production runs for several applications
PASI: Mendoza, Argentina (May 17, 2005)
Paul Avery 36
Grid3 Open Science Grid Iteratively build & extend Grid3
Grid3 OSG-0 OSG-1 OSG-2 …Shared resources, benefiting broad set of disciplinesGrid middleware based on Virtual Data Toolkit (VDT)Emphasis on “end to end” services for applications
OSG collaborationComputer and application scientistsFacility, technology and resource providers (labs,
universities)
Further develop OSGPartnerships and contributions from other sciences,
universities Incorporation of advanced networkingFocus on general services, operations, end-to-end
performance
Aim for Summer 2005 deployment
PASI: Mendoza, Argentina (May 17, 2005)
Paul Avery 37
http://www.opensciencegrid.org
PASI: Mendoza, Argentina (May 17, 2005)
Paul Avery 38
Enterprise
Technical Groups
ResearchGrid Projects
VOs
Researchers
Sites
Service Providers
Universities,Labs
activity1activity
1activity1Activities
Advisory Committee
Core OSG Staff(few FTEs, manager)
OSG Council(all members abovea certain threshold,
Chair, officers)
Executive Board(8-15 representatives
Chair, Officers)
OSG Organization
PASI: Mendoza, Argentina (May 17, 2005)
Paul Avery 39
OSG Technical Groups & Activities
Technical Groups address and coordinate technical areas
Propose and carry out activities related to their given areasLiaise & collaborate with other peer projects (U.S. &
international)Participate in relevant standards organizations.Chairs participate in Blueprint, Integration and Deployment
activities
Activities are well-defined, scoped tasks contributing to OSG
Each Activity has deliverables and a plan… is self-organized and operated… is overseen & sponsored by one or more Technical
GroupsTGs and Activities are where the real work gets done
PASI: Mendoza, Argentina (May 17, 2005)
Paul Avery 40
OSG Technical GroupsGovernance Charter, organization, by-laws,
agreements, formal processes
Policy VO & site policy, authorization, priorities, privilege & access rights
Security Common security principles, security infrastructure
Monitoring and Information Services
Resource monitoring, information services, auditing, troubleshooting
Storage Storage services at remote sites, interfaces, interoperability
Support Centers Infrastructure and services for user support, helpdesk, trouble ticket
Education / Outreach
Training, interface with various E/O projects
Networks (new) Including interfacing with various networking projects
PASI: Mendoza, Argentina (May 17, 2005)
Paul Avery 41
OSG Activities
Blueprint Defining principles and best practices for OSG
Deployment Deployment of resources & servicesProvisioning Connected to deploymentIncidence response
Plans and procedures for responding to security incidents
Integration Testing & validating & integrating new services and technologies
Data Resource Management (DRM)
Deployment of specific Storage Resource Management technology
Documentation Organizing the documentation infrastructure
Accounting Accounting and auditing use of OSG resources
Interoperability Primarily interoperability between Operations Operating Grid-wide services
PASI: Mendoza, Argentina (May 17, 2005)
Paul Avery 42
Connections to European Projects:
LCG and EGEE
PASI: Mendoza, Argentina (May 17, 2005)
Paul Avery 43
The Path to the OSG Operating Grid
OS
G D
eplo
ymen
t A
ctiv
ity
Metrics &Certification
Applicationvalidation
VO Application
SoftwareInstallation
OSG Integration Activity
Release Description
MiddlewareInteroperability
Software &packaging
Functionality & Scalability
Tests
Readiness plan adopted
Service deployment
OS
G O
per
atio
ns-
Pro
visi
on
ing
Act
ivit
y
ReleaseCandidate
Readinessplan
Effort
Resources
feedback
PASI: Mendoza, Argentina (May 17, 2005)
Paul Avery 44
OSG Integration Testbed
Brazil
PASI: Mendoza, Argentina (May 17, 2005)
Paul Avery 45
Status of OSG Deployment OSG infrastructure release accepted for
deployment. US CMS MOP “flood testing” successful D0 simulation & reprocessing jobs running on selected
OSG sites Others in various stages of readying applications &
infrastructure(ATLAS, CMS, STAR, CDF, BaBar, fMRI)
Deployment process underway: End of July? Open OSG and transition resources from Grid3 Applications will use growing ITB & OSG resources
during transition
http://osg.ivdgl.org/twiki/bin/view/Integration/WebHome
PASI: Mendoza, Argentina (May 17, 2005)
Paul Avery 46
Interoperability & Federation Transparent use of Federated Grid infrastructures a
goal There are sites that appear as part of “LCG” as well
as part of OSG/Grid3D0 bringing reprocessing to LCG sites through adaptor nodeCMS and ATLAS can run their jobs on both LCG and OSG
Increasing interaction with TeraGridCMS and ATLAS sample simulation jobs are running on
TeraGridPlans for TeraGrid allocation for jobs running in Grid3
model: with group accounts, binary distributions, external data management, etc
PASI: Mendoza, Argentina (May 17, 2005)
Paul Avery 47
Networks
PASI: Mendoza, Argentina (May 17, 2005)
Paul Avery 48
Evolving Science Requirements for Networks (DOE High Perf. Network
Workshop)
Science Areas
Today End2End
Throughput
5 years End2End
Throughput
5-10 Years End2End
Throughput
Remarks
High Energy Physics
0.5 Gb/s 100 Gb/s 1000 Gb/s High bulk throughput
Climate (Data &
Computation)
0.5 Gb/s 160-200 Gb/s
N x 1000 Gb/s
High bulk throughput
SNS NanoScience
Not yet started
1 Gb/s 1000 Gb/s + QoS for Control Channel
Remote control and time critical throughput
Fusion Energy
0.066 Gb/s(500 MB/s
burst)
0.2 Gb/s(500MB/20 sec. burst)
N x 1000 Gb/s
Time critical throughput
Astrophysics 0.013 Gb/s(1 TB/week)
N*N multicast
1000 Gb/s Computational steering and
collaborations
Genomics Data &
Computation
0.091 Gb/s(1 TB/day)
100s of users
1000 Gb/s + QoS for Control Channel
High throughput
and steering
See http://www.doecollaboratory.org/meetings/hpnpw/
PASI: Mendoza, Argentina (May 17, 2005)
Paul Avery 49
UltraLight: Advanced Networking
in Applications
10 Gb/s+ network• Caltech, UF, FIU, UM, MIT• SLAC, FNAL• Int’l partners• Level(3), Cisco, NLR
Funded by ITR2004
PASI: Mendoza, Argentina (May 17, 2005)
Paul Avery 50
UltraLight: New Information System
A new class of integrated information systems Includes networking as a managed resource for the first
timeUses “Hybrid” packet-switched and circuit-switched
optical network infrastructureMonitor, manage & optimize network and Grid Systems in
realtime
Flagship applications: HEP, eVLBI, “burst” imaging“Terabyte-scale” data transactions in minutesExtend Real-Time eVLBI to the 10 – 100 Gb/s Range
Powerful testbedSignificant storage, optical networks for testing new Grid
services
Strong vendor partnershipsCisco, Calient, NLR, CENIC, Internet2/Abilene
PASI: Mendoza, Argentina (May 17, 2005)
Paul Avery 51
Education and Outreach
PASI: Mendoza, Argentina (May 17, 2005)
Paul Avery 52
iVDGL, GriPhyN Education/Outreach
Basics $200K/yr Led by UT Brownsville Workshops, portals, tutorials New partnerships with
QuarkNet, CHEPREO, LIGO E/O, …
PASI: Mendoza, Argentina (May 17, 2005)
Paul Avery 53
June 2004 Grid Summer School First of its kind in the U.S. (South Padre Island,
Texas)36 students, diverse origins and types (M, F, MSIs, etc)
Marks new direction for U.S. Grid effortsFirst attempt to systematically train people in Grid
technologiesFirst attempt to gather relevant materials in one placeToday: Students in CS and PhysicsNext: Students, postdocs, junior & senior scientists
Reaching a wider audiencePut lectures, exercises, video, on the webMore tutorials, perhaps 2-3/yearDedicated resources for remote tutorialsCreate “Grid Cookbook”, e.g. Georgia Tech
Second workshop: July 11–15, 2005South Padre Island again
PASI: Mendoza, Argentina (May 17, 2005)
Paul Avery 54
QuarkNet/GriPhyN e-Lab Project
http://quarknet.uchicago.edu/elab/cosmic/home.jsp
PASI: Mendoza, Argentina (May 17, 2005)
Paul Avery 55
Student Muon Lifetime Analysis in GriPhyN/QuarkNet
CHEPREO: Center for High Energy Physics Research and Educational OutreachFlorida International University
Physics Learning Center CMS Research iVDGL Grid Activities AMPATH network (S.
America)
Funded September 2003
$4M initially (3 years) MPS, CISE, EHR, INT
PASI: Mendoza, Argentina (May 17, 2005)
Paul Avery 57
NEWS: Bulletin: ONE TWOWELCOME BULLETIN General InformationRegistrationTravel Information Hotel RegistrationParticipant List How to Get UERJ/Hotel Computer AccountsUseful Phone Numbers ProgramContact us: Secretariat Chairmen
Grids and the Digital DivideRio de Janeiro, Feb. 16-20, 2004
Background World Summit on Information Society HEP Standing Committee on Inter-
regional Connectivity (SCIC)
Themes Global collaborations, Grids and
addressing the Digital Divide Focus on poorly connected regionsNext meeting: Daegu, Korea May 23-27, 2005
PASI: Mendoza, Argentina (May 17, 2005)
Paul Avery 58
Partnerships Drive Success Integrating Grids in scientific research
“Lab-centric”: Activities center around large facility“Team-centric”: Resources shared by distributed teams“Knowledge-centric”: Knowledge generated/used by a community
Strengthening the role of universities in frontier researchCouples universities to frontier data intensive researchBrings front-line research and resources to studentsExploits intellectual resources at minority or remote institutions
Driving advances in IT/science/engineeringDomain sciences Computer ScienceUniversities LaboratoriesScientists StudentsNSF projects NSF projectsNSF DOEResearch communities IT industry
PASI: Mendoza, Argentina (May 17, 2005)
Paul Avery 59
Fulfilling the Promise ofNext Generation Science
Supporting permanent, national-scale Grid infrastructure
Large CPU, storage and network capability crucial for scienceSupport personnel, equipment maintenance, replacement,
upgradeTier1 and Tier2 resources a vital part of infrastructureOpen Science Grid a unique national infrastructure for science
Supporting the maintenance, testing and dissemination of advanced middleware
Long-term support of the Virtual Data ToolkitVital for reaching new disciplines & for supporting large
international collaborations
Continuing support for HEP as a frontier challenge driver
Huge challenges posed by LHC global interactive analysisNew challenges posed by remote operation of Global
Accelerator Network
PASI: Mendoza, Argentina (May 17, 2005)
Paul Avery 60
Fulfilling the Promise (2) Creating even more advanced cyberinfrastructure
Integrating databases in large-scale Grid environments Interactive analysis with distributed teamsPartnerships involving CS research with application drivers
Supporting the emerging role of advanced networksReliable, high performance LANs and WANs necessary for
advanced Grid applications
Partnering to enable stronger, more diverse programs
Programs supported by multiple Directorates, a la CHEPREONSF-DOE joint initiativesStrengthen ability of universities and labs to work together
Providing opportunities for cyberinfrastructure training, education & outreach
Grid tutorials, Grid CookbookCollaborative tools for student-led projects & research
PASI: Mendoza, Argentina (May 17, 2005)
Paul Avery 61
Summary Grids enable 21st century collaborative science
Linking research communities and resources for scientific discovery
Needed by global collaborations pursuing “petascale” science
Grid3 was an important first step in developing US Grids
Value of planning, coordination, testbeds, rapid feedbackValue of learning how to operate a Grid as a facilityValue of building & sustaining community relationships
Grids drive need for advanced optical networks Grids impact education and outreach
Providing technologies & resources for training, education, outreach
Addressing the Digital Divide
OSG: a scalable computing infrastructure for science?Strategies needed to cope with increasingly large scale
PASI: Mendoza, Argentina (May 17, 2005)
Paul Avery 62
Grid Project ReferencesOpen Science Grid
www.opensciencegrid.org
Grid3 www.ivdgl.org/grid3
Virtual Data Toolkit www.griphyn.org/vdt
GriPhyN www.griphyn.org
iVDGL www.ivdgl.org
PPDG www.ppdg.net
CHEPREO www.chepreo.org
UltraLight ultralight.cacr.caltech.edu
Globus www.globus.org
Condor www.cs.wisc.edu/condor
LCG www.cern.ch/lcg
EU DataGrid www.eu-datagrid.org
EGEE www.eu-egee.org
PASI: Mendoza, Argentina (May 17, 2005)
Paul Avery 63
Extra Slides
PASI: Mendoza, Argentina (May 17, 2005)
Paul Avery 64
GriPhyN Goals Conduct CS research to achieve vision
Virtual Data as unifying principlePlanning, execution, performance monitoring
Disseminate through Virtual Data ToolkitA “concrete” deliverable
Integrate into GriPhyN science experimentsCommon Grid tools, services
Educate, involve, train students in IT researchUndergrads, grads, postdocs, Underrepresented groups
PASI: Mendoza, Argentina (May 17, 2005)
Paul Avery 65
iVDGL Goals Deploy a Grid laboratory
Support research mission of data intensive experimentsProvide computing and personnel resources at university sitesProvide platform for computer science technology
developmentPrototype and deploy a Grid Operations Center (iGOC)
Integrate Grid software tools Into computing infrastructures of the experiments
Support delivery of Grid technologiesHardening of the Virtual Data Toolkit (VDT) and other
middleware technologies developed by GriPhyN and other Grid projects
Education and OutreachLead and collaborate with Education and Outreach effortsProvide tools and mechanisms for underrepresented groups
and remote regions to participate in international science projects
PASI: Mendoza, Argentina (May 17, 2005)
Paul Avery 66
CMS: Grid Enabled Analysis Architecture
SchedulerCatalogs
Grid ServicesWeb Server
ExecutionPriority
Manager
Grid WideExecutionService
DataManage-
mentFully-ConcretePlanner
Fully-AbstractPlanner
Analysis Client
Virtual Data
Replica
ApplicationsMonitoring
Partially-AbstractPlanner
Metadata
HTTP, SOAP, XML-RPC
Chimera
Sphinx
MonALISA
•ROOT (analysis tool)•Python•Cojac (detector viz)/IGUANA (cms viz)
Clarens
MCRunjob
BOSS
RefDB
POOL
ORCA
ROOT FAMOS
VDT-Server
MOPDB
•Discovery•ACL management•Cert. based access
Clients talk standard protocols to “Grid Services Web Server”
Simple Web service API allows simple or complex analysis clients
Typical clients: ROOT, Web Browser, ….
Clarens portal hides complexity
Key features: Global Scheduler, Catalogs, Monitoring, Grid-wide Execution service
AnalysisClient
PASI: Mendoza, Argentina (May 17, 2005)
Paul Avery 67
“Virtual Data”: Derivation & Provenance
Most scientific data are not simple “measurements”They are computationally corrected/reconstructedThey can be produced by numerical simulation
Science & eng. projects are more CPU and data intensive
Programs are significant community resources (transformations)
So are the executions of those programs (derivations)
Management of dataset dependencies critical!Derivation: Instantiation of a potential data productProvenance: Complete history of any existing data
productPreviously: Manual methodsGriPhyN: Automated, robust
tools
PASI: Mendoza, Argentina (May 17, 2005)
Paul Avery 68
decay = WWWW e Pt > 20
decay = WWWW e
decay = WWWW leptons
mass = 160
decay = WW
decay = ZZ
decay = bb
Other cuts
Scientist adds a new derived data branch & continues analysis
Virtual Data Example: HEP Analysis
Other cuts Other cuts Other cuts
PASI: Mendoza, Argentina (May 17, 2005)
Paul Avery 69
Language: define software environments Interpreter: create, install, configure, update, verify
environmentsVersion 3.0.2 released Jan. 2005 LCG/Scram ATLAS/CMT CMS DPE/tar/make LIGO/tar/make OpenSource/tar/
make
Globus/GPT NPACI/TeraGrid/tar/
make D0/UPS-UPD Commercial/tar/make
Combine and manage software from arbitrary sources.
% pacman –get iVDGL:Grid3
“1 button install”: Reduce burden on administrators
Remote experts define installation/ config/updating for everyone at once
VDT
ATLAS
ATLAS
NPACI
NPACI
D-Zero
D-Zero
iVDGL
iVDGL
UCHEP
UCHEP
% pacman
VDT
CMS/DPE
LIGO
Packaging of Grid Software: Pacman
PASI: Mendoza, Argentina (May 17, 2005)
Paul Avery 70
“I’ve detected a muon calibration error and want to know which derived data products need to be recomputed.”
“I’ve found some interesting data, but I need to know exactly what corrections were applied before I can trust it.”
“I want to search a database for 3 muon events. If a program that does this analysis exists, I won’t have to write one from scratch.”
“I want to apply a forward jet analysis to 100M events. If the results already exist, I’ll save weeks of computation.”
Virtual Data Motivations
VDC
Describe Discover
Reuse Validate
PASI: Mendoza, Argentina (May 17, 2005)
Paul Avery 71
U.S. Funded ProjectsGriPhyN (NSF) iVDGL (NSF)Particle Physics Data Grid (DOE)UltraLightTeraGrid (NSF)DOE Science Grid (DOE)NEESgrid (NSF)NSF Middleware Initiative (NSF)
Background: Data Grid Projects
EU, Asia projectsEGEE (EU)LCG (CERN)DataGridEU national ProjectsDataTAG (EU)CrossGrid (EU)GridLab (EU) Japanese, Korea Projects
Many projects driven/led by HEP + CS Many 10s x $M brought into the field Large impact on other sciences, education
Driven primarily by HEP applications
PASI: Mendoza, Argentina (May 17, 2005)
Paul Avery 72
“Virtual Data”: Derivation & Provenance
Most scientific data are not simple “measurements”They are computationally corrected/reconstructedThey can be produced by numerical simulation
Science & eng. projects are more CPU and data intensive
Programs are significant community resources (transformations)
So are the executions of those programs (derivations)
Management of dataset dependencies critical!Derivation: Instantiation of a potential data productProvenance: Complete history of any existing data
productPreviously: Manual methodsGriPhyN: Automated, robust
tools
PASI: Mendoza, Argentina (May 17, 2005)
Paul Avery 73
Muon Lifetime Analysis Workflow
pythia_input
pythia.exe
cmsim_input
cmsim.exe
writeHits
writeDigis
begin v /usr/local/demo/scripts/cmkin_input.csh file i ntpl_file_path file i template_file file i num_events stdout cmkin_param_fileend
begin v /usr/local/demo/binaries/kine_make_ntpl_pyt_cms121.exe pre cms_env_var stdin cmkin_param_file stdout cmkin_log file o ntpl_fileend
begin v /usr/local/demo/scripts/cmsim_input.csh file i ntpl_file file i fz_file_path file i hbook_file_path file i num_trigs stdout cmsim_param_fileend
begin v /usr/local/demo/binaries/cms121.exe condor copy_to_spool=false condor getenv=true stdin cmsim_param_file stdout cmsim_log file o fz_file file o hbook_fileend
begin v /usr/local/demo/binaries/writeHits.sh condor getenv=true pre orca_hits file i fz_file file i detinput file i condor_writeHits_log file i oo_fd_boot file i datasetname stdout writeHits_log file o hits_dbend
begin v /usr/local/demo/binaries/writeDigis.sh pre orca_digis file i hits_db file i oo_fd_boot file i carf_input_dataset_name file i carf_output_dataset_name file i carf_input_owner file i carf_output_owner file i condor_writeDigis_log stdout writeDigis_log file o digis_dbend
(Early) Virtual Data Language
CMS “Pipeline”
PASI: Mendoza, Argentina (May 17, 2005)
Paul Avery 75
QuarkNet Portal Architecture
Simpler interface for non-experts Builds on Chiron portal
PASI: Mendoza, Argentina (May 17, 2005)
Paul Avery 76
Integration of GriPhyN and IVDGL
Both funded by NSF large ITRs, overlapping periodsGriPhyN: CS Research, Virtual Data Toolkit (9/2000–
9/2005) iVDGL: Grid Laboratory, applications (9/2001–
9/2006) Basic composition
GriPhyN: 12 universities, SDSC, 4 labs (~80 people) iVDGL: 18 institutions, SDSC, 4 labs (~100 people)Expts: CMS, ATLAS, LIGO, SDSS/NVO
GriPhyN (Grid research) vs iVDGL (Grid deployment)GriPhyN: 2/3 “CS” + 1/3 “physics” ( 0% H/W) iVDGL: 1/3 “CS” + 2/3 “physics” (20% H/W)
Many common elementsCommon Directors, Advisory Committee, linked
managementCommon Virtual Data Toolkit (VDT)Common Grid testbedsCommon Outreach effort
ScienceReview
ProductionManager
Researcher
instrument
Applications
storageelement
Grid
Grid Fabric
storageelement
storageelement
data
ServicesServices
discovery
discovery
sharing
VirtualData
Production Analysisparams
exec.
data
composition
VirtualData
planning
Planning
Production Analysisparams
exec.
data
Planning
Execution
planning
Virtual Data
Toolkit
Chimera virtual data
system
Pegasus planner
DAGman
Globus ToolkitCondor
Ganglia, etc.Gri
Ph
yN
O
verv
iew
Execution
PASI: Mendoza, Argentina (May 17, 2005)
Paul Avery 78
Chiron/QuarkNet Architecture
PASI: Mendoza, Argentina (May 17, 2005)
Paul Avery 79
Cyberinfrastructure
“A new age has dawned in scientific & engineering research, pushed by continuing progress in computing, information, and communication technology, & pulled by the expanding complexity, scope, and scale of today’s challenges. The capacity of this technology has crossed thresholds that now make possible a comprehensive “cyberinfrastructure” on which to build new types of scientific & engineering knowledge environments & organizations and to pursue research in new ways & with increased efficacy.”
[NSF Blue Ribbon Panel report, 2003]
PASI: Mendoza, Argentina (May 17, 2005)
Paul Avery 80
Fulfilling the Promise ofNext Generation Science
Our multidisciplinary partnership of physicists, computer scientists, engineers, networking specialists and education experts, from universities and laboratories, has achieved tremendous success in creating and maintaining general purpose cyberinfrastructure supporting leading-edge science.
But these achievements have occurred in the context of overlapping short-term projects. How can we ensure the survival of valuable existing cyber-infrastructure while continuing to address new challenges posed by frontier scientific and engineering endeavors?
PASI: Mendoza, Argentina (May 17, 2005)
Paul Avery 81
Production Simulations on Grid3
Used = 1.5 US-CMS resources
US-CMS Monte Carlo Simulation
USCMS
Non-USCMS