high energy physics & computing grid

HEP and Grid ComputingDr. Jaehoon Yu

High Energy Physics & Computing Grid

Univ. of Texas @ Arlington

Dr. Yu

Feb. 3, 2005 2HEP and Grid ComputingDr. Jaehoon Yu

Outline• High Energy Physics• The problem• A solution• An example of implemented solution• Accomplishments• Future plans• Summary


High Energy Physics

Structure of Matter

10-10m 10-14m 10-15m

<10-18m

10-9m

Matter Molecule Atom Nucleus

u

Quark

<10-19mprotons, neutrons,

mesons, etc.

top, bottom,charm, strange,

up, down

Condensed matter/Nano-Science/ChemistryAtomic Physics

NuclearPhysics

Baryon(Hadron)

Electron(Lepton)

10-2m


Mysteries in High Energy Physics? The “STANDARD MODELSTANDARD MODEL” has been extremely successful (Precision 10-6)

BUT… many mysteries

Why so many quarks/leptons??

Why four forces?? Unification?

Why is there large particle- antipaticle asymmetry?

Does Higgs particle exist?

Where does mass come from??

Are there other theories??


High Energy Physics• Definition: A field of Physics pursues for fundamental

constituents of matter and basic principles of interactions between them How is universe created, and how does it work?

• Use large particle accelerators • Use large particle detectors


The Standard Model• Assumes the following fundamental structure:

Discovered in 2000

Discovered in 1995


Fermilab Tevatron and LHC at CERN• Present world’s Highest Energy

proton-anti-proton collider – Ecm=1.96 TeV (=6.3x10-7J/p

13M Joules on 10-4m2) Equivalent to the kinetic energy of

a 20t truck at a speed 80 mi/hr

Chicago

Tevatron p

p CDF

DØ

• World’s Highest Energy proton-proton collider in 2 years – Ecm=14 TeV (=44x10-7J/p

1000M Joules on 10-4m2) Equivalent to the kinetic energy of

a 20t truck at a speed 6150 mi/hr


High Energy Physics• Definition: A field of Physics pursues for fundamental

constituents of matter and basic principles of interactions between them How is universe created, and how does it work?

• Use large particle accelerators • Use large particle detectors• Large, distributed collaborations

– ~600/experiment for currently operating experiments– ~2000/experiment for future experiments – WWW grew out of HEP to expedite communication

between collaborators


Particle Detection

InteractionPoint

electron

photon

jet

muonneutrino -- or any non-interacting particle missing transverse momentum

B

Scintillating FiberSilicon Tracking

Charged Particle Tracks

Calorimeter (dense)

EM hadronic

Energy

Wire Chambers

Mag

net

Muon Tracks

We know x,y starting momenta is zero, butalong the z axis it is not, so many of our measurements are in the xy plane, or transverse


DØ Detector: Run II

30’

30’

50’

• Weighs 5000 tons• Can inspect

3,000,000 collisions/second

• Will record 50 collisions/second

• Records ~12.5M Bytes/second

• Will record 2 Peta bytes in the current run.


DØ Central Calorimeter 1990


How are computers used in HEP?

Digital Data

Data Reconstruction

pp


qT

ime

p p

q g

K

“par

ton

jet”

“par

ticle

jet”

“cal

orim

eter

jet”

hadrons

CH

FH

EM

Highest ET dijet event at DØHighest ET dijet event at DØ

0.69 GeV, 472E

0.69 GeV, 475E21

T

11T

How does an Event Look in the DØ Detector?


• Current Experiments at Tevatron – Has been taking data for the past 3 years and will continue

throughout much of the decade The immediacy!!!– Current data size close to 1PB and will be over 4 PB by

the end (~100km stack of 100GB disk drives)• 10 – 20 times (~100PB) increase at the future experiments

The Problem


~50M events/mo


• Current Experiments at Tevatron – Has been taking data for the past 3 years and will continue

throughout much of the decade The immediacy!!!– Current data size close to 1PB and will be over 4 PB by

the end (~100km stack of 100GB disk drives)• 10 – 20 times (~100PB) increase at the future experiments

– Detectors are complicated Need many people to construct and make them work

– Collaboration is large and scattered all over the world

The Problem


~700 Collaborators~80 Institutions18 Countries

Typical HEP Collaboration at Present


First Beams: Summer 2007Physics Runs: from Fall 2007

TOTEM

LHCb: B-physics

ALICE : HI

pp s =14 TeV L=1034 cm-2 s-1

27 km Tunnel in Switzerland & France

Large Hadron Collider (LHC) CERN, Geneva: 2007 Start

Large Hadron Collider (LHC) CERN, Geneva: 2007 Start

CMS

Atlas

5000+ Physicists 250+ Institutes 60+ Countries

H. Newman


• Current Experiments at Tevatron – Has been taking data for the past 3 years and will continue throughout much of the

decade The immediacy!!!– Current data size close to 1PB and will be over 4 PB by the end (~100km stack of

100GB disk drives)• 10 – 20 times (~100PB) increase at the future experiments

– Detectors are complicated Need many people to construct and make them work

– Collaboration is large and scattered all over the world– Development and improvements at remote institutions– Optimized resource management, job scheduling, and monitoring tools– Efficient and transparent data delivery and sharing

• Use the opportunity of having large data set in furthering grid computing technology– Improve computational capability for education – Improve quality of life

The Problem


What is a Computing Grid?• Grid: Geographically distributed computing resources

configured for coordinated use• Physical resources & networks provide raw capability• “Middleware” software ties it together


How do HEP physicists communicate?


Old Deployment ModelsStarted with Fermilab-centric SAM infrastructure in place, …

…transition to hierarchically distributed Model


Desktop Analysis Stations

Institutional Analysis Centers

Regional Analysis Centers

Normal InteractionCommunication PathOccasional Interaction Communication Path

Central Analysis Center (CAC)

DAS DAS…. DAS DAS….

IAC ... IAC IAC…IAC

RAC …. RAC

DØ Remote Analysis Model (DØRAM)Fermilab


UTA has the first and the only US RAC

Mexico/Brazil

OU/LU

UAZ

RiceLTU

UTA

KUKSU

Ole Miss

DØRAM Implementation

MainzWuppertal

Munich

AachenBonn

GridKa

(Karlsruhe)DØ Southern Analysis Region (DØSAR) formed around UTA


What can accomplished in an analysis region?• Construct end-to-end service environment in a smaller, manageable

scale• Train and accumulate local expertise and share them• Form a smaller group to work coherently and closely• Draw additional resources from variety of funding sources

– Promote interdisciplinary collaboration• Increase intellectual resources for the experiment

– Enable remote participants to be more actively contribute to the collaboration• Form a grid and use it for DØ

– Simulated data (Monte Carlo) production– Actual data reconstruction– Actual and simulated data analyses

• Promote and improve IAC’s group stature


DØSAR Consortium First Generation IAC’s

University of Texas at Arlington Louisiana Tech University Langston University University of Oklahoma Tata Institute (India)

• Second Generation IAC’s– Cinvestav, Mexico– Universidade Estadual Paulista, Brazil – University of Kansas– Kansas State University

• Third Generation IAC’s– Ole Miss, MS– Rice University, TX– University of Arizona, Tucson, AZ– USTC China– Korea University, Korea

Each 1st generation institution is paired with a 2nd generation institution to help expedite implementation of D0SAR capabilities

Both 1st and 2nd generation institutions can then help the 3rd generation institutions implement D0SAR capabilities


DØSAR Accomplishments• The only established US analysis region within DØ• Constructed and activated a Regional Analysis Center• Formed and activated five new MC production farms• Data access capability implemented in 70% of the sites

• Employed and developed and implemented many useful monitoring tools– Ganglia and MonaLISA– McFarmGraph, McPerM, McQue, and McFarmDB


UTA, The New Way•100 P4 Xeon 2.6GHz CPU = 260 GHz•64TB of Disk space

•84 P4 Xeon 2.4GHz CPU = 202 GHz•7.5TB of Disk space

•Total CPU: 462 GHz•Total disk: 73TB•Total Memory: 168Gbyte•Network bandwidth: 68Gb/sec


Various Monitoring ApplicationsGanglia: Operating since Apr. 2003 McGraph: Operating since Sept. 2003

McPerM: Operating since Sept. 2003McQue: Operating since June 2004

HEP and Grid ComputingDr. Jaehoon Yu

ot

MonaLISA Grid Resource Monitoring


DØSAR Accomplishments• The only established US analysis region within DØ• Formed and activated five new MC production farms• SAM stations are installed in eight sites (three more to go..)• Constructed and activated a Regional Analysis Center• Employed and developed and implemented many useful monitoring tools• Started contributing beyond MC production

• Accumulated large expertise in many areas• Successfully brought in additional computing and

human resources


DØSAR Computing & Human ResourcesInstitutions CPU(GHz) [future] Storage (TB) People

Cinvestav 13 1.1 1F + …

Langston 22 1.3 1F+1GA

LTU 25+[12] 3.0 1F+1PD+2GA

KU 12 2.5 1F+1PD

KSU 40 3.5 1F+2GA

OU 19+[270] 1.8 + 120(tape) 4F+3PD+2GA

Sao Paulo 115+[300] 4.5 2F + Many

Tata Institute 78 1.6 1F + 1Sys

Ole Miss 300 1.5 1F + 1Sys

UTA 520 74 2.5F+ 1sys + 1.1PD + 3GA

Total 1144 + [582] 95 + 120 (tape) 15.5F+3sys+6.1PD+10GA


DØSAR Accomplishments• The only established US analysis region within DØ• Formed and activated five new MC production farms• SAM stations are installed in eight sites (three more to go..)• Constructed and activated a Regional Analysis Center• Employed and developed and implemented many useful monitoring tools• Started contributing beyond MC production• Accumulated large expertise• Successfully brought in additional computing and human resources

• Formed the DØSARGrid and producing simulated events on it


GUI to Access Grid


How does the DØSARGrid work?

Client Site

DØ Grid

Sub. Sites

Reg. Grids

Exe. Sites

Desktop. Clst.

Desktop. Clst.

Ded. Clst.

Ded. Clst.

SAM

JDL


DØSAR Accomplishments• The only established US analysis region within DØ• Formed and activated five new MC production farms• SAM stations are installed in eight sites (three more to go..)• Constructed and activated a Regional Analysis Center• Employed and developed and implemented many useful monitoring

tools• Started contributing beyond MC production• Accumulated large expertise• Successfully brought in additional computing and human resources• Developed McFarm interface to SAMGrid• Formed the DØSARGrid and start producing MC events on the grid

• Promote inter-disciplinary collaborations


What next?• We will participate in large scale DØ data processing in a few

months– 5 TB of data has been pre-staged in preparation

• Must perform data analysis in the region using the regional resources– Large number of data sets has been transferred to UTA– HEP Students are working on their theses analyses using these data

sets– Physics Undergraduate students are working on their class projects

using this data• Preparing the transition into future experiment and exploit it in

DØ• Improve local infrastructure, such as network bandwidths


Network Bandwidth Usage at UTA

DPCC online

DØ and ATLAS Production


What next, cnt’d?• Transform into a legitimate, active Virtual Organization within

the global grid picture Should be gradually done within the next year– Participate in existing US (Open Science Grid) and European

(Enabling Grid for E-science in Europe) – Fully utilize the involvement with future experiments– Turn DØSAR into DOSAR (Data Oriented Super Analysis Region)

• Continue promoting interdisciplinary collaboration• Actively participate and lead grid computing efforts in the

respective states• Employ the grid computing technology not just for research but

also for education


NLR – National LambdaRail

Denver

Seattle

Sunnyvale

LA

San Diego

Chicago Pitts

Wash DC

Raleigh

Jacksonville

Atlanta

KC

Baton Rouge

El Paso - Las Cruces

Phoenix

Pensacola

Dallas

San Ant.Houston

Albuq. Tulsa

New YorkClev

10GB/sec connections


Grid in other disciplines?• Nuclear physics• Bioinformatics • Genetics• Meteorology• Medical science and medicine• Homeland security


Conclusions• To understand the fundamentals of nature, High Energy Physics

– Uses accelerators to look into extremely small distances– Uses large detectors to explore nature– Uses large number of computers to process data– Large amount of data gets accumulated need computing grid to

perform expeditious data analysis• Computing grid needed for other disciplines with large data sets• HEP is an exciting endeavor in understanding nature• Physics analyses at one’s own desktop using computing grid is

close to be a reality• UTA plays a leading role in HEP research and shaping the

future of computing Grid• Computing grid will soon revolutionize everyday lives soon…

high energy physics & computing grid

Documents