Report from HEPCCCReport from HEPCCC
Tobias HaasTobias Haas
XIII HTASCXIII HTASC
13 March, 200313 March, 2003
HEPCCC AgendaHEPCCC Agenda(18 October 2003 at CERN)(18 October 2003 at CERN)
Ian Bird: Deployment Aspects of LCGIan Bird: Deployment Aspects of LCG
Laura Perini: Report from FOCUS Laura Perini: Report from FOCUS
Marcel Kunze: Definition of Tier 0/1/2 centers Marcel Kunze: Definition of Tier 0/1/2 centers
Jacques Delabrouilles: Future computing needs of non-Jacques Delabrouilles: Future computing needs of non-accelerator-based experiments accelerator-based experiments
Fabricio Gagliardi: Status of EDGFabricio Gagliardi: Status of EDG
Larry Price: Coordinating new Grid proposalsLarry Price: Coordinating new Grid proposals
Harvey Newman: Status and Outlook for future networks Harvey Newman: Status and Outlook for future networks
Denis Linglin: The Lyon Biology Grid project Denis Linglin: The Lyon Biology Grid project
iHEPCCC Developments iHEPCCC Developments
Report from FOCUS
The “new” mandate
The issues dealt with in 2002
Perspectives for future
http://t
ilde-djaco
bs.home.ce
rn.ch/~Djaco
bs/Hepcc
cw3/atta
chments/
021018_perini_focu
s.ppt
A new mandate for FOCUS (Forum On Computing: Users and Services)
• Last revised 1st November 2001 Paul Jeffreys, Marco Cattaneo
• Background:– FOCUS’s mandate is changed following the formation of the LHC
Computing Grid Project Structure. – Working assumptions made to determine the revised mandate:– FOCUS’s main role is to look at the implementation, defining,
running, and ultimately phasing out of services.– As previously, it is primarily concerned with IT services run on the
CERN site.– COCOTIME maintains control over resources. – HEPCCC retains responsibility for providing ‘a forum in which the
Directors responsible for Computing at the major European Institutes … are able to discuss the organisation, co-ordination and optimisation of computing in terms both of money and personnel’.
The Terms of Reference• FOCUS will normally meet four times a year as a forum for bringing
together CERN IT providers and users to define and oversee:- the commissioning, operation, prioritisation and decommissioning of services; the services concerned are primarily those on the CERN site;
• it facilitates the agreement of clear definitions of service requirement and provision, and monitors quality of delivery;
• FOCUS concentrates on generic IT activities which have wider relevance than the LHC experiments alone, relate to non-LHC experiments, or disseminate developments from the LHC experiments to other areas;
• FOCUS reports to the Research Sector Management, and in particular to the Director for Technology Transfer and Scientific Computing;
• FOCUS plans and operates across a medium range timescale, typically two years;
• its mandate will be reconsidered at the end of 2002
Activity in 2002
• Issues considered– Decisions
• Preparation for them• Proposals and discussions
– Information exchange• Reports on ongoing IT activities• Updates on status and needs of running experiments
– Matters arising etc.• Way of working
– Draft Agendas prepared at beginning of the year, modify according to the suggestions from IT and users
• Users somewhat too quiet…– Review of membership in progress
March Agenda (1)• 14:10 FOCUS matters
– Welcome of new Chairperson and secretary (H.Hoffmann) 5'Chairperson’s remarks (L.Perini) 5’Review of membership (H.Hoffmann) 10'Agendas for 2002 (J.Boucrot) 10’
• 14:40 COCOTIME matters– Budget situation (W.von Rüden) 10’
Allocations for 2002 (H.Meinhard) 15’
• 15:10 New plan for end of RISC services T.Smith 10’ • 15:30 Tape services issues (TMS, Redwoods, high speed
tape drives) (H.Renshall) 10'
Discussion foreseen for each bullet in every Agenda
March Agenda (2)
• 16:10 LEP long-term data storage and access– Experiments’ requirements (M.Maggi/Aleph) 10'
Possible solutions (H.Renshall) 10’
• 16:40 Linux Service Review (see action 22/4)– Experiments’ feedback (Fons Rademakers) 15’
IT Division plans (B.Panzer-Steindel) 15'
• 17:20 Updates on ongoing IT activities– Security report (see action 21/5.1) (D.Heagerty) 10’
Plans for Objectivity/DB (J.Shiers) 10’
• 17:40 Actions outstanding
June Agenda (1)• 14:15 Reports from running experiments – plans
for 2002 data-taking– COMPASS (B. Gobbo) 10’
– NA 48 (C. Biino) 10’
– NA 49 (A. Sandoval) 10’
• 15:00 Windows 2000 update– Presentation (F. Hemmer) 10’
• 15:20 Plans for external access beyond ACB (VPN etc.)– Presentation (F. Hemmer, D. Heagerty) 10'
June Agenda (2)
• 16:00 Software Development Support and Application software– Development infrastructure (CVS, compilers, tools) ( A. Pfeiffer) 10’ – Libraries and toolkits: Aida, Anaphe (A. Pfeiffer) 15’– Simulation (GEANT 4) (J. Apostolakis) 15’
• 17:00 Updates on ongoing IT activities– Update on Objectivity (J. Shiers) 5’– Linux 7.2 (J. Iven) 10’– CASTOR charging algorithm (H. Renshall) 10’– Legacy UNIX (W. Von Rueden) 5’
• 17:45 Actions outstanding
September Agenda (1)
• 14:15 Update on ongoing IT activities (W. Von Rueden) 40’
• 15:15 Data Import/Export via Network (H. Renshall) 20’
• 16:10 Migration of COMPASS/HARP out of Objectivity (J. Shiers) 15’
• 16:25 Reports from running experiments – experience from 2002 data-taking– NA 48 (F. Marchetto) 10'
HARP (A. Grant) 10'
• 16:55 Computing plans of the EP division (I.Videau) 5’
Snapshot on specific issues
• Linux (slides from Iven)• Objectivity (slides from Shiers)• Mass Storage (slides from Renshall)• Discontinuation of services (Smith, Von Rueden…)• Security (slides from various presentations)
Just some significant examples extracted from FOCUS presentations, many points as interesting as these ones are left out
LINUX Timeline Iven’s June Slides
" last FOCUS presentation: 28.03.02; estimated date for certification: 29.04.02
" “Blocking” products that were late:" CERNLIB: 30.04.02" ANAPHE: 02.05.02" OpenAFS: 13.05.02" ASIS-RPM: 13.05.02
" CERN RedHat Linux 7.2.1 certified on 13.05.02
After the certification• Post-mortem discussion on 24.05.02:
– General certification issues• too late vs. too often• compiler decisions etc..
– Suggestions: • more formal approach, steering body for Linux
certifications (neither CLUG nor FOCUS)• clearly defined environments, explicit dependencies,
“environment responsibles”• Standardize on tools like RPM
• CLUG held on September 13th:– Recommendation: certify 7.3 until November, leave Red Hat 8.0 alone,
look at 8.1 later
– Discussing CLUG future
• Form a “certification team”– Advice Linux team on technical choices, define priorities for releases and
help organising the certification process
– Invitation to user communities sent out early September to nominate a member empowered to speak on their behalf
– Feedback positive, first names received
– Kick-off meeting proposed for 1st half of October
Linux update (1) Von Ruden September slides..
Linux update (2)• Version 6.1.1:
– Currently still running on ~2200 machines
– Main reason: Objectivity (and inertia, still officially supported...)
• CMS: no Objectivity needed for production in 3Q03
• HARP, COMPASS: plan to move off Objectivity in 1Q03
– Red Hat 6.x will become unsupported by RedHat towards end of year (security problems)
– We propose to freeze 6.1.1. now, except for security updates and to stop it in May 2003 (some “pockets” with good justification may go on a bit longer).
Linux update (3)• Version 7.2.1:
• certified in May, currently ~1000 machines
• Only a few large-scale migrations so far (despite pressure during certification):
• LXPLUS migrating to 7.x within 2 months (will be delayed by 7.3.1 certification)
• LXBATCH migrating within 6 months (pending Objectivity and experiments needs)
• LHCb, CMS: code not yet fully tested, may delay until after 7.3.1 certification
• ALICE: 6.1.1 code runs on 7.2
• ATLAS: waiting for next code release (6 weeks) , but no problems expected
• COMPASS: code runs on 7.2, partly migrated desktops, but need Objectivity for data...
–7.2.1 is default version for Desktop Machines
OBjY Shiers’ June slides
• LHC Experiments:– LCG PF (POOL) underway; first release September
• Non-LHC:– Concerns primarily HARP & COMPASS
• Also CHORUS… but manpower questions… on both sides…
– Detailed technical documents being prepared
• LCG PF timescale incompatible with needs of the above• Nevertheless, strong motivation to stay as closely aligned as
possible• These plans are still under discussion…
POOL Timetable
• September 2002: first prototype hybrid data store capable of providing event data persistency in a production setting, and supporting also non-event data persistency. This will be a release for deployment and testing by developers and experts
• Mid-2003: more functional and robust version of the persistency framework. Properly documented and packaged for general use by the experiments
• Early 2005: Completion of the fully functional persistency framework
HARP• 2002 is last year of data taking: total ~30TB
– Simple access patterns; mainly raw data– Persistent<->Transient converters implemented (DATE)
• Need for conditions DB – Propose Oracle implementation vs current Objectivity/DB version: same
(abstract) interface
• Expressed requirement for DB solution– Navigation, transactions, … + metadata
• Propose conversion of Objy data to Oracle 9i R2– 9iR1 bugs fixed; significant performance improvements– HARP / NOMAD data models demonstrated with 9i – Running on standard Intel / Linux systems
• IT-DB contact: Andrea Valassi (+Dirk Geppert)
COMPASS
• Data taking now – 2004 (and beyond??)– LHC-style volumes & data rates
• Few persistent classes: <10
• Propose hand-coded converters– As for HARP, but streaming to files in DATE format
– No need for – or advantage from – more general solution
– DB layer for meta-data + event lookup
• Conditions DB as for HARP– i.e. Oracle implementation of agreed abstract interfaces
• IT-DB contact: Marcin Nowak
COMPASS (September)
• COMPASS will have collected over 300TB of data by end of 2002 run– Approximately ½ the size of BaBar database
• This has to be migrated From old tape media to new From legacy ODBMS format to new
• 300TB @ 10MB / s = 365 days• Such major migrations are likely to reoccur in the future
– At very least from one medium to another
COMPASS / HARP / LCG
• High-level – many similarities:– Hybrid solutions combining Database layer with object streaming– Metadata applications e.g. conditions DB– Actual storage mechanism transparent to applications and may well
change with time!
• LCG: converters (will be) generic– COMPASS / HARP already have hand-written converters from
DATE format
• HARP: streamed objects in DB• COMPASS: streamed objects in files
Objy - Reminder
• Licenses are perpetual
• Use only on officially supported platformshttp://wwwinfo.cern.ch/db/objectivity/servicestools/versionsplatforms.html
TMS (1 of 2) March Renshall’s Slides
• Managed storage implies end users no longer own or manipulate tapes (though we are in a transition phase as regards efficient co-location)
• CASTOR includes its own volume manager already containing much of the functionality of TMS.
• The CERN Tape Management System will not be used by the LHC experiments
• TMS (and its companion SYSREQ server) run on three very old SUN servers. It would be a non-negligible cost (hardware and software) to migrate them.
• We propose to shut down TMS at the end of 2002 providing restricted functionality alternatives for experiments still using it.
TMS (2 of 2)
• To stop TMS we believe we need to enhance support in CASTOR for : – tape location in libraries. Only library names – supported and query is privileged.– write-locking volumes. Currently a privileged operation.– tag (comment) field per volume. Needs to be added.
• CASTOR command line interfaces to the above exist but we could simulate ‘sysreq tms …’ calls if really required. Which experiments would need this ? Largest non-CASTOR user is probably NA48. Already using CASTOR for reconstruction and analysis - could they switch for raw data this year ?
• NA49 Sony tapes would be entered into the CASTOR volume manager.
• We would keep a frozen flat file extract of the final TMS database
• After 2002 Opal will be the only production user of FATMEN and will provide the effort (via S.Oneale) to make it TMS independent
Redwoods (2 of 2)
• We propose to stop all Redwood usage by end 2002 with a contingency up to end April 2003.
• We propose to copy remaining NA48 Redwoods to the new higher density STK drive when available (we would have to borrow drives from LCG until we get our own). We have 150 KCHF for this - about 50 KCHF short so will recuperate later (by CASTOR repack).
• We propose to reduce to 4 maintained drives from now and reactivate 4 drives to help in the NA48 Redwood copying.
• We will prepare a costed plan to upgrade existing drives to the higher density model including reusing existing media at higher density through the planned CASTOR repack functionality.
• We propose to block access to Redwoods already copied to CASTOR when experiments agree but as soon as possible.
Proposal for charging algorithm (1 of 2)
• At the November Focus we proposed to charge for tape usage:– Proposal for charging algorithm in 2002– Take into account:
• Amount of data on tape– Only non-deleted data is billed– Drive compression factor is taken into account
• Tape I/O activity: number of Gbytes transferred to and from tape• Tape device time (to favor fast or lightly loaded disk servers)
• Gigabyte cost in 2002: 2 CHF uncompressed (i.e. on disk occupancy)• We want to modify this to rather charge for the number of mounts and the
number of Gbytes transferred where the normalisation is that a mount/demount cycle has the same cost as transferring 1GB since both take about 2 minutes of real time.
• We suggest (this is a political decision) to set the costs to recuperate the media cost of user tapes data - about 60 KCHF/year (currently single copy only). In practise this is spread among about 15 experiments to pay from 1 to 5 KCHF/year.
CHARGING PROPOSAL DROPPED AFTER FOCUS DISCUSSION
Accelerated Schedule for RISC decommissioning (T.Smith in March)
• AIX– HPSS 3 2002/04/30– Fatmen/HEPDB 1 2002/04/30
• DUX– shiftDELPHI 18 2002/07/24– shiftALEPH 1 2002/07/24– shiftNOMAD 2 2002/06/30 - 2002/12/31– DXPLUS 5 2002/07/24 - 2002/08/31– shiftSLAP 14 2002/12/31
• HP– HPPLUS 3 2002/07/24 - 2002/08/31
• SGI– shiftOPAL 2 2002/07/24– shiftL3 1 2002/07/24 - 2002/12/31– shiftDELPHI 1 2002/07/24
0
100
200
300
400
500
600
2000
0120
0006
2000
1120
0016
2000
2120
0026
2000
3120
0036
2000
4120
0046
2000
5120
0104
2001
0920
0114
2001
1920
0124
2001
2920
0134
2001
3920
0144
2001
4920
0202
2002
0720
0212
2002
1720
0222
2002
2720
0232
2002
3720
0242
2002
4720
0252
2003
05
Year/Week
#C
PU
s
W/NT
AIX
IRIX
HP-UX
DUX
Solaris
RISC Reduction (Von Rueden in September)
2000
2001
2002
2003
Security (Von Rueden in September)• Closure of off-site telnet access to CERN
– Proposed date for closure is before Easter 2003
– Date needs to be coordinated with user communities
• Closure of off-site ftp access to CERN– When non clear-text replacement services are available
• Passwords and sensitive data encrypted– As soon as feasible for the application services
• Registration of systems requiring off-site applications on high numbered ports
– Initially ssh, others added as tools improve
• Additional security checks for systems requesting access in the firewall
– Initially web servers, others as tools improve
VPN (Hemmer in June)
• “Virtual Private Network”• Is a technology that can be used to access any resource that
has been restricted to the CERN Intranet when you are using a computer outside CERN
Using an ISP Using an ISP thru a VPN
Pilot VPN Proposal
• Establish a VPN pilot service– Based on same technology than ACB– Restricted to managed computers on CERN Linux machines and
NICE 2000
• Requirements– A NICE username with a secure password– An explicit registration
• Pilot success criteria's– User needs satisfied– Scalability– Reasonable security checks can be implemented
End of NICE 95/NT (Hemmer in June)• Freeze NICE 95/NT by end of 2001
– Still installable by floppy– No new applications, functionality, etc
• Stop (central) support by mid 2002– No answers from helpdesk– Solution to problems: upgrade to W2K– Still installable by floppy
• Ensure servers are running until end of 2002– NICE 95/NT environment still working
• NICE 95/NT for new PC’s ?– Unlikely new PC’s will be able to run Windows 95– We still have a stock of Windows 95 capable PC’s
Desktop Forum: Plan seems acceptable by all divisions except LHC
FOCUS-centric view
FOCUS
HEPCCC
DesktopForum
Cocotime
EP Forum
Report upwards
Receive advice
Report to Focus
Cross-Cross-
secretariessecretaries
Swap agenda
items
ACCU?
LCG LCG
FOCUS Future
• The LCG cloud is expanding• The number of users who interact with Computing
Services via FOCUS only is contracting
• Which mandate for FOCUS in order to be as useful in LHC/LCG times as it was in the past, and has been also in 2002?
This last question is the conclusion of my talk…
Future computing needs of non-accelerator-based
experiments
Jacques DELABROUILLEPCC- Collège de France
No simple answer..
http://t
ilde-djaco
bs.home.ce
rn.ch/~Djaco
bs/Hepcc
cw3/atta
chments/
021018_delabrouille_non-
accelerator.p
pt
Outline
- Specifics of non-accelerator data
online trigger + events vs. other type of data
- Aspects of "non-event" type data processing
type of data; type of processing; simulations; storage; handling (I/O); computing power
- Some non-accelerator-based experiments
Edelweiss, EROS, ANTARES, HESS (gamma), Supernovae, CMB, ...
- Observational science
observations vs. experiments; numerical investigation; virtual observatories; short-timescale projects; merging information
Specifics of non-accelerator data
I - "Event-type" experiments :
• (Accelerators); • Astroparticle data;
• Neutrinos (Antares, …)• Direct detection of dark matter (Edelweiss, …)• Ultra-High Energy Cosmic Rays (Auger, EUSO, …)• High Energy Gamma rays (HESS, Celeste, …)
• Typically fast acquisition, a lot of storage needed, online trigger, post-processing of a selection of events (event by event)
• The computing can be • massively parallelised• distributed among several computing centers
Specifics of non-accelerator data
II - "Other-type" experiments :
• Cosmic Microwave Background• Planck, Archeops, Olimpo, …• TBytes of timeline data, Gbytes of maps on disk
• Supernovae / Lensing• EROS, SNLS (Megacam), SN Factory, SNAP• Tbytes of maps on disk
• Gravity wave detectors• VIRGO, (LIGO, LISA)
• Others …
Data handling/processing aspects
• Continuous (slow) sampling acquisition
• Modest storage (<100 Tbytes), on disk rather than tape (e.g. HPSS)
• Joint processing of possibly large data (up to Tbytes) for some exp.
• A lot of I/O (random accesses in the whole data set)
• Varying CPU depending on the processing needed
• Increasing importance of (heavy) Monte-Carlo processing of simulated data sets (possibly suited for distributed processing)
Exemple I : Edelweiss
Direct detection of dark matter via interaction of WIMPS in a large crystal (bolometer) cooled to ~10K at Modane (present)
"Event-type" experiment
20 100 1000 detectors (5-10 years scale),
~10 Mbytes/day/detector in data acquisition mode
~100 Mbytes/day/detector in calibration mode
Somewhat larger numbers in the test phase (less online compression)
(Overall modest as compared to LHC experiments)
Exemple II : HESS
Observation of fluorescence light due to high energy gamma raysusing 4 telescopes with a 1000-pixel focal plane (phototubes)(present)
"Event-type" experiment;
About 40 Tbytes of data per year, x2 or x3 in later phases
CPU annual needs ~few 105 hours
Exemple III : ANTARES
Underwater neutrino telescope collecting data in an array of ~1000 phototubes in the mediterranean sea (being built)
"Event-type" experiment
First level trigger produces ~1010 events per yr (100 Tbytes)
Second-level trigger to identify events well suited to distributed computing time
CPU annual needs ~few 107 hours
Exemple I : Edelweiss
Direct detection of dark matter via interaction of WIMPS in a large crystal (bolometer) cooled to ~10K at Modane (present)
"Event-type" experiment
20 100 1000 detectors (5-10 years scale),
~10 Mbytes/day/detector in data acquisition mode
~100 Mbytes/day/detector in calibration mode
Somewhat larger numbers in the test phase (less online compression)
(Overall modest as compared to LHC experiments)
Exemple II : HESS
Observation of fluorescence light due to high energy gamma raysusing 4 telescopes with a 1000-pixel focal plane (phototubes)(present)
"Event-type" experiment;
About 40 Tbytes of data per year, x2 or x3 in later phases
CPU annual needs ~few 105 hours
Exemple III : ANTARES
Underwater neutrino telescope collecting data in an array of ~1000 phototubes in the mediterranean sea (being built)
"Event-type" experiment
First level trigger produces ~1010 events per yr (100 Tbytes)
Second-level trigger to identify events well suited to distributed computing time
CPU annual needs ~few 107 hours
Exemple IV : EROS
A good exemple of (qualitatively) very different data processing
Monitoring of ~107 star luminosity curves during 6 years (1996-2002)
1300 images of 4x106 pixels per night (10 Gbytes/night), for a total of 2.5 million images and ~20 TBytes of data in 6 yrs
Very long events ( up to ~6 months or more!)
One very specific aspect : matrix transposition .The data comes "per day", but is analysed "per star" in the end
Exemple V : Supernovae
Monitoring images of many galaxies to detect supernovae explosions
Three main (complementary) experiments• SNLS : Supernovae Legacy Survey• SNF (SNIFS) : The Supernovae "Factory" ("calibration")• SNAP : satellite dedicated to wide-field photometry
Important (?) aspect : transient phenomena + alert• Detecting a supernova candidate online triggers more data taking (spectra) during the few next days• Requirement on computing power availability
Large fraction of data has to be on disk
SNF/SNIFS - SNLS
SNF : Supernovae Factory (SNIFS) : search for a small number (~100) nearby supernovae (2002-2007)
Small : 4 Tbytes total, 40,000 hrs of CPU
SNLS : Supernovae Legacy Survey : search for distant supernovae with Megacam (2002-2007)
Same data used by different groups/institutes for different purposes (weak shear)
10 Tbytes/yr, ~50 Tbytes total, ~500,000 hrs of CPU
Data processing cut in simple tasks (read-process-write) well suited for parallel processing
SNAP
SNAP : Supernovae Acceleration Probe (begin 2008-2012?)
1000 Tbytes for three years
10-50 Tbytes disk
100 processors (1 GHz)
Note : particularly large data set for a space-borne experiment
Exemple VI : CMB experiments
• Observation of spatial brightness/polarisation fluctuations of the Cosmic Microwave Background emission (no intrinsic time variation!)
• PLANCK (2007-2009), archeops, olimpo, ...
~100 two-year long timelines sampled at ~100 Hz (1TByte)
• Timelines are combined in a joint way to form ~50 sky maps of 10 million pixel each
• A large fraction (>20 Gbytes) of the data must be stored in (shared) memory, the rest on disk with rapid access (a lot of I/O).
Most typical operations:•matrix (or filtering) operations M.x on the timelines•FFTs
The map-making problem
y = P x + n
Data stream1010 samples
Pointing matrix1010 x 107 entries
(sparse)
map107 pixels
noise stream1010 samples
Current plans
Iterative close-to-optimal processing solutions (map-making) scaling as
•Memory O(n)•FLOPS O(n log n)
Iterative processing : map-making steps repeated many times
Errors computed by Monte-Carlo : repeat the process many times
Most of processing done on a dedicated machine, except well defined tasks suited for parallel/vectorial/distributed computing (?)
Observational science ...
OBSERVATIONS ≠ EXPERIMENTS
We have no direct access to the objects studied
Transient rare events require "trigger" + observe• Supernovae, Gamma-ray bursts requirements on computing power availability
Need numerical simulations
Numerical investigations
• Growing need for computationally very intensive numerical investigations (both large scale and Monte-Carlo)
• Astrophysical complex objects/phenomena (supernovae explosions, various extreme events)
• Large size systems(the Universe …! )
• Computationally very demanding - virtually no limit(hundreds... of hours on huge parallel or vectorial computers)
• Priorities/scales still subject of discussion in the community
Putting observations together ...
Very different "observations" (in terms of experimental setup, data description, data processing) are various means of accessing higher-level information
• Observation at various wavelengths (radio, optical, gamma…) or with various carriers (photons, neutrinos, protons, GW...)
• Observation of different phenomena depending each in a non-trivial way on a common set of global parameters
• example : the rate of SN explosions, the statistics of CMB fluctuations, the properties of galaxy clusters… all depend on a small set of cosmological parameters
Qualitative evolution
Sparse, incomplete data
rich, complex, diverseand complementary data sets
Virtual observatories / data mining ...
Access needed to virtual observatories (data bases)
Type of access still unclearfrom copy once relevant data in repositories
to multiple random access to distributed data
Objective of access still unclearfrom merge high level (processed) information
to merge low-level data for the processing
Virtual observatories, Access to diverse data
Access to computing
power
Data merging,Joint analyses
Numerical investigations
Short timescale projects
• (Competitive) area of research still open to surprises and new ideas (theoretical and experimental) generating short timescale projects :
• EROS (microlensing)• Archeops (CMB), ...
New specific computing needs may appear tomorrow...
Summary
• Data type (events vs. other type of data) "event-type" processing (AUGER, ANTARES, HESS, EDELWEISS…) not fundamentally different for non-accelerator and accelerator based experiments (different scales though)
• Non "event-type" data (Archeops, PLANCK, SNLS, virgo?) typically requires • global analysis of a (moderately) large quantity of data• large memory and disk rather than storage (I/O intensive)• needed CPU time very much data-set (and method) dependent• needed CPU time still subject to debating in the communities
• Observations (vs. Experiments)Joint analysis of distributed complex data setsProcessing availability for "alerts" (supernovae, gamma-ray bursts...)
• Generation of "numerical data sets" : • growing computationnaly simulation needs• probably major source of need for distributed computing time
ICFA SCIC Report on WorldICFA SCIC Report on World Networks to the HEPCCC Networks to the HEPCCC
Harvey B. NewmanHarvey B. Newman California Institute of TechnologyCalifornia Institute of Technology
HEPCCC MeetingHEPCCC MeetingOctober 18, 2002October 18, 2002
http://t
ilde-djaco
bs.home.ce
rn.ch/~Djaco
bs/Hepcc
cw3/atta
chments/
021018_newman_icfasc
ic.ppt
Transatlantic Net WG (HN, L. Price) Bandwidth Requirements (2001) [*] Transatlantic Net WG (HN, L. Price) Bandwidth Requirements (2001) [*]
2001 2002 2003 2004 2005 2006
CMS 100 200 300 600 800 2500
ATLAS 50 100 300 600 800 2500
BaBar 300 600 1100 1600 2300 3000
CDF 100 300 400 2000 3000 6000
D0 400 1600 2400 3200 6400 8000
BTeV 20 40 100 200 300 500
DESY 100 180 210 240 270 300
CERN BW
155-310
622 2500 5000 10000 20000
[*] [*] Installed BW. Maximum Link Occupancy 50% Installed BW. Maximum Link Occupancy 50%
AssumedAssumed
See http://gate.hep.anl.gov/lprice/TANSee http://gate.hep.anl.gov/lprice/TAN
History – One large Research Site
Projections: 0.5 to 24 Tbps by ~2012
Much of the Traffic:SLAC IN2P3/RAL/INFN;
via ESnet+France;Abilene+CERN
ESnet History Research ISP
100% growth in traffic/yr for last 12 yearsContinuous upgradesIncrease packet size (bulk throughput apps)
As of 10/2002 Plan Accelerated: 10 Gbps
Asap
Baseline BW for the US-CERN Link: HENP Transatlantic WG (DOE+NSF)Baseline BW for the US-CERN Link: HENP Transatlantic WG (DOE+NSF)
DataTAG 2.5 Gbps Research Link since Summer 2002DataTAG 2.5 Gbps Research Link since Summer 2002 10 Gbps Link for Production (2.5) & Research by 1Q 200310 Gbps Link for Production (2.5) & Research by 1Q 2003
Transoceanic Networking
Integrated with the Abilene,
TeraGrid, Regional Nets
and Continental Network
Infrastructuresin US, Europe,
Asia, South America
Baseline evolution typicalBaseline evolution typicalof major HENPof major HENP
links 2001-2006 links 2001-2006
NLNLSURFnet
GENEVA
UKUKSuperJANET4
ABILENE
ABILENE
ESNETESNET
CALREN
CALREN
ItItGARR-B
GEANT
NewYork
FrFrRenater
STAR-TAP
STARLIGHT
DataTAG ProjectDataTAG Project
EU-Solicited Project. EU-Solicited Project. CERNCERN, PPARC (UK), Amsterdam (NL), and INFN (IT);, PPARC (UK), Amsterdam (NL), and INFN (IT);and US (DOE/NSF: UIC, NWU and Caltech) partnersand US (DOE/NSF: UIC, NWU and Caltech) partners
Main Aims: Main Aims: Ensure maximum interoperability between US and EU Grid ProjectsEnsure maximum interoperability between US and EU Grid ProjectsTransatlantic Testbed for advanced network researchTransatlantic Testbed for advanced network research
2.5 Gbps Wavelength Triangle 7/02 (10 Gbps Triangle by Spring 2003)2.5 Gbps Wavelength Triangle 7/02 (10 Gbps Triangle by Spring 2003)
Wave
Triangle
Assisted by Loans: Level3 (OC192) and Cisco (10GbE and 16X1GbE)Assisted by Loans: Level3 (OC192) and Cisco (10GbE and 16X1GbE)
iGrid2002: OC192+OC48 iGrid2002: OC192+OC48 SetupSetup
September 2002September 2002
Short Term Donation from Level 3
Short Term Donation from Level 3
Argonne
Starlight-Chicago
Amsterdam
CERN
LHCnet Network : Late 2002-2003LHCnet Network : Late 2002-2003
Development and testsDevelopment and tests
Abilene MRENESnet STARTAPNASA
Linux PC forPerformance tests
& Monitoring
CERN -Geneva
GEANT Switch IN2P3 WHO
Linux PC forPerformance tests
& Monitoring
Caltech/DoE PoP – StarLight Chicago
0.622 to 2.5G 0.622 to 2.5G (Prod.)(Prod.)
Cisco 7609CERN
Cisco 7609Caltech(DoE)
Alcatel 7770DataTAG(CERN)
Cisco 7606DataTAG(CERN)
Juniper M10DataTAG(CERN)
Cisco 7606Caltech(DoE)
Juniper M10Caltech(DoE)
Alcatel 7770DataTAG(CERN)
2.5 – 7.5G 2.5 – 7.5G (R&D)(R&D)
Optical Mux/DmuxAlcatel 1670
Optical Mux/DmuxAlcatel 1670
HENP Scenario Limitations:Technologies and Costs
HENP Scenario Limitations:Technologies and Costs
Router Technology and Costs Router Technology and Costs (Ports and Backplane) (Ports and Backplane)
Computer CPU, Disk and Bus, I/O ChannelComputer CPU, Disk and Bus, I/O Channel Speeds to Send and Receive Data Speeds to Send and Receive Data
Link Costs: Unless Dark Fiber (?)Link Costs: Unless Dark Fiber (?) MultiGigabit Transmission Protocols MultiGigabit Transmission Protocols
End-to-End End-to-End ““100 GbE” Ethernet (or something else) by100 GbE” Ethernet (or something else) by
~2006: for LANs to match WAN speeds ~2006: for LANs to match WAN speeds
ICFA and International Networking ICFA and International Networking
ICFA Statement on Communications in Int’l HEPICFA Statement on Communications in Int’l HEPCollaborations of October 17, 1996Collaborations of October 17, 1996 See See http://www.fnal.gov/directorate/icfa/icfa_communicaes.htmlhttp://www.fnal.gov/directorate/icfa/icfa_communicaes.html
““ICFA urges that all countries and institutions wishing ICFA urges that all countries and institutions wishing to participate even more effectively and fully in to participate even more effectively and fully in international HEP Collaborations should:international HEP Collaborations should: Review their operating methods to ensure they Review their operating methods to ensure they
are fully adapted to remote participation are fully adapted to remote participation Strive to provide the necessary communications Strive to provide the necessary communications
facilities and adequate international bandwidth” facilities and adequate international bandwidth”
ICFA Network Task Force: 1998 Bandwidth Requirements Projection
(Mbps)
ICFA Network Task Force: 1998 Bandwidth Requirements Projection
(Mbps)1998 2000 2005
BW Utilized Per Physicist(and Peak BW Used)
0.05 - 0.25(0.5 - 2)
0.2 – 2(2-10)
0.8 – 10(10 – 100)
BW Utilized by a UniversityGroup
0.25 - 10 1.5 - 45 34 - 622
BW to a Home Laboratory OrRegional Center
1.5 - 45 34 - 155 622 - 5000
BW to a Central LaboratoryHousing One or More MajorExperiments
34 - 155 155 - 622 2500 - 10000
BW on a Transoceanic Link 1.5 - 20 34 - 155 622 - 5000
100–1000 X Bandwidth Increase Foreseen for 1998-2005100–1000 X Bandwidth Increase Foreseen for 1998-2005See the ICFA-NTF Requirements Report:See the ICFA-NTF Requirements Report:
http://l3www.cern.ch/~newman/icfareq98.html
NTF
ICFA Standing Committee on Interregional Connectivity (SCIC)
ICFA Standing Committee on Interregional Connectivity (SCIC)
Created by ICFA in July 1998 in Vancouver ; Following ICFA-NTFCreated by ICFA in July 1998 in Vancouver ; Following ICFA-NTF CHARGE: CHARGE:
Make recommendations to ICFA concerning the Make recommendations to ICFA concerning the connectivity between connectivity between the Americasthe Americas, Asia and Europe, Asia and Europe As part of the process of developing theseAs part of the process of developing these
recommendations, the committee should recommendations, the committee should Monitor traffic Monitor traffic Keep track of technology developmentsKeep track of technology developments Periodically review forecasts of future Periodically review forecasts of future
bandwidth needs, and bandwidth needs, and Provide early warning of potential problemsProvide early warning of potential problems
Create subcommittees when necessary to meet the chargeCreate subcommittees when necessary to meet the charge The chair of the committee should report to ICFA once perThe chair of the committee should report to ICFA once per
year, at its joint meeting with laboratory directors (Feb. 2003)year, at its joint meeting with laboratory directors (Feb. 2003) Representatives: Major labs, ECFA, ACFA, NA Users, S. AmericaRepresentatives: Major labs, ECFA, ACFA, NA Users, S. America
ICFA-SCIC Core MembershipICFA-SCIC Core Membership
Representatives from major HEP Representatives from major HEP laboratories:laboratories:
Manuel Delfino Manuel Delfino (CERN)(CERN)
to W. Von Ruedento W. Von Rueden Michael Ernst Michael Ernst (DESY)(DESY) Matthias Kasemann (FNAL) Matthias Kasemann (FNAL) Yukio Karita Yukio Karita (KEK)(KEK) Richard Mount Richard Mount (SLAC)(SLAC)
User Representatives User Representatives Richard Hughes-Jones (UK)Richard Hughes-Jones (UK) Harvey Newman Harvey Newman (USA) (USA)
Dean Karlen Dean Karlen (Canada)(Canada) For Russia: For Russia:
Slava Ilyin Slava Ilyin (MSU)(MSU)
ECFA representatives:ECFA representatives:
Frederico Ruggieri (INFN Frederico Ruggieri (INFN Frascati),Frascati),Denis Linglin (IN2P3, Denis Linglin (IN2P3, Lyon)Lyon)
ACFA representatives:ACFA representatives:
Rongsheng XuRongsheng Xu (IHEP Beijing) (IHEP Beijing)
HwanBae Park HwanBae Park (Kyungpook Nat’l (Kyungpook Nat’l University)University)
For South America:For South America: Sergio F. Novaes Sergio F. Novaes (University of Sao Paulo)(University of Sao Paulo)
ICFA SCIC Meetings[*] and TopicsICFA SCIC Meetings[*] and Topics
Focus on the Digital Divide This YearFocus on the Digital Divide This Year Identification of problem areas; work on ways to improveIdentification of problem areas; work on ways to improve
Network Status and Upgrade Plans in Each CountryNetwork Status and Upgrade Plans in Each Country Performance (Throughput) Evolution in Each Country,Performance (Throughput) Evolution in Each Country,
and Transatlantic and Transatlantic Performance Monitoring World-Overview Performance Monitoring World-Overview (Les Cottrell, IEPM Project) (Les Cottrell, IEPM Project) Specific Technical Topics (Examples):Specific Technical Topics (Examples):
Bulk transfer, QoS, Collaborative Systems, Security, VOIPBulk transfer, QoS, Collaborative Systems, Security, VOIP Preparation of Reports to ICFA (Lab Directors’ Meetings)Preparation of Reports to ICFA (Lab Directors’ Meetings)
Last Report: World Network Status and Outlook - Feb. 2002Last Report: World Network Status and Outlook - Feb. 2002 Next Report: Next Report: Digital Divide, +Digital Divide, + Monitoring, Advanced Monitoring, Advanced
Technologies, and Requirements Evolution – Feb. 2003Technologies, and Requirements Evolution – Feb. 2003
[*] Six Meetings in 2002; Next at KEK In December 13.[*] Six Meetings in 2002; Next at KEK In December 13.
SCIC Sub-Committees SCIC Sub-CommitteesWeb Page Web Page http://cern.ch/ICFA-SCIC/http://cern.ch/ICFA-SCIC/
Monitoring:Monitoring: Les Cottrell Les Cottrell (http://www.slac.stanford.edu/xorg/icfa/scic-netmon) (http://www.slac.stanford.edu/xorg/icfa/scic-netmon) With Richard Hughes-Jones (Manchester), With Richard Hughes-Jones (Manchester), Sergio Novaes (Sao Sergio Novaes (Sao Paolo); Paolo); Sergei Berezhnev (RUHEP), Fukuko Yuasa (KEK), DanielSergei Berezhnev (RUHEP), Fukuko Yuasa (KEK), DanielDavids (CERN), Sylvain Ravot (Caltech), Shawn McKee (Michigan)Davids (CERN), Sylvain Ravot (Caltech), Shawn McKee (Michigan)
Advanced Technologies:Advanced Technologies: Richard Hughes-Jones w Richard Hughes-Jones with ith Vladimir Vladimir Korenkov (JINR, Dubna), Olivier Martin(CERN), Korenkov (JINR, Dubna), Olivier Martin(CERN), Harvey NewmanHarvey Newman
End-to-end Connectivity:End-to-end Connectivity: Richard Mount (SLAC) Richard Mount (SLAC) With Michael Ernst, Denis Linglin, With Michael Ernst, Denis Linglin, Alexandre Sztajnberg (Rio, Brazil)Alexandre Sztajnberg (Rio, Brazil)
The Digital Divide:The Digital Divide: Alberto Santoro (Rio, Brazil) Alberto Santoro (Rio, Brazil) With Slava Ilyin, Yukio Karita, David O. WilliamsWith Slava Ilyin, Yukio Karita, David O. Williams Also Dongchul Son (Korea), Hafeez Hoorani (Pakistan), Also Dongchul Son (Korea), Hafeez Hoorani (Pakistan),
Sunanda Banerjee (India), Vicky White (FNAL)Sunanda Banerjee (India), Vicky White (FNAL) Key Requirements:Key Requirements: Harvey Newman Harvey Newman
Also Charlie Young (SLAC)Also Charlie Young (SLAC)
Network Progress andIssues for Major Experiments
Network Progress andIssues for Major Experiments
Major R&E Networks have weathered the economic “storm” Major R&E Networks have weathered the economic “storm” Backbones & major links advancing rapidly to 10 Gbps Backbones & major links advancing rapidly to 10 Gbps
rangerange ““Gbps” end-to-end throughput data flows have beenGbps” end-to-end throughput data flows have been
tested; will be in production soon (in 1-2 years) tested; will be in production soon (in 1-2 years) Network advances are changing the view of the net’s rolesNetwork advances are changing the view of the net’s roles
Progress to and beyond 10 Gbps within next few yearsProgress to and beyond 10 Gbps within next few years Likely to have a profound impact on the experiments’ Likely to have a profound impact on the experiments’
Computing Models, and bandwidth requirementsComputing Models, and bandwidth requirements More dynamic view: dynamic path provisioning;More dynamic view: dynamic path provisioning;
GB to TB data transactionsGB to TB data transactions Net R&D Driven by Advanced integrated applications, such Net R&D Driven by Advanced integrated applications, such
as Data Grids, that rely on seamless LAN and WAN operation as Data Grids, that rely on seamless LAN and WAN operation With reliable, quantifiable (monitored), high performanceWith reliable, quantifiable (monitored), high performance
All of the above will further open the Digital Divide All of the above will further open the Digital Divide chasm. We need to take action chasm. We need to take action
ICFA SCIC: R&E Backbone and International Link Progress
ICFA SCIC: R&E Backbone and International Link Progress
GEANT Pan-European BackboneGEANT Pan-European Backbone ( (http://www.dante.net/geanthttp://www.dante.net/geant)) Now interconnects >31 countries; many trunks Now interconnects >31 countries; many trunks 2.5 and 10 2.5 and 10
GbpsGbps UKUK: : SuperJANET Core at SuperJANET Core at 10 Gbps10 Gbps
2.5 Gbps NY-London, with 622 Mbps to ESnet and Abilene2.5 Gbps NY-London, with 622 Mbps to ESnet and Abilene France (IN2P3): France (IN2P3): 2.5 Gbps2.5 Gbps RENATER backbone by mid-October RENATER backbone by mid-October
Lyon-CERN Link being Upgraded to 1 Gbps EthernetLyon-CERN Link being Upgraded to 1 Gbps Ethernet Proposal for dark fiber to CERN by end 2003 Proposal for dark fiber to CERN by end 2003
SuperSINET (Japan):SuperSINET (Japan): 10 Gbps IP and 10 Gbps Wavelength10 Gbps IP and 10 Gbps Wavelength Core Core Tokyo to NY Links: Now 2 X 2.5 Gbps; Need to get to StarlightTokyo to NY Links: Now 2 X 2.5 Gbps; Need to get to Starlight
CA*net4 (Canada):CA*net4 (Canada): Interconnect customer-owned dark fiber Interconnect customer-owned dark fiber nets across Canada at nets across Canada at 10 Gbps10 Gbps, started July 2002, started July 2002 ““Lambda-Grids” by ~2004-5Lambda-Grids” by ~2004-5
GWIN (Germany):GWIN (Germany): 2.5 Gbps Core2.5 Gbps Core; Connect to US at 2 X 2.5 Gbps;; Connect to US at 2 X 2.5 Gbps;Support for SILK Project: Satellite links to FSU Republics Support for SILK Project: Satellite links to FSU Republics
Russia: Russia: Now 155 Mbps links to MoscowNow 155 Mbps links to Moscow Moscow-Starlight Link to Moscow-Starlight Link to 155 Mbps155 Mbps (US NSF + Russia (US NSF + Russia
Support)Support) Moscow-Stockholm Link; connect to GEANT at Moscow-Stockholm Link; connect to GEANT at 155 Mbps155 Mbps
R&E Backbone and Int’l Link ProgressR&E Backbone and Int’l Link Progress
Abilene (Internet2) UpgradeAbilene (Internet2) Upgrade from 2.5 to from 2.5 to 10 Gbps10 Gbps Underway Underway Encourage high throughput use for targeted applications Encourage high throughput use for targeted applications
ESNET: UpgradeESNET: Upgrade to 10 Gbps “Now” to 10 Gbps “Now” US-CERNUS-CERN
to 622 Mbps in August; Move to STARLIGHTto 622 Mbps in August; Move to STARLIGHT 2.5G Research Triangle from August 2002 2.5G Research Triangle from August 2002
STARLIGHT-CERN-NL; to 10G in 2003STARLIGHT-CERN-NL; to 10G in 2003 SLAC + IN2P3 (BaBar) SLAC + IN2P3 (BaBar)
To ~100 Mbps throughput over US-CERN and Renater linksTo ~100 Mbps throughput over US-CERN and Renater links 600 Mbps Throughput is BaBar Target for this Year 600 Mbps Throughput is BaBar Target for this Year
(with ESnet and IN2P3 Link Upgrades)(with ESnet and IN2P3 Link Upgrades) FNALFNAL
ESnet Link Upgrade to 622 MbpsESnet Link Upgrade to 622 Mbps Plans for dark fiber to STARLIGHT, planned for this FallPlans for dark fiber to STARLIGHT, planned for this Fall
NY-Amsterdam NY-Amsterdam Donation Donation from Tyco, from Tyco, September 2002: September 2002: Arranged by IEEAF: Arranged by IEEAF: 622 Gbps+10 Gbps Research 622 Gbps+10 Gbps Research WavelengthWavelength
National Research Networks in Japan
National Research Networks in Japan
SuperSINET SuperSINET Started operation January 4, 2002Started operation January 4, 2002 Support for 5 important areas:Support for 5 important areas:
HEP,HEP, Genetics, Nano-Technology, Genetics, Nano-Technology,Space/Astronomy, Space/Astronomy, GRIDsGRIDs
Provides 10 Provides 10 ’s:’s: 10 Gbps IP connection 10 Gbps IP connection Direct intersite GbE linksDirect intersite GbE links 7 Univ. Connected; 7 Univ. Connected; Add 3 More this Month Add 3 More this Month
HEPnet-J HEPnet-J Reconstructed with Reconstructed with MPLS-VPN in SuperSINET MPLS-VPN in SuperSINET
Proposal: Two TransPacific Proposal: Two TransPacific 2.5 Gbps Wavelengths, and 2.5 Gbps Wavelengths, and Japan-CERN Grid Testbed by ~2003 Japan-CERN Grid Testbed by ~2003
Tokyo
Osaka
Nagoya
Internet
Osaka U
Kyoto U
ICR
Kyoto-U
Nagoya U
NIFS
NIG
KEK
Tohoku U
IMS
U-TokyoNAO
U Tokyo
NII Hitot.
NII Chiba
IP
WDM path
IP router
OXC
ISAS
National R&E Network ExampleGermany: DFN Transatlantic Connectivity 2002
STM 4
STM 16
STM 16
FSU Connections via satellite:FSU Connections via satellite:Yerevan, Minsk, Almaty, BaikalYerevan, Minsk, Almaty, Baikal Speeds of 32 - 512 kbpsSpeeds of 32 - 512 kbps
SILK Project (2002): NATO fundingSILK Project (2002): NATO funding Links to Caucasus and CentralLinks to Caucasus and Central Asia (8 Countries) Asia (8 Countries)
In 2001-2 64-512 kbpsIn 2001-2 64-512 kbpsProposed VSAT for 10-50 X BW:Proposed VSAT for 10-50 X BW: NATO + State Funding NATO + State Funding
2 X OC48: NY-Hamburg 2 X OC48: NY-Hamburg and NY-Frankfurtand NY-Frankfurt
Direct Peering to Abilene (US) and Direct Peering to Abilene (US) and Canarie (Canada)Canarie (Canada)
UCAID will adding another 2 UCAID will adding another 2 OC48’s; in a Proposed Global OC48’s; in a Proposed Global Terabit Research Network (GTRN) Terabit Research Network (GTRN)
Leading-Edge e2e services & experimentalnetwork facilities via MetaPoPs and inter-gigapop links for research & next gen. tech., arch., grids, DTF expansion, content, sensors, apparatus …Up to 40 dedicated 10 gigabit waves with up to hundreds of dedicated gigabit links system-wide.
8/14/02
10 gbsTyco IEEAFdonation
10 gbsTyco
IEEAF Donation
MetaPoPs & Core Nodes
ADM sites
Desired Expansion PoPs/ADM
National LightRail” (NLR) NLR Under Consideration
International Broadband
Metro/WAN owned Fiber ‘Rings’ connecting strategic R&E endpoints.
PIT
BOS
NY
DC
LA
SUN
SEA
DEN
CHI
National LightRail: cost-effective owned lit R&E fiber fabric
ATL
CLV?
Canariefabric
Raleigh?
7/03 footprint
IEPM: PingER DeploymentIEPM: PingER Deployment
Measurements fromMeasurements from 34 monitors in 14 countries34 monitors in 14 countries Over 600 remote hostsOver 600 remote hosts Over 72 countries Over 72 countries Over 3300 monitor-remote site pairsOver 3300 monitor-remote site pairs Measurements go back to Jan-95Measurements go back to Jan-95 Reports on RTT, loss, reachability, jitter, Reports on RTT, loss, reachability, jitter,
reorders, duplicates …reorders, duplicates …
Countries monitoredCountries monitored Contain 78% of Contain 78% of
world populationworld population 99% of online users 99% of online users
of the Internetof the Internet Mainly A&R sitesMainly A&R sites
Monitoring Sites
Remote Sites
History – Loss Quality (Cottrell)
Fewer sites have very poor to dreadful performance More have good performance (< 1% Loss)
History - Throughput Quality Improvements from US
Bandwidth of TCP < MSS/(RTT*Sqrt(Loss)) (1)
(1) Macroscopic Behavior of the TCP Congestion Avoidance Algorithm1) Macroscopic Behavior of the TCP Congestion Avoidance Algorithm, , Matthis, Semke, Mahdavi, Ott, Computer Communication Review 27(3), Matthis, Semke, Mahdavi, Ott, Computer Communication Review 27(3), July 1997July 1997
80% annual improvement Factor ~100/8 yr
Progress: but Digital Divide is Maintained
True End to End Experience
User perception ApplicationOperating systemHost IP stackHost network cardLocal Area Network Campus backbone
networkCampus link to regional
network/GigaPoPGigaPoP link to
Internet2 national backbones
International connections
EYEBALL
APPLICATION
STACK
JACK
NETWORK
. . .
. . .
. . .
. . .
Work on the Digital Divide:Several Perspectives
Work on the Digital Divide:Several Perspectives
Identify & Help Solve Technical Problems: Identify & Help Solve Technical Problems: Nat’l, Regional, Last 10/1/0.1 kmNat’l, Regional, Last 10/1/0.1 km
Inter-Regional Proposals (Example: Brazil)Inter-Regional Proposals (Example: Brazil) US NSF Proposal (10/2002); possible EU LIS ProposalUS NSF Proposal (10/2002); possible EU LIS Proposal
Work on Policies and/or Pricing: pk, in, br, cn, SE Europe, Work on Policies and/or Pricing: pk, in, br, cn, SE Europe, …… E.g. RoEduNet (2-6 Mbps); Pricing not so differentE.g. RoEduNet (2-6 Mbps); Pricing not so different
from US-CERN price in 2002 for a few Gbpsfrom US-CERN price in 2002 for a few Gbps Find Ways to work with vendors, NRENs, and/or Gov’tsFind Ways to work with vendors, NRENs, and/or Gov’ts
Use Model Cases: Installation of new advanced fiber Use Model Cases: Installation of new advanced fiber infrastructures; Convince Neighboring Countriesinfrastructures; Convince Neighboring Countries Poland (to 5k km Fiber); Hungary; Slovakia; IrelandPoland (to 5k km Fiber); Hungary; Slovakia; Ireland
Exploit One-off Solutions: Exploit One-off Solutions: E.g. extend the SILK Project E.g. extend the SILK Project (DESY/FSU satellite links) to a SE European site(DESY/FSU satellite links) to a SE European site
Work with other organizations:Work with other organizations: Terena, Internet2, AMPATH, Terena, Internet2, AMPATH, IEEAF, UN, etc. IEEAF, UN, etc. to help with technical and/or political sol’nsto help with technical and/or political sol’ns
GEANT
155Mbps
34Mbps 34Mbps
34Mbps
34Mbps
34MbpsAnnual Cost
> 1 MEuro
Romania: 155 Mbps to GEANT and Bucharest;
Inter-City Links of 2-6 Mbps
Romania: 155 Mbps to GEANT and Bucharest;
Inter-City Links of 2-6 Mbps
Yugoslavia
Digital Divide Committee
HUNGARY
Losses: World by sub-region, Jan ‘02• <1%=good, <2.5%=acceptable, < 5%=poor, >5%=bad
• Russia, S America bad
• Balkans, M East, Africa, S Asia, Caucasus poor
Monitored Region \ Monitor Country
BR (1)
CA (2)
DK (1)
DE (1)
HU (1)
IT (3)
JP (2)
RU (2)
CH (1)
UK (3)
US (16) Avg Region
Avg -(H
Avg NA + WEU + JP Pairs
COM 0.2 0.3 0.3 0.2 COM 0.27 23
Canada 1.8 1.6 0.3 0.5 9.0 0.3 1.4 21.7 0.7 0.7 0.5 3.5 Canada 0.74 126
US 0.4 2.6 0.2 0.3 8.0 0.1 1.4 13.8 0.3 1.3 0.9 2.7 US 0.88 2149
C America 0.9 0.9 C America 0.89 19
Australasia 0.8 1.8 1.3 Australasia 1.30 18
E Asia 1.2 3.5 1.0 1.1 9.0 0.9 2.0 5.2 1.5 1.4 1.5 2.6 E Asia 1.61 215
Europe 0.4 5.6 0.3 0.5 5.4 0.4 1.3 15.5 1.1 1.0 1.0 2.9 Europe 1.38 852
NET 1.7 6.2 1.0 1.3 8.0 1.6 3.6 21.9 0.7 0.8 0.9 4.3 NET 2.00 85
FSU- 4.5 0.5 9.8 0.5 1.6 11.2 4.3 1.2 2.0 4.0 FSU- 2.09 48
Balkans 3.8 3.8 Balkans 3.83 109
Mid East 4.6 1.4 3.0 8.5 2.8 3.2 11.8 2.0 2.5 2.1 4.2 Mid East 2.70 57
Africa 5.8 1.5 12.0 1.2 4.2 11.9 2.0 1.9 2.5 4.8 Africa 2.72 45
Baltics 5.3 0.8 2.3 7.7 2.2 3.5 10.8 4.8 2.1 3.9 4.3 Baltics 3.12 67
S Asia 1.6 7.3 0.1 3.1 9.2 3.0 3.9 17.9 1.5 3.1 3.0 4.9 S Asia 3.12 97
Caucasus 3.2 3.2 Caucasus 3.22 19
S America 24.1 11.3 0.6 0.9 6.7 12.9 7.7 23.0 9.3 1.1 6.6 9.5 S America 6.30 203
Russia 35.9 24.1 22.2 13.4 23.8 21.7 13.6 0.7 8.7 24.1 12.7 18.3 Russia 17.57 91
Avg 7.5 6.9 2.8 2.4 9.8 3.7 3.9 13.8 3.1 3.2 2.8 4.4 Avg 3.16
Pairs 64 144 54 67 70 203 190 114 209 192 1990 Pairs
NREN Core Network Size (Mbps-km):http://www.terena.nl/compendium/2002
NREN Core Network Size (Mbps-km):http://www.terena.nl/compendium/2002
Logarithmic Scale
1k
100k
100
100M
10M
1M
10kRo
It
PlGrIr
Ukr
Hu
Cz
Es
Nl
Fi
Ch
Lagging
In Transition
Leading
Advanced
Cr
Digital Divide ActivitiesDigital Divide Activities Questionnaire Being Distributed (Discuss At ICFA Meeting)Questionnaire Being Distributed (Discuss At ICFA Meeting)
CERN/IT to Assist with Web Form; Online SubmissionCERN/IT to Assist with Web Form; Online Submission Plan on Project to Build HENP World Network Map;Plan on Project to Build HENP World Network Map;
Updated and Maintained on a Web Site, Backed by Database: Updated and Maintained on a Web Site, Backed by Database: Systematize and Track Needs and StatusSystematize and Track Needs and Status Information: Link Bandwidths, Utilization, Quality, Pricing, Information: Link Bandwidths, Utilization, Quality, Pricing,
Local Infrastructure, Last Mile Problems, Vendors, etc. Local Infrastructure, Last Mile Problems, Vendors, etc. Identify Urgent Cases; Focus on Opportunities to HelpIdentify Urgent Cases; Focus on Opportunities to Help
First ICFA SCIC Workshop: Focus on the Digital DivideFirst ICFA SCIC Workshop: Focus on the Digital Divide Target Date February 2004 in Rio de Janeiro (LISHEP)Target Date February 2004 in Rio de Janeiro (LISHEP) Organization Meeting July 2003Organization Meeting July 2003 Plan Statement at the WSIS, Geneva (December 2003)Plan Statement at the WSIS, Geneva (December 2003) Install and Leave Behind a Good NetworkInstall and Leave Behind a Good Network Then 1 (to 2) Workshops Per Year, at Sites that Need HelpThen 1 (to 2) Workshops Per Year, at Sites that Need Help
Digital Divide Sub-Committee: Questionnaire Response Extract:
Institution Computing / networking needs related to HEP Other
UERJ, Brazil HEPGRID PROJECT presented for financial support to work on CMS; T3 then T1. Last Mile Problem: Need to reach RNP with good routing from UERJ
Wait for approved funds to build a T3; In 2005/6 a T1
Cinvestav, Mexico
Dedicated 2 Mbps link for HEP group Needed (now 2 X 2 Mbps for Whole Community; Not Enough)
UNLP, Argentina
a) LAN upgrade to 100 Mbps; b) LAN-to-WAN upgrade to 4 Mbps
JSI, Slovenia Additional bandwidth needed for HEP; now 2 Mbps for > 1000 people
QAU, Pakistan In terms of bandwidth need to upgrade but no last mile connection problem. High Prices
Also plans for T2 at NUST
TIFR, India Will have a Tier3 Grid node; Need network bandwidth upgrades at reasonable price (Telecom monopoly)
IFT-UNESP Brazil
Will maintain a farm for Monte Carlo studies and a Tier 3 Grid node; need more bandwidth
University of CYPRUS
HEP group intends to build a type T2 or T3 Grid node, and contribute to MC Production. Need to upgrade Network to Gigabit/sec. In principle there is no access limit the Network now. But daily traffic load is the real limitation.
34 Mbps Link to GEANT.University pays for Network
Groningen Carrier Hotel: March 2002
“Cultivate and promote practical
solutions to delivering scalable, universally available and equitable access
to suitable bandwidth and
necessary network resources in support
of research and education
collaborations.”
http://www.ieeaf.org
NY-AMS Done 9/02
CA-Tokyo by ~1/03
(Research)
Tyco Transpacific Donation
Plan: Santa Clara-JP-HK-TW-CN-SG
CN
SG
PERTH
GHANA
Buenos Aires/San
Paolo
St. Petersburg
Kazakhstan Uzbekistan
ChenaiNavi
Mumbai
Barcelona GreeceMD
NL
CA
Global Quilt Initiative – GMRE Initiative - 001
Global Medical Research Exchange Initiative
Bio-Medicine and Health Sciences
Layer 1 – Spoke & Hub Sites
Layer 2 – Spoke & Hub Sites
Layer 3 – Spoke & Hub Sites
Proposed Global Research and Education Network for Physics
Networks, Grids and HENPNetworks, Grids and HENP Next generation 10 Gbps network backbones are Next generation 10 Gbps network backbones are
almost here: in the US, Europe and Japanalmost here: in the US, Europe and Japan First stages arriving, starting nowFirst stages arriving, starting now
Major transoceanic links at 2.5 - 10 Gbps in 2002-3Major transoceanic links at 2.5 - 10 Gbps in 2002-3 Getting high (reliable; Grid) application performance Getting high (reliable; Grid) application performance
across networks means!across networks means! End-to-end monitoring; a coherent approach End-to-end monitoring; a coherent approach Getting high performance (TCP) toolkits in users’ handsGetting high performance (TCP) toolkits in users’ hands Working in concert with AMPATH, Internet2, Terena; Working in concert with AMPATH, Internet2, Terena;
DataTAG, the Grid projects and the Global Grid Forum DataTAG, the Grid projects and the Global Grid Forum Network improvements are especially needed in Network improvements are especially needed in
SE Europe, So. America; SE Asia, and Africa:SE Europe, So. America; SE Asia, and Africa: Key Examples: India, Pakistan, China; Brazil; Romania Key Examples: India, Pakistan, China; Brazil; Romania
Removing regional, Last mile bottlenecks and compromises Removing regional, Last mile bottlenecks and compromises in network quality are now (in all world regions)in network quality are now (in all world regions) On the critical pathOn the critical path
Networking for HENPNetworking for HENPClosing the Digital DivideClosing the Digital Divide
What ICFA, HEPCCC and the HENP Community Can DoWhat ICFA, HEPCCC and the HENP Community Can Do Spread the message: ICFA SCIC is there to helpSpread the message: ICFA SCIC is there to help Help identify and highlight specific needs (to Work On)Help identify and highlight specific needs (to Work On)
Policy problems; Last Mile problems; etc.Policy problems; Last Mile problems; etc. Encourage Joint programs [such as in DESY’s Silk project; Encourage Joint programs [such as in DESY’s Silk project;
Japanese links to SE Asia and China; AMPATH to So. America] Japanese links to SE Asia and China; AMPATH to So. America] NSF & LIS Proposals: US and EU to South AmericaNSF & LIS Proposals: US and EU to South America
Make direct contacts, arrange discussions with gov’t officialsMake direct contacts, arrange discussions with gov’t officials ICFA SCIC is prepared to participate ICFA SCIC is prepared to participate
Help Start, or Get Support for Workshops on Networks (& Grids) Help Start, or Get Support for Workshops on Networks (& Grids) Discuss & Create opportunities Discuss & Create opportunities Encourage, help form funded programs Encourage, help form funded programs
Help form Regional support & training groups (requires funding)Help form Regional support & training groups (requires funding)
LHCC 18 October 2002
Lyon biology grid project1)IN2P3 was willing to open partially the
services of its computing center to the french BioMedical research community.
Setting-up of collaborations with biology labs, in the Lyon area first.
2)DATAGRID and its WP10 has been an important opportunity to foster links with the Bio community. Submit production jobs or web requests thru grid tools is one goal.
3) But the opening is going beyond the sole GRID aspects.
http://t
ilde-djaco
bs.home.ce
rn.ch/~Djaco
bs/Hepcc
cw3/atta
chments/
021018_linglin
_biogrid-v2
.ppt
LHCC 18 October 2002
The collaboration with biologists
Real collaborations and grid effort have started only when we opened one post to deal with the Bio needs = May 2002.
In addition to EDG WP10, Collaborations with 3 labs (CNRS-Univ.) have started:
- IBCP (protein bio-chemistry)
- BBE (Biometry & evolutive Biology)
- ERIC (Mammography, breast cancer)
LHCC 18 October 2002
Bio needs : CPU, storage, gridification
CPU : - Protein sequence comparison algorithms- philogenic tree calculations- Metadata generation for medical imaging- STORAGE :- Reference data bases for proteins and genomics- Medical images for diagnostic assistance- Metadata for client radio-medical images- GRID ?- Only a subset deserves gridification.
LHCC 18 October 2002
Dealing with biologists ? strategy Culture in Bio research is different : there are no such
collaborations as we are used to in physicsFor computing, common thinking is still one PC and all the services + software on this single PC.This is the main problem : eg. genomic software and its environment (tools, libraries, OS) need to be tested & disseminated before running on several sites as it is common in HEP eg. to perform LHC Data Challenges
Going from a software running locally with plenty of "do-it yourself" fixes to a "full-proof, ready to install everywhere" product is not an easy task.
Two problems to be solved : cultural and technical. The cultural barrier is no more a problem with our
collaborators, but structuration, organisation, support is still lacking. We can push but not replace.
LHCC 18 October 2002
Strategy = cultural + technical =
Ease access to such a computing service center to which they are not used to.
Analyse their needs and propose suited solutions.
For a selected subset, set up a grid access.
LHCC 18 October 2002
Bio-med regional grid Select 1 "grid-aware" Bio lab and 2 HEP labs, plus
one application running as a web request = protein structure comparisons
Web-interfaced grid access (portal in Bio lab) Application, libraries and DataBases need to be
installed and updated regularly Job submission / web requests to 3 clusters (CC-
IN2P3, IBCP and LPC Clermont) from this portal User-friendly = a simple web request Use datagrid middleware = the resource broker of
EDG +etc.. Beyond this test, will connect other bio applications
to the grid.
LHCC 18 October 2002
IBCPportal
ResourceResourceBroker CCBroker CC
Web request
Compute+serviceCompute+serviceCluster CC-IN2P3Cluster CC-IN2P3
Lyon Biogrid test
CC-IN2P3 ready
Compute+serviceCompute+serviceCluster LPC Cluster LPC
Clermont-FerrandClermont-Ferrand
Compute+serviceCompute+serviceCluster IBCP LyonCluster IBCP Lyon
Result
On each site :
- updated dayly database
- job submission system
- visible from outside
LHCC 18 October 2002
Situation CC-IN2P3 : tools are ready since september LPC Clermont-F. : need to open their farm and
set-up a job submission tool. IBCP : idem
+ dayly DB export + portal modif. (checked it was not a major change
Expected a first test in early october, but we are still waiting for the other 2 labs to be ready. We (can) help, but we are helpless on several IBCP tasks. Still in progress.
LHCC 18 October 2002
Medium-term strategy
We have set-up good relations with a few local labs,well-known. Being neighbours is important. Tighter links with one (also active in WP10) : chosen for the grid production test.
These labs learn how to use a computing center, to express their needs, with figures, prepare a computing model. All this is new.
Will serve as propagandists for the others Lyon area (including Clermont, with IN2P3 as
one driving force) recognized as important for BioMed grids.
LHCC 18 October 2002
Conclusion Collaborations with biologists go beyond the
grid : try to disseminate our computing culture, of which grid is an important part.
An individual evaluation is needed to adapt the Bio needs to CC architecture
Hope to get first tests on 3-sites bio-grid very soon, but as everywhere, speed is driven by the slowest partner.
One related success to be quoted : the first open healthgrid workshop to prepare 6th R&D european program will be held in Lyon, 16-17 january 2003. IN2P3 co-organiser and in scientific comm.
iHEPCCC DevelopmentsiHEPCCC Developments
HEPCCC Working Group formed during 10/18/03 HEPCCCHEPCCC Working Group formed during 10/18/03 HEPCCC– Guy Wormser (Chair), Song, Kawabata (ACFA), Avery, Price (HEPAP), Guy Wormser (Chair), Song, Kawabata (ACFA), Avery, Price (HEPAP),
Barreira (ECFA), Haas (HEPCCC/HTASC)Barreira (ECFA), Haas (HEPCCC/HTASC)
Several meetings (phone, video, in person) over the last few months Several meetings (phone, video, in person) over the last few months (last one at SLAC in January)(last one at SLAC in January)
Draft charter presented to ICFA at the Tokyo meeting in FebuaryDraft charter presented to ICFA at the Tokyo meeting in Febuary
G. Wormser writes:G. Wormser writes:– good overall positive reaction,good overall positive reaction,
– need to revise the membership rules,need to revise the membership rules,
– suggestion to create a bureau that could effectively meet in person three suggestion to create a bureau that could effectively meet in person three tiles a year while the general assembly could take place once a year in tiles a year while the general assembly could take place once a year in person and by video/phone otherwise.person and by video/phone otherwise.
Final Suggestion on charter to HEPCCC at the 4 April meeting.Final Suggestion on charter to HEPCCC at the 4 April meeting.