grid computing @ bifi1 grid computing at bifi: description of resources & examples of use...
Post on 21-Dec-2015
217 views
TRANSCRIPT
Grid Computing @ BIFI 1
Grid computing at BIFI:Description of resources &
examples of use
seminar 8th June 2005Guillermo Losilla Anadón
Grid Computing @ BIFI 2
BIFI: grid-related projects
• EGEE ("Enabling Grids for E-sciencE")– European grid for scientific use open 24
hours-a-day
• IrisGrid– National initiative
• I4Grid
Grid Computing @ BIFI 3
EGEE: predecessors
• DataGrid– First European initiative to create a pan-European
Grid
• LCG (LHC Computing Grid)– Built to deal with the huge
computing needs required to process the data that will be produced by the LHC
taken from GridCafé ®
Grid Computing @ BIFI 4
EGEE: project structure
• Extremely hierarchized:– Three main areas organised into 11 activities:
• NA: Networking Activities (5 acts)• SA: Specific Service Activities (2 acts)• JRA: Joint Research Activities (4 acts)
– 11 partner federations
• BIFI SA1 SWE federation(Spain & Portugal)
Grid Computing @ BIFI 5
EGEE: current status
• Currently using LCG grid middleware
• EGEE grid midleware (gLite) still in development
• Some figures:
taken from GSTAT (http://goc.grid.sinica.edu.tw/gstat/) 19:01h 7th June 2005
EGEE is currently the most powerful Grid infrastructure but...
...it is not enough for LHC!!!
Grid Computing @ BIFI 6
EGEE/LCG: architecture & services
• WMS (Workload Management System)– Manages the whole process of submitting a job to the
Grid– Uses JDL (Job Description Language) for describing
jobs
• DMS (Data Management System)– Manages the storing of data in the Grid– Identification(GUID, LFN, SURL) & copy of files– RLS(Replica Location Service) & LFC(LCG File
Catalog)
Grid Computing @ BIFI 7
EGEE/LCG: architecture & services
• IS (Information System)– Provides information about the Grid resources
and their status– Compliant with the GLUE schema– MDS (Monitoring & Discovery System) vs
RGMA (Relational Grid Monitoring Architecture)
– Source of information for Monitoring & Accounting services
Grid Computing @ BIFI 8
EGEE/LCG: node types
• UI (User Interface): point of entrance to the Grid• CE (Computing Element): access point to a farm of
computing nodes• WN (Worker node): computing • SE (Storage Element): access to storage element (disk,
tape...)• RB (Resource Broker): allocation and monitorization of
jobs• MON-BOX: Information System (RGMA-based)• BDII (Berkeley DB Information Index): Information
System cache• PS (Proxy Server): proxy certificates renewals
Grid Computing @ BIFI 9
EGEE/LCG resources at BIFI
• 1 UI: ui-egee.bifi.unizar.es
• 1 CE: ce-egee.bifi.unizar.es
• 1 SE + 1 Mon-Box: se-egee.bifi.unizar.es
• 1 WN
Using RB, PS & BDII from other sites(PIC & IFAE)
Grid Computing @ BIFI 10
EGEE: security & authorization
• Most services are built on top of GSI (Grid Security Infrastructure)
Every user must own, not only an user account in a UI node, but a valid digital certificate (issued by a suported CA) to be able to use the Grid
• Authorization enabled by Virtual Organization Organizations
Every user must join a virtual organization (CMS, Atlas, Alice, LHCb, SWETEST...)
Grid Computing @ BIFI 11
Examples of use: Acess to a UI
guillermo@lxbifi08:~> ssh -X [email protected]
[guillermo@ui-egee guillermo]$ grid-proxy-initYour identity: /C=ES/O=DATAGRID-ES/O=BIFI/CN=Guillermo
Losilla AnadonEnter GRID pass phrase for this identity:Creating proxy .......................................................... DoneYour proxy is valid until: Wed Jun 8 09:39:11 2005
Grid Computing @ BIFI 12
Examples of use: job submission
[guillermo@ui-egee guillermo]$ globus-job-run ce-egee.bifi.unizar.es /bin/pwd
/home/swetest011
Simplest job submission:Simplest job submission:
Complex job SubmissionComplex job Submission
[guillermo@ui-egee tests]$ more testJob.jdl
Executable = "testJob.sh";
StdOutput = "testJob.out";
StdError = "testJob.err";
InputSandbox = {"./testJob.sh"};
OutputSandbox = {"testJob.out","testJob.err"};
Grid Computing @ BIFI 13
Examples of use: job submission[guillermo@ui-egee tests]$ edg-job-list-match --vo swetest testJob.jdl
Selected Virtual Organisation name (from --vo option): swetestConnecting to host lcgrb02.ifae.es, port 7772
*************************************************************************** COMPUTING ELEMENT IDs LIST The following CE(s) matching your job requirements have been found:
*CEId* ce00.inta.es:2119/jobmanager-lcgpbs-swetest ce01.lip.pt:2119/jobmanager-lcgpbs-swetest ce2.egee.cesga.es:2119/jobmanager-lcgpbs-swetest lcg-ce.usc.cesga.es:2119/jobmanager-lcgpbs-swetest lcg2ce.ific.uv.es:2119/jobmanager-lcgpbs-short ramses.dsic.upv.es:2119/jobmanager-pbs-swetest lcg2ce.ific.uv.es:2119/jobmanager-lcgpbs-swetest lcg2ce.ific.uv.es:2119/jobmanager-lcgpbs-swetestL***************************************************************************
Grid Computing @ BIFI 14
Examples of use: job submission[guillermo@ui-egee tests]$ edg-job-submit --vo swetest testJob.jdl
Selected Virtual Organisation name (from --vo option): swetestConnecting to host lcgrb02.ifae.es, port 7772Logging to host lcgrb02.ifae.es, port 9002
********************************************************************************************* JOB SUBMIT OUTCOME The job has been successfully submitted to the Network Server. Use edg-job-status command to check job current status. Your job identifier (edg_jobId) is:
- https://lcgrb02.ifae.es:9000/6scdvJ4Y1QU8yeMjH1zBhw*********************************************************************************************[guillermo@ui-egee tests]$ edg-job-status https://lcgrb02.ifae.es:9000/6scdvJ4Y1QU8yeMjH1zBhw
*************************************************************BOOKKEEPING INFORMATION:
Status info for the Job : https://lcgrb02.ifae.es:9000/6scdvJ4Y1QU8yeMjH1zBhwCurrent Status: ScheduledStatus Reason: Job successfully submitted to GlobusDestination: lcg2ce.ific.uv.es:2119/jobmanager-lcgpbs-shortreached on: Tue Jun 7 19:56:35 2005*************************************************************
Grid Computing @ BIFI 15
Examples of use: job submission[guillermo@ui-egee tests]$ edg-job-get-output https://lcgrb02.ifae.es:9000/6scdvJ4Y1QU8yeMjH1zBhw
Retrieving files from host: lcgrb02.ifae.es ( for https://lcgrb02.ifae.es:9000/6scdvJ4Y1QU8yeMjH1zBhw )
********************************************************************************* JOB GET OUTPUT OUTCOME
Output sandbox files for the job: - https://lcgrb02.ifae.es:9000/6scdvJ4Y1QU8yeMjH1zBhw have been successfully retrieved and stored in the directory: /tmp/jobOutput/guillermo_6scdvJ4Y1QU8yeMjH1zBhw
*********************************************************************************
[guillermo@ui-egee tests]$ ll /tmp/jobOutput/guillermo_6scdvJ4Y1QU8yeMjH1zBhw/total 4-rw-rw-r-- 1 guillermo guillermo 0 jun 7 22:02 testJob.err-rw-rw-r-- 1 guillermo guillermo 168 jun 7 22:02 testJob.out
[guillermo@ui-egee tests]$ more /tmp/jobOutput/guillermo_6scdvJ4Y1QU8yeMjH1zBhw/testJob.outTue Jun 7 21:58:22 CEST 2005wn066.ific.uv.es/home/swetest002/globustmp.wn066.23526.0/
WMS_wn066_023996_https_3a_2f_2flcgrb02.ifae.es_3a9000_2f6scdvJ4Y1QU8yeMjH1zBhw
Grid Computing @ BIFI 16
Examples of use: data management
[guillermo@ui-egee tests]$ edg-gridftp-ls gsiftp://se-egee.bifi.unizar.es/storage
swetestaliceatlascmsdteamlhcbsixt
[guillermo@ui-egee tests]$ edg-gridftp-mkdir gsiftp://se-egee.bifi.unizar.es/storage/swetest/prueba
[guillermo@ui-egee tests]$ edg-gridftp-ls gsiftp://se-egee.bifi.unizar.es/storage/swetest/
prueba
Grid Computing @ BIFI 17
Examples of use: data management
$ lcg-cr -l mi_fichero_en_el_Grid-d sfn://se-egee.bifi.unizar.es/storage/swetest/test/mi_test --vo swetest file://`pwd`/mi_test
Guid: guid:5def36aa-324e-4206-a631-a56093b8b02b
$ lcg-lr --vo swetest guid:5def36aa-324e-4206-a631-a56093b8b02bLocation: sfn://se-egee.bifi.unizar.es/storage/swetest/test/mi_test
$ lcg-lr --vo swetest mi_fichero_en_el_GridLocation: sfn://se-egee.bifi.unizar.es/storage/swetest/test/mi_test
$ lcg-cp --vo swetest guid:5def36aa-324e-4206-a631-a56093b8b02b file://`pwd`/mi_fichero_recuperado
Grid Computing @ BIFI 18
Examples of use: Information System
[guillermo@ui-egee tests]$ lcg-infosites --vo swetest se
**************************************************************These are the related data for swetest: (in terms of SE)**************************************************************
Avail Space(Kb) Used Space(Kb) Type SEs----------------------------------------------------------51354616 21660820 disk se01.lip.pt11781208 134239228 disk se00.inta.es17971988 10066312 disk aker.dsic.upv.es64635064 2473188 disk lcg-se.usc.cesga.es5503300 2331648 disk se2.egee.cesga.es1000000000000 500000000000 mss castorgrid.ific.uv.es0 500000000000 mss castorsrm.ifae.es1000000000000 500000000000 mss castorsrm.ific.uv.es
[guillermo@ui-egee tests]$ ldapsearch -LLL -x -H ldap://ce-egee.bifi.unizar.es:2170 -b "mds-vo-name=bifi,o=grid"
Grid Computing @ BIFI 19
Useful links
• Grid computing at BIFI: http://bifi.unizar.es/research/computing/grid/gridcomputing.html
• EGEE project: http://public.eu-egee.org/
• LCG project: http://lcg.web.cern.ch/LCG/
• LCG-2 User Guide: https://edms.cern.ch/file/454439//LCG-2-UserGuide.html