• ATLAS Metadata Interface.• Portal page. http://ami.in2p3.fr/• A generic cataloging framework – used

in ATLAS for dataset discovery

AMI

1

•Data in AMI is organized in CATALOGUES. Catalogues are assigned to a project and sub-project.

• mc08-Production (Holds 2008 MonteCarlo datasets)• data08-RealData (cosmics and first beam)

Dataset Discovery• Means finding the names of valid datasets to use in

your analysis.• An ATLAS (container) dataset is either

– A number of files output of DAQ, and managed by Tier 0. (RAW data and fast reconstruction)

– A number of files produced by one or more production system tasks. Each task makes a set of (TID) datasets. (MC and reprocessing.)

• The distributed data management (DDM) manages datasets as containers of task datasets.

• Physicists doing analysis normally should worry only about the containers, and let the infrastructure take care of the rest.

2

An introduction to datasets and their nomenclature

3

Project.datasetNumber.physicsShort.productionStep.dataType.version[/]

dataNN_* or mcNN_*

ESD, AOD

Concatanation of configurations

Project.runNumber.streamType.productionStep.dataType.version[/]

digit, recon, merge, etc…

mc09.106099.PhojetCdiff.evgen.EVNT.e456

4

5

Where does AMI get its data?• Real data : From the Tier 0 : DAQ data, and first

reconstruction is registered in AMI < 5 minutes after it is declared “ready” (both datasets and files). Config tags for real data.

• Monte Carlo and reprocessing.– From the Task Request DB : Tasks, dataset names, MC and

reprocessing config tags– From the production DB : Finished tasks – files and metadata.

• From physicists.– M.C. GEN datasets– MC Dataset number info, physics group owner,…– DPD tags– …

A precision about REAL DATA registration in AMI

data09_somethingStart of Run. Tier 0 declares empty datasets in DQ2Tier 0 adds files to the dataset as they arrive from SFO (LB by LB)

End of Run. Dataset closed in DQ2. AMI Flag raised. Arrives in AMI within 5 minutes.

Users may find data in DQ2 several hours before AMI sees the dataset.

7

• Overview page :https://ami.in2p3.fr/AMI/servlet/net.hep.atlas.Database.Bookkeeping.AMI.Servlet.Command?linkId=246– Notice that not all datasets are VALID.

• Invalid because all tasks failed, declared bad, or deleted.• Invalid datasets are hidden by default.

– Project tags are controlled, for official datasets; they follow the nomenclature specification and are declared by data preparation coordination. (data09_7TeV for example)

• If you know the project tag this is perhaps the simplest way to get to the list.

8

The “Simple” search• Simple Search – search on part of the

name.– Example “L1Calo “– Note that “%” is used for wild carding.

• Search on keywords, and comments – a search in many fields – takes longer.

• Tutorial link: http://ami.in2p3.fr:8080/opencms/opencms/AMI/www/Tutorial/Simple_search_interface.html

9

The “Advanced” search

• Examples to try:– Physics group = “phys-beauty”

• Notice that there are results in several catalogues.

• Tutorial link: http://ami.in2p3.fr:8080/opencms/opencms/AMI/www/Tutorial/Advanced_search_interface.html

• This interface will de redesigned in the next few months.

10

Looking at the results.• Use any search interface to get

“physics_MinBias ” data for 2009• Use the “group by” under data_type to find

the AOD datasets. (409 2010-02-08)• “Filter” on the dataset name, or the

runNumber to find 142383.• http://ami.in2p3.fr:8080/opencms/opencms/A

MI/www/Tutorial/Exploring_the_results.html• http://ami.in2p3.fr:8080/opencms/opencms/A

MI/www/Tutorial/Refine_the_Results.html

11

Looking at the results some more• Click on “additional fields” . Notice the prodsysStatus.

Choose one which has “EVENTS_AVAILABLE”. (more)

• Click on “details” – to get the “child” elements.• Click on “event_range” to get the prodsys task details.• Click on “files” to browse the files.• Notice the “history” buttons (“bread crumb trail”). You

can even remove them.• Click on “Provenance” of dataset. You can use this to

navigate along the production chain. For REAL data one gets back to the RAW type. (For MC to EVGEN and even GEN input. From EVGEN you can see the jobOptions.)

12

Getting to the data• Look for an mc09 dataset with

“EVENTS_AVAILABLE” mc09_900GeV.105004.pythia_ddiff.merge.AOD.e466_s667_s668_d258_r1026_

r1051 .• Click on “DQ2”. This page shows an

aggregation of information from DQ2. You can subscribe to the dataset from this page if you like, going through the DDM permission system of course.

• AMI will also make you “dq2-get” commands, but in general you should not do it.

13

Config Tags• Example: e466_s667_s668_d258_r1026_r1051 (last field of

dataset name)• e466 -> event generation parameters

s667, s668 -> simulation parameters (simul.HITS, merge.HITS)

d258 -> digitizationr1026,r1051 -> reconstruction parameters

• Interpretation of Config tagshttp://ami.in2p3.fr/opencms/opencms/AMI/www/ReferenceTables/

• Searching starting from the Config Tag.http://ami.in2p3.fr/opencms/opencms/AMI/www/Tutorial/ConfigTags

• Comparing tags (try r1026 and r1051) from the simple search page.

14

AMI Accounts

• Logging on to AMI.– In general you do not need to log on to read (at the

moment)– You can make an AMI account to access a

personal page.– You must log on for any writing operation.– Once you log on to AMI you can make bookmarks.– Tutorial link :

http://ami.in2p3.fr/opencms/opencms/AMI/www/Tutorial/Other_AMI_basic_functionalities.html

15

Other stuff.• pyAMI. Everything in AMI can be obtained

from the python client. http://ami.in2p3.fr/opencms/opencms/AMI/www/Client/pyAMISecure_and_cmt

• Ad Hoc queries. (For really advanced users!) http://ami.in2p3.fr/opencms/opencms/AMI/www/Tutorial/Refine_the_search.html