data size and io performance report
DESCRIPTION
Data size and IO performance report. ATLAS Software & Computing Workshop. Data size MC http://athena-infoioperformance.web.cern.ch/athena-infoIOperformance/MCsizes.html Streams http://athena-infoioperformance.web.cern.ch/athena-infoIOperformance/StreamsSizes.html - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Data size and IO performance report](https://reader036.vdocuments.mx/reader036/viewer/2022062721/568137f3550346895d9fb425/html5/thumbnails/1.jpg)
Ilija Vukotic [email protected]
Data size and Data size and IO performance reportIO performance report
ATLAS Software & Computing Workshop
![Page 2: Data size and IO performance report](https://reader036.vdocuments.mx/reader036/viewer/2022062721/568137f3550346895d9fb425/html5/thumbnails/2.jpg)
• Data size
– MC http://athena-infoioperformance.web.cern.ch/athena-infoIOperformance/MCsizes.html
– Streams http://athena-infoioperformance.web.cern.ch/athena-infoIOperformance/StreamsSizes.html
– Categories http://athena-infoioperformance.web.cern.ch/athena-infoIOperformance/T0streams.html
• IO performance
• Possibilities for improvement
20/04/23 Ilija Vukotic 2
![Page 3: Data size and IO performance report](https://reader036.vdocuments.mx/reader036/viewer/2022062721/568137f3550346895d9fb425/html5/thumbnails/3.jpg)
MC event size
20/04/23 Ilija Vukotic 3
![Page 4: Data size and IO performance report](https://reader036.vdocuments.mx/reader036/viewer/2022062721/568137f3550346895d9fb425/html5/thumbnails/4.jpg)
MC event size
20/04/23 Ilija Vukotic 4
Note: trigger size is not realistic as used trigger menu was not realistic. Problem rectified with addition of new aliases:Physics_default -> Physics_pp_v1MC_loose_default -> MC_pp_v1_loose_mc_prescaleMC_tight_default -> MC_pp_v1_tight_Mc_prescale (see bug report #74712).
Size AOD Size ESD
RecoTrf 572 +27.17%* 1880 +33.09%
stdSim 381 +11.18% 1620 +24.03%
fullSim 381 +11.18% 1620 +24.03%
physSim 320 +11.96% 1520 +24.79%
*Compared to 22 Aug. 2010
![Page 5: Data size and IO performance report](https://reader036.vdocuments.mx/reader036/viewer/2022062721/568137f3550346895d9fb425/html5/thumbnails/5.jpg)
Streams sizes
20/04/23 Ilija Vukotic 5
Only runs with 5k+ eventsand good LBs
Shows:•AODs,ESDs and all DESDs•Events/total size/ev. size
![Page 6: Data size and IO performance report](https://reader036.vdocuments.mx/reader036/viewer/2022062721/568137f3550346895d9fb425/html5/thumbnails/6.jpg)
Streams sizes
20/04/23 Ilija Vukotic 6
Stream Size[GB]
ESD 1023834
AOD 92319
DESDM_EGAMMA 55288
DESD_SGLEL 44191
DESD_SGLMU 39375
DESDM_TRACK 26548
DESD_MBIAS 16841
DESD_CALJET 12370
DESDM_MET 5108
DESD_PHOJET 4281
DESD_SGLMUDESD_SGLEL
DESDM_EGAMMAAODESD
![Page 7: Data size and IO performance report](https://reader036.vdocuments.mx/reader036/viewer/2022062721/568137f3550346895d9fb425/html5/thumbnails/7.jpg)
20/04/23 Ilija Vukotic 7
Categories
![Page 8: Data size and IO performance report](https://reader036.vdocuments.mx/reader036/viewer/2022062721/568137f3550346895d9fb425/html5/thumbnails/8.jpg)
Categories
20/04/23 Ilija Vukotic 8
EgammaL1CaloMuonsJetTauETMiss
https://spreadsheets.google.com/ccc?key=0AiPvgbRljNCodG5IY1BseHRHa2R5WjZoMGRnLXp0dHc&hl=en
Details and list of collections in categories can be found at:
![Page 9: Data size and IO performance report](https://reader036.vdocuments.mx/reader036/viewer/2022062721/568137f3550346895d9fb425/html5/thumbnails/9.jpg)
20/04/23 Ilija Vukotic 9
IO performance
Slow
Bad CF
KEEP IN MINDIn real data TrackCollections are even larger !
![Page 10: Data size and IO performance report](https://reader036.vdocuments.mx/reader036/viewer/2022062721/568137f3550346895d9fb425/html5/thumbnails/10.jpg)
20/04/23 Ilija Vukotic 10
IO performance
• Do we really need all of these?
• No way to say if object is ever used.
• What is so terrible about having a name attached to each object we store?
![Page 11: Data size and IO performance report](https://reader036.vdocuments.mx/reader036/viewer/2022062721/568137f3550346895d9fb425/html5/thumbnails/11.jpg)
Possibilities for improvement
• TrackCollection/TrackParticle container
– Maybe 50kb in size and factor 2 in time
– Few MB in memory
• CaloShowerContainer
– Probably can go down in size
• Calo(Topo)TowerContainer
– Gets recreated at read time
– Must be possible to optimize further
• TRT_Drift_Circle_Container
– Must be faster and smaller
20/04/23 Ilija Vukotic 11
My estimate:
In total between 50 and 200kB/ev can be shaved in converters
Should be possible >100ms/ev (currently at 480 ms/ev)
Other options:
Object removal.
Cuts tuning