National Center for Supercomputing ApplicationsUniversity of Illinois at Urbana-Champaign

The Dark Energy Survey Middleware

LSST Workflow Workshop 09/2010

Michelle Gower

NCSA

on behalf of the DES Data Management Team and DES Collaboration

DES Data Management

• Observing from 2011-2016:• 200 TB raw data, 4 PB of products• 100 TB object catalog• 120+ CPU years/year of processing

• Processing includes:• Removing the instrument signature• Removing artifacts (e.g. planes)• Calibrating/registering• Feature detection• Feature analysis• Monitoring/quality assessment

Requirements

• Level of parallelism changes through processing• Easily modified by scientists• Local and remote clusters• Work with project’s Archive system• Monitoring of jobs while they’re running• Less research/More production

Processing Framework Overview

4

ArchiveNodes

Target (HPC)Machines

NotificationService

Database

Pipelines of AppModules

EventMonitor

ArchivePortal

Orchestration

DAF

Middleware

• Workflow• Condor DAGman

• Job submission• Condor-G to pre-WS GRAM for TeraGrid resources• Condor (vanilla jobs) for local machines

• File transfer• GridFTP using clients uberftp and globus-url-copy

• Runtime Monitoring• Elf/Ogrescript

Workflow

Crosstalk CreateCor ImCorrect

Masking AstroRefine

Remap PSFModel WeakLensing

Make Bkgd

• Run queries to get master input file lists• Stage input images to target machine• Generate Jobs: input lists, job descriptions, DAG• Setup target machine• Stage generated lists and files to target machine• Make timestamp on target machine• Target jobs run• Ingestion of new or modified files

Job Generation

Upcoming Orchestration Work

• Skipping modules• Repeatedly using set of blocks (Campaign

processing)• Let condor manage vanilla jobs per machine

• threaded vs serial

• Finer grain restarts

Notification Events System

AppModule (Elf/OgreScript)

Science Code

ActiveMQ Message Bus

Event StorePortal Monitor

DES Monitor Portal

• Current Technology• Drupal CMS / PHP, Javascript/Ajax

• Tools for Viewing Active Processing Status • High level Alert Monitoring

• Quick navigation to event logs

• Quality Assurance Profiles

• Histograms of QA metrics, outlier detection

• Processing Timing Summary Report

• Middleware level timing profile

Acknowledgements

• DESDM Team:• Jim Myers, Terry McLaren• Joe Mohr• Bob Armstrong, Dora Cai, Ankit Chandra, Greg Daues,

Shantanu Desai, Michelle Gower, Wayne Hoyenga, Chit Khin, Kailash Kotwani

• Past DESDM Team Members, DES Project Team and Collaboration Members

• National Science Foundation

• http://cosmology.illinois.edu/DES/