From photons to catalogs

• Cosmological survey in visible/near IR light using 4 complementary techniques to characterize dark energy:

I. Cluster Counts

II. Weak Lensing

III. Large-scale Structure

IV. Supernovae• Two multiband (photometric) surveys:

5000 deg2 grizY to 24th mag AB griz

10-30 deg2 repeat (SNe)• Build new 3 deg2 FOV multi-CCD camera,

Data management system, improve Blanco facilities

Blanco 4-meter at CTIO

2/2911/08/2011 APS-DPF 2011 DESDM I.Sevilla

Credit:NOAO

11/08/2011 APS-DPF 2011 DESDM I.Sevilla 3/29

SDSS-II DES

Area 10000 5000

Nb. of CCDs 22 62

Resolution 120 Mpix 570 Mpix

Raw data/night 200 GB/night 300 GB/night

Catalog size 18 TB 100 TB (est.)

Total data volume 60 TB 4 PB (est.)

Credit: Kotwani et al. (2010)

Transfer

Process

Archive

Distribute

4/2911/08/2011 APS-DPF 2011 DESDM I.Sevilla

Transfer

CTIO NCSA 300 GB/night in 18 h

Process

Archive

Distribute

5/2911/08/2011 APS-DPF 2011 DESDM I.Sevilla

Transfer

Process

Orchestration:

NCSA HPC nodes

Archive

Distribute

6/2911/08/2011 APS-DPF 2011 DESDM I.Sevilla

Transfer

Process

Archive

Results Archive nodesOracle DB

Distribute

7/2911/08/2011 APS-DPF 2011 DESDM I.Sevilla

Transfer

Process

Archive

Distribute

Through web portals

8/2911/08/2011 APS-DPF 2011 DESDM I.Sevilla

11/08/2011 APS-DPF 2011 DESDM I.Sevilla 9/29

Exposure consists of 62+ CCD images – 570 Mpix - 3deg2

Survey is ~150,000 100-sec exposures over 525 nights

Auxiliary CCDs record images for autoguiding and calibration

11/08/2011 APS-DPF 2011 DESDM I.Sevilla 10/29

X 300 + calib. = RAW DATA

we send this to NCSA

Correct for cross-talk among CCDs.

Correct for bias levels, non-uniformities, other optical and electrical effects.

11/08/2011 APS-DPF 2011 DESDM I.Sevilla 11/29

DETRENDED DATA

Credit:NOAO

11/08/2011 APS-DPF 2011 DESDM I.Sevilla 12/29

We need reference star catalogs

Full focal plane is fit to single solution

Correct optical distortion at focal plane

Credit: E.Bertin

We use Sextractor, SCAMP software by E.Bertin

REDUCED DATA11/08/2011 APS-DPF 2011 DESDM I.Sevilla 13/29

Use standard star fields at different angles in the sky (X) with known fluxes (m_std and color_std) and relate with instrumental flux (m_inst):

Make big least squares solution for a,b,k; apply results to science images.

€

minst −mstd = a+ bCCDncolorstd + kXinputoutput

At this point we have, for every night, approx. 300 exposures corrected by:

• Instrumental effects

• Absolute position

• Absolute photometry

This is the nightly processing.

We store these in the archive (+ auxiliary images, info).

11/08/2011 APS-DPF 2011 DESDM I.Sevilla 14/29

11/08/2011 APS-DPF 2011 DESDM I.Sevilla 15/29

COADDED DATA

Go deeper; calibrate better

BUT

Point spread function is inhomogeneous: each exposure has different quality

single exposure

single exposure

single exposure

single exposure

single exposure

single exposure

single exposure

11/08/2011 APS-DPF 2011 DESDM I.Sevilla 16/29

COADDED DATA

Find PSF in each image

Homogeneize per coadd tile

PSF HOMOGENEIZED

0.77 ’’0.77 ’’

1.32 ’’1.32 ’’

0.94 ’’0.94 ’’

0.94 ’’0.94 ’’

It takes 30x more time for coaddition with respect to nightly:

• PSF has to be extracted• PSF has to be homogeneized• Actual addition of image and recomputation of errors

Additionally:

• Global photometric calibration among all images of the season

We store these in the archive (+ auxiliary images, info).

11/08/2011 APS-DPF 2011 DESDM I.Sevilla 17/29

11/08/2011 APS-DPF 2011 DESDM I.Sevilla 18/29

CATALOGS

E.Bertin

This step is performed with the SExtractor package

Position, shape and photometry is calculated.

11/08/2011 APS-DPF 2011 DESDM I.Sevilla 19/29

Hundreds of millions of objects with hundreds of columns each:

11/08/2011 APS-DPF 2011 DESDM I.Sevilla 20/29

Hundreds of millions of objects with hundreds of columns each:

11/08/2011 APS-DPF 2011 DESDM I.Sevilla 21/29

Hundreds of millions of objects with hundreds of columns each:

11/08/2011 APS-DPF 2011 DESDM I.Sevilla 22/29

Photometric redshift pipeline: take fluxes in five bands -> estimate redshift

Weak lensing pipeline: identify stars and construct PSF -> deconvolve from galaxies to obtain shear.

Difference imaging pipeline (SN): subtract images from different epochs to look for transient phenomena.

mag_band_g = 20.7mag_band_r = 19.2mag_band_i = 18.5… + errors, other estimates

(this only one kind of photo-z! More estimations foreseen)

DESDM Photoz pipeline(neural network)

11/08/2011 APS-DPF 2011 DESDM I.Sevilla 23/29

Photometric redshift pipeline: take fluxes in five bands -> estimate redshift

Weak lensing pipeline: identify stars and construct PSF -> deconvolve from galaxies to obtain shear.

Difference imaging pipeline (SN): subtract images from different epochs to look for transient phenomena.

Eliminate instrumentalsignature

Obtain true shape (intrinsic galaxy shape+shear)

DESDM

WL pipeline

local PSF

11/08/2011 APS-DPF 2011 DESDM I.Sevilla 24/29

Photometric redshift pipeline: take fluxes in five bands -> estimate redshift

Weak lensing pipeline: identify stars and construct PSF -> deconvolve from galaxies to obtain shear.

Difference imaging pipeline (SN): subtract images from different epochs to look for transient phenomena.

Credit: Pan-STARRS

11/08/2011 APS-DPF 2011 DESDM I.Sevilla

Produce cosmological simulations

Process through atmosphere, detectors, include ‘nasty’ stuff

25/29

11/08/2011 APS-DPF 2011 DESDM I.Sevilla 26/29

Galaxies

Stars

11/08/2011 APS-DPF 2011 DESDM I.Sevilla 27/29

BCS Images of First SPT Clusters

Credit: Blanco Cosmology Survey

Real data sets from the Blanco Cosmology Survey, SPT, SCS (Mosaic2 camera).

Large scale management with SDSS data.

11/08/2011 APS-DPF 2011 DESDM I.Sevilla

The Dark Energy Survey (next talks!) will make use of a large, scalable data management system to process and archive raw images and science ready data products.

Acceptance of the system is underway (results end of year). Tests on real DES data expected for first months of 2012.

Raw and reduced images to be released yearly, catalogs at midpoint and end of survey.

Community pipeline getting ready for usage of the DES camera starting May 2012.

28/29

11/08/2011 APS-DPF 2011 DESDM I.Sevilla

11/08/2011 APS-DPF 2011 DESDM I.Sevilla

Development Funding ~$6MDevelopment Funding ~$6M $4 million from NSF$4 million from NSF $1.78 million (in kind) from NCSA/U Illinois, Fermilab, IAP and $1.78 million (in kind) from NCSA/U Illinois, Fermilab, IAP and

MunichMunich $300K from DES collaboration for Community Pipeline$300K from DES collaboration for Community Pipeline

(new simulations with more DES-like systematics coming up by Stanford team)

Carnero et al. 2010

Perform full-depth observations in 100 sq.deg.

Area is off main survey

Overlap existing datasets when possible

Run acceptance tests on data