vstcen data management divisionnapolita/vstmeeting/vstcen-dmd_vst... · 2009. 8. 24. · quality...
TRANSCRIPT
VSTceN-DMD Napoli 20-07-2009
VSTceN
Data Management
Division
A. Grado
INAF
VSTceN-DMD Napoli 20-07-2009
VSTceN-DMDGoals
� Data processing hardware infrastructure� Data processing and data storage
� Data processing software infrastructure � Pipelines and data archive
� Coordination of data reduction � Users support
INAF
VSTceN-DMD Napoli 20-07-2009
VSTceN-DMD
WF related projects
Wide field HW/SW related projects since the VST start:
4 projects for SW development
4 projects for HW/GRID R&D/procurement
3 projects for SW development
INAF
VSTceN-DMD Napoli 20-07-2009
VSTceN-DMD
WF related projects INAF
1998 - OACDF (OAC � pilot project at [email protected] )1999 - WFIX (OAC-Terapix � tools used in Terapix and Bonn pipeline)2000 - BEOWFI (OAC-ESO � photometric calib pipeline of [email protected]
running on a beowulf cluster)2001 - COFIN Capaccioli (PartenoBEO � Beowulf procurement )2001 - AstroWise (OmegaCen NL, Terapix, ESO, USM, OAC � QC on
pre-reduction, variability tool)2001 - FIRB-GRID Wp10 (OAP, OAT, OAC � WF pre-reduction on grid)2002 - COFIN-GRID (OAT, OAP, OAC � tests on Beowulf in grid)2004 - COFIN Capaccioli (Data Storage procurement)2005 - AstromC (OAC � WFIX porting to C++, used in LBC pipeline)2005 - VSTarchive (VSTceN � archive SW for VST images)2006 - Efits (VSTceN-Sternberg Institute � C library for fits files) 2006 - VST-Tube (OAC-VSTceN � flexible pipeline for WF images )
VSTceN-DMD Napoli 20-07-2009
VSTceN-DMD
VST data flow/volumeVSTceN GoT: ~ 60 nights/yearKids: ~ 40 nights/year/node
Disk shipping trough Diplo-bag each 4 working days
Data transferred to Naples via network.
Each night: 50 ÷ 170 GB (16 bits not compressed, 2 bands/nights)
INAF
VSTceN-DMD Napoli 20-07-2009
VSTceN-DMD
Available HW
Image processing: 80 cores (2 concurrent users)
Data analysis and numerical simulations: 32 CPU
Data storage: 80 TB (very high quality) (enough for 2 years of VST operation)
New cluster (80 cores) and data storage extension (65 TB) installed on May 2009 with Regione Campania funds
VSTceN-DMD Napoli 20-07-2009
VSTceN HW infrastructuresEMC NS704 schematic view
The system is fully redundant (no single point of failure)
The system is remotely controlled by the EMC, intervention in 4 hours in case of failure.
Presently populated with 77 TB disks (raw capacity)
13 CPU just to control the disks!!
VSTceN-DMD Napoli 20-07-2009
VSTceN-DMD SW
VST data handling
needs implemented
Archive raw data and pipeline products AW, VSTarchive
Process images up to science grade level 2 pipelines (AW, VST-Tube)
Provide Quality Control AW, QualityFits
INAF
VSTceN-DMD Napoli 20-07-2009
VSTceN-DMD AstroWise
An FP5 RTD EC funded Project
AW goal: develop and maintain a computational environment to process wide-field imaging data
AW is: a pipeline embedded in a data base using an object oriented like approach
AW official release: November 2006 (see www.astro-wise.org)
VSTceN-DMD Napoli 20-07-2009
VSTceN-DMD AstroWise
The AstroWise idea
AstroWise has been designed to operate in a multi-node environment and to trace the reduction history of large and distributed projects (e.g. KIDS)
� Raw pixel data� Pipelines/cal files All integrated in one information � Catalogues system
This implies a complex DB administration and SW structure
VSTceN-DMD Napoli 20-07-2009
VSTceN-DMD AstroWise
The AstroWise features
� distributed services� multiple instrument support� processing GRID / Storage DISTRIBUTED� driven by DB query of user� publish directly in EURO-VO� easy to go from source in catalog back to pixel
VSTceN-DMD Napoli 20-07-2009
VSTceN-DMD AstroWise
The federate AstroWise environment
Official pipeline for KIDS
Napoli-node has contributed:
quality control pre-reduction
data analysis (variability tool)Scamp in AW (recently)
Planned an Astro-Wise course for the national community
VSTceN-DMD Napoli 20-07-2009
VSTceN-DMD SWVST-tube
Complementary pipeline to Astro-Wisedeveloped completely in Naples.
Why?
� Pipeline working without DB� Easy to adapt to peculiar data reduction
model� Easy for new algorithms testing � Data reduction cross check
Unique characteristics: e.g. true noise map propagation
VSTceN-DMD Napoli 20-07-2009
VSTceN-DMD SWVST-tube
It is a pipeline but also a collection of tools
� aperture and PSF (S. Zaggia) photometry � Illumination correction surface tool� MAgliM (limiting magnitude vs magnitude and surface brightness � Mask (automatic halo/spike mask)� VSTSim (images simulator)
VSTceN-DMD Napoli 20-07-2009
VSTceN-DMD SWVST-tube
Automatic spikes and halos detection (ZhuoyiHuang)
VSTceN-DMD Napoli 20-07-2009
VSTceN-DMD Processing capability3 tough examples - versatility
VST-Tube adapted for the reduction of CFHT12k images (VIRMOS collaboration)
4 deg2 distributed over 51 nights
(Lamareille et al. 2009 A&A 495,53)
Photometric comparison with SDSS as function of celestial coordinates Typical σ < 0.1 mag
∆mag
(sds
s–
us)
VSTceN-DMD Napoli 20-07-2009
VSTceN-DMD Processing capability 3 tough examples – astrometry
Globular cluster NGC2808 4 deg2 observed with [email protected] in V and I band
V-I
V
PSF photometry on coaddedmosaics for a total of 16 pointings:
Completely automatic reduction and catalog extraction in collaboration with S. Zaggia (INAF-OAPd).
S. Zaggia, A. Grado et al. , Tidal Tails in Globular Clusters.The case of NGC2808(A&A, in preparation)
color turn-off stars σ < ± 0.02 mag
VSTceN-DMD Napoli 20-07-2009
VSTceN-DMD Processing capability 3 tough examples - photometric accuracy
NGC3115 observed with [email protected]
V-I
V
Surface photometry of NGC3115
With plates (Capaccioli et al. 1987 ) exp.time = 34 Ks
With WFI + VST-Tube (Grado et al. in preparation)
exp.time = 4.2 Ks
1 2 3 4 5 6 7
18
20
22
24
26
28
30
32
(distance from center)1/4
Sur
face
brig
htne
ss (
mag
arc
sec−
2 )
(distance from center)1/4
Sur
face
brig
htne
ss (
mag
arcs
ec-2
)
VSTceN-DMD Napoli 20-07-2009
VST-DMDSW
Other SW products
- VSTceN data Archive
- AstromC : sw for astrometric calibration (used also for LBC pipeline)
- EFits : new Fits library with noise propagation map A. Grado, M. Pavlov and L. Limatola, 2007, "Efits: A New Efficient and Flexible FITS Library ", The 2007 ESO Instrument Calibration Workshop. Ed. Springer, p. 217
- Fitshdr :Fits header manipulation- Astrobench : hw benchmark for astronomical applications
VSTceN-DMD Napoli 20-07-2009
Example of Efits weight map
NGC4697
mosaic obtained with [email protected] data
Section of the weight map
VSTceN-DMD Napoli 20-07-2009
VSTceN-DMD Cost breakdown
Costs since VSTceN start activities
HW Total cost 222.725 €
Cofin Capaccioli
Regione Campania
INAF
AW
PremioFondazioneCorteseBanco Napoli
SW total cost164.450 €
Cofin Capaccioli
Regione Campania
INAF
AW
Premio FondazioneCorteseBanco Napoli
VSTceN-DMD Napoli 20-07-2009
VSTceN-DMD
Conclusions
• The hardware is in place and working
• AW is available (federated DB under set-up)
• The VST-archive is ready
• VST-Tube is currently used for science projects
• VST-Tube 2.0 with Efits under test
INAF
VSTceN-DMD Napoli 20-07-2009
VSTceN-DMDInvolved people Activities
2 - VST-Tube
6 - Efits
1 - Astro-Wise
4 - Hardware
* Sternberg Institute Moscow
** now at OAT
3 - DB admin
Bold-underlined-blue are not staff
8 - VSTarchive
A. Grado1,2,4,6,8
F. Getman1,3.4
L. Limatola2,6,8
M. Radovich1,5,7
M. Pavlov*6
E. Puddu1
R. Silvotti**1
J. Alcala’1
S. Zaggia2
5 - AstromC
7 - Photcal