australian virtual observatory – 2003 workshop atnf, marsfield, 17-18 nov 2003 gemini and the...

Australian Virtual Observatory – 2003 Workshop

ATNF, Marsfield, 17-18 Nov 2003

Gemini and the Virtual Observatory

Warrick Couch, UNSW(aided and abetted by Colin

Aspin, Gemini Obs)



The Gemini Observatory

• twin 8.2m telescopes, 4-5 instruments per telescope, many different modes per instrument.

Visible Near-IR Mid-IR

Gemini North

GMOS – imager/spec

(l/slit, MOS, IFU)

NIRI – imager/spec

ALTAIR – AO feed

Michelle – imager/spec

Gemini South

GMOS – imager/spec

(l/slit, MOS, IFU)

GNIRS – l/slit spec Phoenix – HiRes spec

T-ReCS – imager/spec

Gemini North – Mauna Kea Gemini South – Cerro Pachon



The Gemini Observatory

• 6 partner countries, 1 host country (Chile), 1 host institution (UHawaii) broad and diverse set of scientific users & programs.

• programs (currently) involve ‘PI’ rather than ‘survey’ science.

• majority (80-90%) of observations done in ‘queue’ mode science data must come with good calibration & site weather data… but greatly facilitates a good archive!

• standard proprietary period = 18 months• all new instruments undergo ‘science verification’ data

available in 2-3 months!

Gemini North – Mauna Kea Gemini South – Cerro Pachon



• Apr 1998 : GSC/Board endorse GSA resolution.• Sep 1998 : Gemini sponsored GSA workshop and budget line

approved by Board/Director.• Nov 2000 : Functional and Performance Requirements Document and

Operational Concept Definition Document issued by Gemini.• Sep 2001 : Colin Aspin became GSA Project Scientist. • Nov 2001 : Phase II Operations and Development Plan review at CADC.• May 2002 : Concept Design Document issued by CADC.• Jan 2003 : Content Overview Document and Meta-Data DataBase Definition

Document issued by Gemini. • Feb 2003 : Phase II ODP contract signed!

The road to a Gemini Science Archive (GSA):

10 November 2003: the Gemini Science Archive prototype (GASp) goes public!!



The Basic Archive/Prototype(http://cadc.hia.arc.ca/gemini)

• Basic retrieval of raw science & calibration data to (registered) users after proprietary period.

• Web-based search engine at CADC with numerous options for retrieval. Same functionality as CFHT, JCMT, etc archives.

• All ingested data shown in ‘results page’ but only non-proprietary data available for download.

• FITS headers: only meta-data in prototype.• Data retrieved via e-transfer on extraction and

packaging. • Operational: data ingest of GN (facility instrument)

data up to date, of GS data (visitor instruments) – 90% complete. GMOS-S ingestion under way.



Transferal of Data to Archive

• Current Status : data from GN and GS are packaged on DVD-R & shipped each 2-4 weeks.

• Data Ingestion is therefore delayed by packaging and mailing of hard media.

• Plan is to setup e-transfer of data from GN and GS to CADC. VPN connection to be used for security. Staging areas at GN and GS will be monitored by CADC and when new data is found, e-transfer, validation and ingestion takes place (by end 2003).

• Current I2 network gives ~150KB/sec/connection. Tested up to 6 simultaneous connections -> 900KB/sec. (typical GMOS image is ~50MB).



Data Distribution to PIs

• Current Status : data from GN and GS are packaged on CD-R or DAT and shipped directly to PI.

• Plan is to have PI obtain data from Queue or Classical run via CADC GSA interface via e-transfer.

• If hard-media is required by PI then CADC would make media from archive data and ship.

• Requires special/secure access for PIs to proprietary data (their own).

• Requires access to image data, calibration data and meta-data.

• Timescale : mid-2004.



The fully capable GSA:• Addition of complete meta-data access using

MDDB. This includes ‘results page’ display of meta-data.

• Ingestion of results of OLDP system as on-line preview.

• Associations of datasets (e.g. J,H,K)• Associations of calibrations datasets.• Improved user-interface(s).• Batch user-interface (e-mail).• Network data transfer to CADC : PI distribution from

CADC.



The Meta-Data Data Base (MDDB)

• MDDB resides at CADC.• Ingest tools (provided by CADC) run at

GN/GS for ingesting meta-data into MDDB.• Meta-data :

– augmented FITS headers, – environmental & weather data including images

(via Gemini weather Server), – observing proposals/programs/sequences, – observing logs,– publication lists (continually updated?)



Data Integrity/Security

• Gemini makes and keeps ‘local’ copies of all data on-site.

• Data sent/e-transferred to CADC and ingested.• CADC implementing off-site backup at UVic for

science, calibration and meta- data.• All data e-transfers done through VPN, encrypted.• Checksum integrity check planned at all sites

(summits, base facilities, CADC, UVic).• Likely setup effective ‘mirror’ of GSA at GS for

performance improvement/security.



Advanced Capabilities(to be further discussed/finalized)

• On-line, on-the-fly, data processing using Gemini iraf package for (re-)generation of optimally reduced data.

• Observations Tables: archive-independent tables of scientifically interesting data supersets. Allows cross archive searches. (NGC1333 x-ray to sub-mm)

• Source/Object Tables: archive-independent tables of sources extracted from Gemini data and other archives. (SVS13 fluxes 0.5um to 1mm).

• GUI improvements to support above and other developments.



GSA within the VO World• Overall Goal: to produce Gemini ‘data products’

which are VO-compliant.• Data products: conform with the VO concept that they

are independent of the telescope/instrument data taken with, and conform with some (yet to be?) agreed VO standard for packaging data + metadata + anything else, so that it can be read by any VO system.

Natural extension of GSA and advanced capabilities!



Documentation

• Gemini web-site: http:/www.gemini.edu/sciops/data/dataArchive.html

• Prototype cookbook in preparation.• Support for GSA through Gemini HelpDesk

– Tier 1 : National Gemini Offices– Tier 2 : CADC– Tier 3 : Gemini (Colin Aspin)



Possible Aus-VO involvement• Biggest Bottle-neck to generating scientific results from

Gemini observations is in the data reduction. (IRAF-based reduction scripts are available but have proven to be user-unfriendly and unreliable.)

• Problem recognised by Gemini and more resources now being devoted to developing “fully processed” data pipelines; however, more FTEs needed and welcomed!!

• Channel some Aus-VO resources into developing pipelines for instrument modes popular with Aus Gemini users??



Two points to take away…..• GSA increased Aus access to Gemini through making

ALL science observations taken across the partnership available (remembering that HST archive has increased its usefuless by ~ x3!!)

• Gemini must be a success story for Aus only 8m telescope facility we have a direct share in; it is not cheap (US$1/sec); need to MAXIMISE its use (science time + GSA) and scientific outcomes (papers, publicity etc)!

australian virtual observatory – 2003 workshop atnf, marsfield, 17-18 nov 2003 gemini and the...

Documents

archive data

data ingestion

new data

data distribution

image data

ingested data

transferal of data

data sente