The Virtual Observatory and Ground-Based Data

John HuchraHarvard-Smithsonian

Center for Astrophysics

http://cfa-www.harvard.edu/~huchra

http://cfa-www.harvard.edu/~huchra/zcat

“We have plenty of information technology,

but not enough information.”S. Harris

Abstract:

• The era of extremely large, public databases in astronomy is upon us. Such databases are opening the field to new research and new researchers. However it is important to ensure that the resources are available to properly archive ground-based astronomical data, and to include the necessary quality checks, documentation and calibrations.

• A Virtual Observatory (VO) without proper archives will have limited usefulness. This also implies that with limited resources, not all data can or should be archived. NASA already has a good handle on US space-based astronomical data. Agencies and organizations that operate astronomical facilities, particularly ground-based observatories, need to plan and budget for these activities now.

• We should not underestimate the effort required to produce high quality data products that will be useful to the broader community.

• Currently the best way to fill archives is with data from surveys.

• This will continue to be the case for most ground-based observatories in the near future.

Astronomy from 1970 to 2003• Astronomy has been transformed by: Mainframes workstations Print media web Plates Digital detectors Punch cards Tapes Hard disk Mail Fax e-mail and this transformation is accelerating!

• There are both new scientific fields and new ways of doing science. We are being driven by larger and larger projects and also the cost of doing business.

• Cost is important! A key and *recurring* recommendation of the Astronomy Decadal Surveys has been to ensure the funding for infrastructure (read, in part, archives) and for the actual cost of doing science.

The Virtual Observatory• In recognition of this and of several other

contributing factors (e.g. more papers are now being written with HST archival data than with newly taken data), the most recent US Decadal Survey ranked as its highest priority small project (both ground- and space-based) the creation of a

National Virtual Observatory

• The NVO is now under development and is to have 4 main functions:

1. Integration of Major Archives

2. Advanced Services

3. Standards and tool development

4. Education and Public Outreach

and should be coordinated but distributed.

• The NVO should/will be comprised of several “things”:

A. Data Archives

B. Software/Algorithm Suites

C. Hardware

D. People (a.k.a. Human Capital)

Filling the NVO:

• What makes up the Data Archives?

1.Historical Archives (The Plate Stacks, NSSDC)

2. Knowledge Bases/Environments (including print & electronic media, e.g. ADS, NED, SIMBAD…)

3. Survey Data (DPOSS, 2MASS, FIRST, … 2dF, ROSAT, SDSS)

4. Mission Data (HST, ROSAT, IUE … JWST)

5. General Data (NOAO, ESO, NRAO, … Gemini,

Keck, ALMA, GSMT)

• Some of these items have been (more or less) taken care of. (1) is no longer growing since more and more of today’s data is going into (2).

• Some have been well attended to, (3) especially by NASA and its activities like MAST, HEASARC and IPAC.

• Some are best described as unmitigated archival disasters (5), especially at the US private observatories !!!

What is the best way to fill the VO today?

Surveys!

• Not counting the ‘bits’ in Knowledge bases, the vast majority of data available to the community today comes from survey databases, especially the all-sky maps from the optical and IR digitized sky surveys (DPOSS and DPOSS2, 2MASS) measured in multiple terabytes.

• The philosophy of surveys is changing from cataloging objects (e.g. SAO stars, the APM catalog, and even the 2MASS PSC), to serving images. The sky, digitized at the current Nyquist resolution around 1 micron, i.e. 0.1” pixels, and at full photometric resolution is approximately 100 Terabytes per bandpass.

• There are now maps of the whole sky

at all wavelengths/energies from a few kev to a meter (ROSAT, DPOSS, 2MASS, IRAS, FIRST, DPOSSII, NVSS, SUMSS). More and better maps are coming (e.g. SDSS & Galex), but some wavelengths/energies are still only poorly studied.

How are surveys handled?

The 2 Micron All-Sky Survey

My favorite example of a modern all-sky survey is 2MASS and its associated redshift survey, 2MRS.

2MASS was conceived in 1988 as a precursor to SIRTF. Its goal was to produce a map of the sky 1000 times deeper than the original TMSS, and to cover both hemispheres.

The survey specifications regarding uniformity, completeness and reliability were rigorous from the start. Strict controls were in place for the life of the project to ensure that those goals were met above all other desires (beware mission creep!).

2MASS Project Timeline:• 1988 Initial SMEX proposal to NASA

• 1991 Second NASA/NSF proposal (ground)

• 1993 Initial funding, Prototype camera

• 1995 IPAC start, Pipeline Build

• 1997 First light (FLWO)

• 1999 First Incremental Data Release

• 2000 Second DR, Observations Complete

• 2003 Final Data/Catalog Release

2MASS Milky Way

2MASS Galaxy Key Project Goals 1. Verify Galaxy Product Uniformity and Completeness Maps Bias Studies 2. Near-IR galaxy Properties 3. Galaxy Density Field to z ~ 0.1 Bright redshifts to K < 12.2, all, ~100,000 Faint redshifts, K < 13.5, 1-in-10, 150,000 4. Galaxy Velocity Field to z ~ 0.05, CMB Dipole Tully-Fisher to edge on sprials, Dn-sigma to E’s 5. AGN

DARK MATTER!

The 2MASS Redshift Survey• Motivation for 2MRS is simple: (a) Previous optical surveys, although

densely sampled, are biased (the other use of the word) by dust, star formation, etc. Most are based on crude photographic magnitudes, those that aren’t are not very extensive and are biased against the galactic plane.

(b) ongoing deeper optical surveys (SDSS , 2dF) will only cover a portion of the sky and are still biased by dust, SF, etc.

( c) Previous IR surveys were based on IRAS and are not very dense (e.g. smoothing lengths of 1000 km/s are needed, which are too large to see our motions inside the local supercluster, and are also biased by star formation --- the IRAS surveys have almost no early type galaxies

which are the objects that dominate dense regions.

They do not trace the local baryon/mass density. 2MASS does and does so through the plane of the Milky Way.

• Magenta V < 1000 km/s Magenta V < 1000 km/s

• Blue 1000 < V < 2000 km/sBlue 1000 < V < 2000 km/s

• Green 2000 < V < 3000 km/sGreen 2000 < V < 3000 km/s

Red 3000 < v < 4000

Blue 4000 < v < 5000Green 5000 < v < 6000

Red 6000 < v < 7000 km/s

Blue 7000 < v < 8000 km/s

Green 8000< v < 9000 km/s

Beware the warts! Extended Sources, Not All Galaxies

2MASS Sky Coverage Filter

Dollars and Sense:• A lesson can be learned from the cost breakdown of

the 2MASS project:• Hardware Two 1.3-m telescopes --- $1.5 M Two 3-channel cameras --- $1.5 M Other Hardware --- $0.5 M Personnel + Science Operations for 5 years --- $1.5 M Team Science --- $1.0 M Management --- $1.5 M Software & Archiving Pipeline ---- $16 M Archive Build & maintain --- $17 M ------------------------------------------------------- Total ~ $41.5 M

Project cost was driven by the archive!

Whence General Data?• For well over two decades, NASA has had a

policy of archiving and releasing raw and pipeline reduced data from space missions. It is that legacy which has driven the 2MASS output. It is not cheap, however. The MAST archive at StScI was developed from the original DMF (Data Management facility) DADS (Data Archive and Distribution Service) and cost between $30M and $60M to produce (depending on real vs optimistic accounting).

• Thus the much more general problem is what to do with the general data from ground-based telescopes. Some archives exist or are planned --- ESO for the VLT and Gemini will archive their data, and those observatories currently have budgets that can handle that load. Scaling from the SAO Telescope Data Center, a proper type 3 data archive (data quality control, media updates, indexed, retrieval software, etc.) at NOAO would cost between $2M and $3M/year and a few times that for initial construction. There is not yet full support for this activity at NOAO, at the Keck Observatory or at many other large public and private facilities. Does the VO need them?

• Connected with this is the broad and political issue of data ownership. When does data become public? A many, if not most, ground-based observatories, especially those w/o a working high level archive, the answer now is never.

• The NASA rule is that data is generally made public after 1 year unless it is taken as part of a long project or on a PI class mission. Should this apply to all data?

• While funding was identified in the

Astronomy and Astrophysics decadal survey for the development and deployment of the NVO in the US, no funding has been so identified for the archiving at both public and private ground-based observatories necessary to make NVO a complete success. We have violated the tenet of always recommending the resources necessary to get the science done!

These archives and the associated data policy need thought and $$$ now!

Conclusions and Recommendations• 1. The Virtual Observatory is an excellent

idea and will transform the way we do science.

• 2. Surveys will continue to play a (the!) major role in astronomy and are the best short term way of feeding the VO.

• 3. We need to be aware of archiving needs, especially for ground-based astronomy.

• 4. Data policy needs to be discussed. What do we really want/need in the VO?

• 5. The full cost of building and maintaining ground-based archives is not dissimilar to the cost of the physical facilities. In some cases it is much larger.

• 6. Standards, standards, standards!

• 7. There is absolutely no substitute for looking at the data.

Acknowledgements:• The 2MASS Extragalactic team: John Huchra (CfA),

Tom Jarrett (IPAC), Steve Schneider (UMass), Roc Cutri (IPAC), Lucas Macri (NOAO), Jeff Mader & Ted George (Texas), Mike Skrutskie (UVa), Chris Kochanek , Emilio Falco, Mike Pahre, Nathalie Martimbeau, Natalya Bizunok (CfA), C. Pantoja (PR)

• The 6dF Team: W. Saunders, Q. Parker, M. Colless, O. Lahav, G. Mamon, J. Lucey, E. Sadler, F. Watson, B. Boyle, R. Kraan-Korteweg, K. Wakamatsu, + W. van Driel, A. Fairall

• SAO Telescope Data Center: M. Kurtz, D. Mink, S. Tokarz, W. Wyatt, P. Berlind & M. Calkins

• This work has been supported in part by NASA, the Smithsonian Institution and a small research grant from the AAS.

• 2MASS was supported by NASA, the NSF, the Air Force, the Navy, the state of Massachusetts and the Smithsonian Institution.