1000 1100 0101 1100 1010 1110 1011 1001 1111 0010 01000 1100 0101 1100 1010 1110 1011 1001 1111 0010...
TRANSCRIPT
1000 1100 0101 1100 1010 1110 1011 1001 1111 0010 0
1000 1100 0101 1100 1010 1110 1011 1001 1111 0010
1000 1100 0101 1100 1010 1110 1011 1001 1111 0010
1000 1100 0101 1100 1010 1110 1011 1001 1111 0010
1000 1100 0101 1100 1010 1110 1011 1001 1111 0010
1000 1100 0101 1100 1010 1110 1011 1001 1111 0010 0
1000 1100 0101 1100 1010 1110 1011 1001 1111 0010
1000 1100 0101 1100 1010 1110 1011 1001 1111 0010
1000 1100 0101 1100 1010 1110 1011 1001 1111 0010
1000 1100 0101 1100 1010 1110 1011 1001 0100 1011
1000 1100 0101 1100 1010 1110 1011 1001 1111 0010
1000 1100 0101 1100 1010 1110 1011 1001 1111 0010
1000 1100 0101 1100 1010 1110 1011 1001 1111 0010 0
1000 1100 0101 1100 1010 1110 1011 1001 1111 0010 0The National Virtual ObservatoryThe National Virtual Observatory
When and where are discoveries made? Always at the edges and boundaries Going deeper, using more colors….Metcalfe’s law Utility of computer networks grows as the number of possible connections: O(N2)VO: Federation of N archives Possibilities for new discoveries grow as O(N2)Current sky surveys have proven this amplification Very early discoveries from SDSS, 2MASS
Exponential growth Data will be never centralizedMore responsibility on projects Becoming Publishers and Curators Larger fraction of budget spent on software Lot of development duplicated, wastedMore standards are needed Easier data interchange, fewer toolsMore templates are needed Develop less software on your own
Astrophysical data is growing exponentially Doubling every year (Moore’s Law): both data sizes and number of data setsComputational resources scale the same way Constant $$$ will keep up with the dataMain problem is the software component Currently components are not reused Software costs are increasingly larger fraction Aggregate costs are growing exponentially
RolesAuthorsPublishersCuratorsConsumers
TraditionalScientistsJournalsLibrariesScientists
EmergingCollaborationsProject www siteBigger ArchivesScientists
Trends
Discoveries
Data Publishing
Changing Roles
Astrophysics has good track recordFITS: universally used to share low level data Individual images, tables, filesBut: new industry standards emerging XML, SOAPRequired by modern data exchange More dynamic (streams, queries) Merging heterogeneous sourcesGood time to adopt…
Accessing remote data: WWW, FTP Data formatted in certain ways (HTML, FITS)Accessing remote computing: Hard configured local area clusters Remote supercomputers Need to move data to the computing Resources do not always match problem
Standardizing distributed data Web Services, supported on all platforms Custom configure remote data dynamically XML: Extensible Markup Language SOAP: Simple Object Access Protocol WSDL: Web Services Description LanguageStandardizing distributed computing Grid Services Configure remote computing dynamically Build your own remote computer, and discard Virtual Data: new data sets on demand
Standards
Accessing Data Today
Distributed Services
Total area of 3m+ telescopes in the world in m2, total number of CCD pixels in Megapix, as a function of time. Growth over 25 years is a factor of 30 in glass, 3000 in pixels.
gamma-ray
x-rayUV Opt
IRradio
Terabytes
No. Collections0
10
20
30
40
50
60
70
80
90
The data sets in astrophysics today span the whole electromagnetic spectrum, with a large sky coverage. The total volume of data today is about 100TB, in over 50 collections. New surveys, like GALEX and PRIME will dramatically improve the all-sky, all-wavelength coverage of the Universe.
Next generation astronomical instruments have always enabled discoveries beyond their original goals. For example, HST’s new ACS camera sees 3000 galaxies where previously only a few hundred were known. NVO can capitalize on the richness of these new multiwavelength surveys.
Building the Framework for the NVO
Funded by NSF ITR (2001), $10M over 5 yearsCollaboration of 20 organizations
Astronomy data centersNational observatoriesSupercomputer centersUniversity departmentsComputer science/IT specialists
PI/Project Director: Alex Szalay (JHU)CoPI: Roy Williams (Caltech/CACR)
Major milestones associated with simple, intermediate and complex science demos
First science demonstrations planned for Jan 2003
External Advisory CommitteeExecutive CommitteeWorking GroupsProject TeamsEducation and Outreach Initiatives
Astrophysical Virtual Observatory funded by European Commission (€3.3 million, three years)AstroGrid, funded by UK e-science
program (£5 million, three years)Canadian Virtual ObservatoryNational initiatives in Germany, Russia, Australia,
Japan, China, India, …International VO roadmap developedInternational VO Alliance has been formed
Metadata information about resources - Waveband - Sky coverage - Translation of names to universal dictionarySimple search patterns on the resources - Cone Search - Image mosaic - Unit conversionsSimple filtering, counting, histogrammingOn-the-fly recalibrations
Built on Atomic ServicesPerform more complex tasks
- Automated resource discovery- Cross-identifications- Photometric redshifts- Outlier detections- Visualization facilities
Expectation:Build custom portals in days from existing
building blocks(like in IRAF or IDL)
Define commonly used `atomic’ servicesBuild higher level toolboxes/portals on topWe do not build `everything for everybody’Use the 90-10 rule: Define the standards and interfaces Build the framework Build the 10% that are used by 90% Let users build the rest from components
The NVO: How Will it Work?
Atomic Services
Higher Level Services
• NSF ITR NVO project is one of three major and numerous other NSF ITR NVO project is one of three major and numerous other small VO-related initiatives now underway world-widesmall VO-related initiatives now underway world-wide
• NVO is adopting, adapting, or developing necessary technology as NVO is adopting, adapting, or developing necessary technology as derived from science requirementsderived from science requirements
• NVO project is dealing with many of the management challenges NVO project is dealing with many of the management challenges that will face the ultimate VO organizationthat will face the ultimate VO organization
• NVO is inevitable—the next logical step in the evolution of the NVO is inevitable—the next logical step in the evolution of the astronomical research environmentastronomical research environment
SummarySummary
International Collaboration
Schedule
Organization
The NVO CollaborationThe NVO Collaboration
Sponsored by the National Science Foundation’s Information Technology Sponsored by the National Science Foundation’s Information Technology Research ProgramResearch Program
USNO
Data Mining Find unusual patterns and features