![Page 1: Data mining and Visualization in Radio Astronomybkent/talks/bkent_vis2010.pdf · Visualization challenges • Sheer file and database size! • Variable reduction and Analysis efficiency](https://reader030.vdocuments.mx/reader030/viewer/2022041022/5ed3430bd520ca3ce638e694/html5/thumbnails/1.jpg)
Data mining and Visualization in Radio Astronomy
Brian R. KentNational Radio
Astronomy Observatory
http://www.cv.nrao.edu/~bkent/
Visualize 2010
![Page 2: Data mining and Visualization in Radio Astronomybkent/talks/bkent_vis2010.pdf · Visualization challenges • Sheer file and database size! • Variable reduction and Analysis efficiency](https://reader030.vdocuments.mx/reader030/viewer/2022041022/5ed3430bd520ca3ce638e694/html5/thumbnails/2.jpg)
STScI and NASA
![Page 3: Data mining and Visualization in Radio Astronomybkent/talks/bkent_vis2010.pdf · Visualization challenges • Sheer file and database size! • Variable reduction and Analysis efficiency](https://reader030.vdocuments.mx/reader030/viewer/2022041022/5ed3430bd520ca3ce638e694/html5/thumbnails/3.jpg)
STScI and NASA
![Page 4: Data mining and Visualization in Radio Astronomybkent/talks/bkent_vis2010.pdf · Visualization challenges • Sheer file and database size! • Variable reduction and Analysis efficiency](https://reader030.vdocuments.mx/reader030/viewer/2022041022/5ed3430bd520ca3ce638e694/html5/thumbnails/4.jpg)
Katie Chenoweth et al.
![Page 5: Data mining and Visualization in Radio Astronomybkent/talks/bkent_vis2010.pdf · Visualization challenges • Sheer file and database size! • Variable reduction and Analysis efficiency](https://reader030.vdocuments.mx/reader030/viewer/2022041022/5ed3430bd520ca3ce638e694/html5/thumbnails/5.jpg)
Springel et al.
![Page 6: Data mining and Visualization in Radio Astronomybkent/talks/bkent_vis2010.pdf · Visualization challenges • Sheer file and database size! • Variable reduction and Analysis efficiency](https://reader030.vdocuments.mx/reader030/viewer/2022041022/5ed3430bd520ca3ce638e694/html5/thumbnails/6.jpg)
Abazajian et al 2009 and the SDSS Survey
![Page 7: Data mining and Visualization in Radio Astronomybkent/talks/bkent_vis2010.pdf · Visualization challenges • Sheer file and database size! • Variable reduction and Analysis efficiency](https://reader030.vdocuments.mx/reader030/viewer/2022041022/5ed3430bd520ca3ce638e694/html5/thumbnails/7.jpg)
New Astronomy Across the Spectrum
More pixels Faster computing More data
Sensitive “cameras” Larger surveys Higher resolution
![Page 8: Data mining and Visualization in Radio Astronomybkent/talks/bkent_vis2010.pdf · Visualization challenges • Sheer file and database size! • Variable reduction and Analysis efficiency](https://reader030.vdocuments.mx/reader030/viewer/2022041022/5ed3430bd520ca3ce638e694/html5/thumbnails/8.jpg)
Jorden 2008 (Astrophysics Detector Workshops)
![Page 9: Data mining and Visualization in Radio Astronomybkent/talks/bkent_vis2010.pdf · Visualization challenges • Sheer file and database size! • Variable reduction and Analysis efficiency](https://reader030.vdocuments.mx/reader030/viewer/2022041022/5ed3430bd520ca3ce638e694/html5/thumbnails/9.jpg)
Visualization challenges
• Sheer file and database size!• Variable reduction and Analysis efficiency• Viewing modes for 3D and 4D data• Cross correlation of cataloged data• Efficient database mining• Mosaicking and projection algorithms
Center for Cosmo Physics
![Page 10: Data mining and Visualization in Radio Astronomybkent/talks/bkent_vis2010.pdf · Visualization challenges • Sheer file and database size! • Variable reduction and Analysis efficiency](https://reader030.vdocuments.mx/reader030/viewer/2022041022/5ed3430bd520ca3ce638e694/html5/thumbnails/10.jpg)
Use of Montage for large mosaicking…
Montage allows:
Online combination of imaging surveys
Limited online combination of user images
Unlimited combination programmatic interface
Berriman et al. 2006
![Page 12: Data mining and Visualization in Radio Astronomybkent/talks/bkent_vis2010.pdf · Visualization challenges • Sheer file and database size! • Variable reduction and Analysis efficiency](https://reader030.vdocuments.mx/reader030/viewer/2022041022/5ed3430bd520ca3ce638e694/html5/thumbnails/12.jpg)
The Arecibo Radio Telescope andArecibo L-band Feed Array
![Page 13: Data mining and Visualization in Radio Astronomybkent/talks/bkent_vis2010.pdf · Visualization challenges • Sheer file and database size! • Variable reduction and Analysis efficiency](https://reader030.vdocuments.mx/reader030/viewer/2022041022/5ed3430bd520ca3ce638e694/html5/thumbnails/13.jpg)
The new surveys of Radio Astronomy
• Meridian transit survey strategy.• 7000 square degrees of high galactic
latitude sky with the 7-element Arecibo L-band Feed Array.
• Sensitivity down to 2 x 107 Msun at DVirgo= 16.7 Mpc
• Primary sky targets include Virgo, Leo, and void regions.
• Objectives include public release catalogs and statistics, HI mass function determinations for various environments, and cross-correlation with other wavelength regimes.
Generating ~50 TB of processed datacubesKent et al. 2008, 2009
1225-1525 MHz
![Page 14: Data mining and Visualization in Radio Astronomybkent/talks/bkent_vis2010.pdf · Visualization challenges • Sheer file and database size! • Variable reduction and Analysis efficiency](https://reader030.vdocuments.mx/reader030/viewer/2022041022/5ed3430bd520ca3ce638e694/html5/thumbnails/14.jpg)
The new surveys of Radio Astronomy
• Meridian transit survey strategy.• 7000 square degrees of high galactic
latitude sky with the 7-element Arecibo L-band Feed Array.
• Sensitivity down to 2 x 107 Msun at DVirgo= 16.7 Mpc
• Primary sky targets include Virgo, Leo, and void regions.
• Objectives include public release catalogs and statistics, HI mass function determinations for various environments, and cross-correlation with other wavelength regimes.
Generating ~50 TB of processed datacubesKent et al. 2008, 2009
1225-1525 MHz
![Page 15: Data mining and Visualization in Radio Astronomybkent/talks/bkent_vis2010.pdf · Visualization challenges • Sheer file and database size! • Variable reduction and Analysis efficiency](https://reader030.vdocuments.mx/reader030/viewer/2022041022/5ed3430bd520ca3ce638e694/html5/thumbnails/15.jpg)
Right Ascension=12 hr
Arecibo Large Sky Survey
(Equatorial Coordinates)
![Page 16: Data mining and Visualization in Radio Astronomybkent/talks/bkent_vis2010.pdf · Visualization challenges • Sheer file and database size! • Variable reduction and Analysis efficiency](https://reader030.vdocuments.mx/reader030/viewer/2022041022/5ed3430bd520ca3ce638e694/html5/thumbnails/16.jpg)
FITS to IDL
Bandpass Calibration
Interferenceflagging
2D signal extraction
Data cube gridding
Baseline Flatfield
Data Cube Visualization
3D signal extraction
Flux Measurement
ALFALFA Data Processing Pipeline
Export
![Page 17: Data mining and Visualization in Radio Astronomybkent/talks/bkent_vis2010.pdf · Visualization challenges • Sheer file and database size! • Variable reduction and Analysis efficiency](https://reader030.vdocuments.mx/reader030/viewer/2022041022/5ed3430bd520ca3ce638e694/html5/thumbnails/17.jpg)
Frequency (100 MHz wide)
Tim
e (s
econ
ds)
Distant GalaxyGPSLong range radar
Milky Way
![Page 18: Data mining and Visualization in Radio Astronomybkent/talks/bkent_vis2010.pdf · Visualization challenges • Sheer file and database size! • Variable reduction and Analysis efficiency](https://reader030.vdocuments.mx/reader030/viewer/2022041022/5ed3430bd520ca3ce638e694/html5/thumbnails/18.jpg)
Each Data Element: • 4096 Freq. Channels• 2 polarizations• 7 receivers• 1 second sampling• 600 records• 20736 pixels
![Page 19: Data mining and Visualization in Radio Astronomybkent/talks/bkent_vis2010.pdf · Visualization challenges • Sheer file and database size! • Variable reduction and Analysis efficiency](https://reader030.vdocuments.mx/reader030/viewer/2022041022/5ed3430bd520ca3ce638e694/html5/thumbnails/19.jpg)
GRIDview: Spectroscopy & Visualization analysis tool
• Flexibility for – Native IDL structures– FITS data with astronomy headers
• Suite of OOPs include:– Visualization and imaging procedures– Spectral line measurement and fitting tools– Catalog and database programs
• Extensibility:– Import other user catalogs (ASCII, FITS, XML)– Add other imaging databases (WSDL, XML)
![Page 20: Data mining and Visualization in Radio Astronomybkent/talks/bkent_vis2010.pdf · Visualization challenges • Sheer file and database size! • Variable reduction and Analysis efficiency](https://reader030.vdocuments.mx/reader030/viewer/2022041022/5ed3430bd520ca3ce638e694/html5/thumbnails/20.jpg)
![Page 21: Data mining and Visualization in Radio Astronomybkent/talks/bkent_vis2010.pdf · Visualization challenges • Sheer file and database size! • Variable reduction and Analysis efficiency](https://reader030.vdocuments.mx/reader030/viewer/2022041022/5ed3430bd520ca3ce638e694/html5/thumbnails/21.jpg)
Map display window
Spectral normalization display window
![Page 22: Data mining and Visualization in Radio Astronomybkent/talks/bkent_vis2010.pdf · Visualization challenges • Sheer file and database size! • Variable reduction and Analysis efficiency](https://reader030.vdocuments.mx/reader030/viewer/2022041022/5ed3430bd520ca3ce638e694/html5/thumbnails/22.jpg)
Grid header and coordinate information
Cube manipulation controls
![Page 23: Data mining and Visualization in Radio Astronomybkent/talks/bkent_vis2010.pdf · Visualization challenges • Sheer file and database size! • Variable reduction and Analysis efficiency](https://reader030.vdocuments.mx/reader030/viewer/2022041022/5ed3430bd520ca3ce638e694/html5/thumbnails/23.jpg)
“Slide” to a lower Doppler velocity range
Galaxies are detected…
Nice spiral UGC 8475
![Page 24: Data mining and Visualization in Radio Astronomybkent/talks/bkent_vis2010.pdf · Visualization challenges • Sheer file and database size! • Variable reduction and Analysis efficiency](https://reader030.vdocuments.mx/reader030/viewer/2022041022/5ed3430bd520ca3ce638e694/html5/thumbnails/24.jpg)
Overlay catalogs from various sources…
![Page 25: Data mining and Visualization in Radio Astronomybkent/talks/bkent_vis2010.pdf · Visualization challenges • Sheer file and database size! • Variable reduction and Analysis efficiency](https://reader030.vdocuments.mx/reader030/viewer/2022041022/5ed3430bd520ca3ce638e694/html5/thumbnails/25.jpg)
Examine continuum maps…
![Page 26: Data mining and Visualization in Radio Astronomybkent/talks/bkent_vis2010.pdf · Visualization challenges • Sheer file and database size! • Variable reduction and Analysis efficiency](https://reader030.vdocuments.mx/reader030/viewer/2022041022/5ed3430bd520ca3ce638e694/html5/thumbnails/26.jpg)
Fetch radio images…
![Page 27: Data mining and Visualization in Radio Astronomybkent/talks/bkent_vis2010.pdf · Visualization challenges • Sheer file and database size! • Variable reduction and Analysis efficiency](https://reader030.vdocuments.mx/reader030/viewer/2022041022/5ed3430bd520ca3ce638e694/html5/thumbnails/27.jpg)
“Slide” to a lower Doppler velocity range
Galaxies are detected…
…including this nice spiral!
Back to Spectral mode…
![Page 28: Data mining and Visualization in Radio Astronomybkent/talks/bkent_vis2010.pdf · Visualization challenges • Sheer file and database size! • Variable reduction and Analysis efficiency](https://reader030.vdocuments.mx/reader030/viewer/2022041022/5ed3430bd520ca3ce638e694/html5/thumbnails/28.jpg)
Grab spectrum from a single pixel in the data cube.
Fetch images from online databases
![Page 29: Data mining and Visualization in Radio Astronomybkent/talks/bkent_vis2010.pdf · Visualization challenges • Sheer file and database size! • Variable reduction and Analysis efficiency](https://reader030.vdocuments.mx/reader030/viewer/2022041022/5ed3430bd520ca3ce638e694/html5/thumbnails/29.jpg)
Fetch data from NASA Extragalactic database
![Page 30: Data mining and Visualization in Radio Astronomybkent/talks/bkent_vis2010.pdf · Visualization challenges • Sheer file and database size! • Variable reduction and Analysis efficiency](https://reader030.vdocuments.mx/reader030/viewer/2022041022/5ed3430bd520ca3ce638e694/html5/thumbnails/30.jpg)
Obtain a larger image and position
![Page 31: Data mining and Visualization in Radio Astronomybkent/talks/bkent_vis2010.pdf · Visualization challenges • Sheer file and database size! • Variable reduction and Analysis efficiency](https://reader030.vdocuments.mx/reader030/viewer/2022041022/5ed3430bd520ca3ce638e694/html5/thumbnails/31.jpg)
Query archival databases
![Page 32: Data mining and Visualization in Radio Astronomybkent/talks/bkent_vis2010.pdf · Visualization challenges • Sheer file and database size! • Variable reduction and Analysis efficiency](https://reader030.vdocuments.mx/reader030/viewer/2022041022/5ed3430bd520ca3ce638e694/html5/thumbnails/32.jpg)
![Page 33: Data mining and Visualization in Radio Astronomybkent/talks/bkent_vis2010.pdf · Visualization challenges • Sheer file and database size! • Variable reduction and Analysis efficiency](https://reader030.vdocuments.mx/reader030/viewer/2022041022/5ed3430bd520ca3ce638e694/html5/thumbnails/33.jpg)
Summary• The GRIDview suite is a fast, OOP suite written in IDL
for 3D spectroscopic visualization.• The software is used on several large astrophysical
projects across 40 professional and academic institutions.
• Information: http://www.cv.nrao.edu/~bkent/computing