a real-time software backend for the gmrt : towards hybrid backends casper meeting capetown 30th...
TRANSCRIPT
A A real-timereal-time software backend for the software backend for the
GMRT : towards hybrid backendsGMRT : towards hybrid backends
CASPER meeting Capetown 30th September 2009
Collaborators : Collaborators :
Jayanta Roy (NCRA)Jayanta Roy (NCRA)
Yashwant Gupta (NCRA)Yashwant Gupta (NCRA)
Ue-Li Pen (CITA)Ue-Li Pen (CITA)
Jeff Peterson (CMU)Jeff Peterson (CMU)
Jitendra Kodilkar (NCRA)Jitendra Kodilkar (NCRA)
Sanjay Kudale (NCRA)Sanjay Kudale (NCRA)
Existing GMRT backend Existing GMRT backend
30 antennas, 32 MHz, dual pol, 30 antennas, 32 MHz, dual pol, correlator + beam former correlator + beam former
Based on ASICs and DSPs ; Based on ASICs and DSPs ; designed in early 1990s ; running designed in early 1990s ; running for more than 10 yrs nowfor more than 10 yrs now
Limitations :Limitations : Lack of flexibility in time and Lack of flexibility in time and
frequency resolution frequency resolution Limited number of bits Limited number of bits Obsolescence; difficult to Obsolescence; difficult to
maintain maintain
New, desirable features :New, desirable features : Capability for recording raw Capability for recording raw
voltage signalsvoltage signals Facility to add new algorithms Facility to add new algorithms
e.g. RFI mitigation e.g. RFI mitigation Ability to form multiple beams Ability to form multiple beams
within the primary beam within the primary beam
Software backends Software backends
Software based back-ends :Software based back-ends : Few made to order hardware Few made to order hardware
components ; mostly off-the-shelf components ; mostly off-the-shelf items items
Easier to program ; more flexible Easier to program ; more flexible
GMRT Software Back-end (GSB) GMRT Software Back-end (GSB) :: 32 antennas32 antennas 32 MHz bandwidth, dual pol32 MHz bandwidth, dual pol Net input data rate : 2 Net input data rate : 2
Gsamples/sec Gsamples/sec FX correlator + beam former FX correlator + beam former Uses off-the-shelf ADC cards, CPUs Uses off-the-shelf ADC cards, CPUs
and switches to implement a and switches to implement a fully fully real-time back-endreal-time back-end
Raw voltage recording to disks, for Raw voltage recording to disks, for all antennas; off-line read back & all antennas; off-line read back & analysis analysis
Currently status : completed and Currently status : completed and ready to release as observatory ready to release as observatory facility (from 15facility (from 15thth October) October)
The GMRT software backend : The GMRT software backend : block diagram block diagram
The ADC Card The ADC Card
PCI-X compliant ADC card PCI-X compliant ADC card (Acquisition Logic)(Acquisition Logic)
4 channels, 8 bits each4 channels, 8 bits each
External trigger (from GPS pulse)External trigger (from GPS pulse)
External reference clock (from Rb)External reference clock (from Rb)
Several operating modes :Several operating modes : 33 / 66 MHz sampling, 4 33 / 66 MHz sampling, 4
channels (8 / 4 bits)channels (8 / 4 bits) 100 MHz, 2 channels (4 bits)100 MHz, 2 channels (4 bits) 200 MHz, 1 channel (2 bits)200 MHz, 1 channel (2 bits)
Data transfer from ADC card to Data transfer from ADC card to memory using DMA @ 145 MB/smemory using DMA @ 145 MB/s
Software flow : real-time mode Software flow : real-time mode
PAPABeamBeam
IAIABeamBeamADCADC
16 MHz16 MHzor or
32 MHz32 MHz
(with (with AGC)AGC)
Int Int Delay Delay CorrectCorrect
Filter Filter ++
DesampDesamp
FFTFFT++
FSTC FSTC & &
FringeFringeMACMAC
BeamBeamformerformer
visibilitiesvisibilities
64 analog64 analogInputsInputs
(32 ants, (32 ants, 2 pols)2 pols)
Software flow : real-time mode Software flow : real-time mode
Software flow : raw voltage mode Software flow : raw voltage mode
Performance Optimisation Performance Optimisation
Network transfer optimisation : jumbo packets Network transfer optimisation : jumbo packets
Computation optimisation : Computation optimisation : Intel IPP routines (for FFT)Intel IPP routines (for FFT) Vectorised operationsVectorised operations Cache optimisationCache optimisation Multi-threading load balancingMulti-threading load balancing
Performance specs :Performance specs : Better than 85% compute efficiencyBetter than 85% compute efficiency $190 / baseline ; 250 Mflops / W$190 / baseline ; 250 Mflops / W
Sample Results : Imaging Sample Results : Imaging
J1609+266 calibrator J1609+266 calibrator field at 1280 MHzfield at 1280 MHz
8.5 hrs synthesis image8.5 hrs synthesis image
Central source : 4.83 JyCentral source : 4.83 Jy
Noise level at HPBW : Noise level at HPBW : 34 microJy34 microJy
Dynamic range achieve : Dynamic range achieve : ~ 1.5 x10~ 1.5 x1055
Sample Results : Beamforming Sample Results : Beamforming
Phasing the array using a point Phasing the array using a point source calibratorsource calibrator
Single pulses from PSR B0329+54Single pulses from PSR B0329+54
Sample Results : Sample Results : Coherent Dedispersion Coherent Dedispersion
Real-time Data MonitoringReal-time Data Monitoring
New Capabilities : RFI mitigation New Capabilities : RFI mitigation
MAD filtering on raw time resolution data to eliminate bursty, MAD filtering on raw time resolution data to eliminate bursty, time domain RFI : works very nicelytime domain RFI : works very nicely
Future Plans : Multiple BeamformingFuture Plans : Multiple Beamforming
Offline processing mode of software back-end : beam-forming to produce Offline processing mode of software back-end : beam-forming to produce multiple multiple phased array beams covering the primary beamphased array beams covering the primary beam – for pulsar and transient science – for pulsar and transient science
Multiple Beamforming Multiple Beamforming
Can get full phased array sensitivity over the entire field of view Can get full phased array sensitivity over the entire field of view
Useful for searching for pulsars in extended targets (e.g. SNRs, GCs) Useful for searching for pulsars in extended targets (e.g. SNRs, GCs)
New Plans : Transient New Plans : Transient Search PipelineSearch Pipeline
Collaborative effort of NCRA and Swinburne University of Technology Running in piggy-back mode simultaneously with other observations Object of interest : Fast transients -- nanosecond to 100's of millisecond Event detection : based on the sensitivity of 8 antennae incoherent array
beamover 32 MHz Search in Dispersion measure space : Discriminate out fast
radio transient from RFI. Coincidence or anti-coincidence filter : Multiple sub-array
multiple beam coincidence filter reduces the false triggers due to direction-location dependent RFI.
Efficient RFI removal technique
New Plans : Transient Search New Plans : Transient Search Pipeline Pipeline
Multiple incoherent beam Multiple incoherent beam formation for transient pipeline formation for transient pipeline
3 incoherent array beams : PSR B0329+54
Detection of RFI by coincidence filter
RFI ?
Beam-1
Beam-2
Beam-3
Future Possibilities : Towards Future Possibilities : Towards Hybrid Designs Hybrid Designs
Upgraded GMRT : 32 stations, Upgraded GMRT : 32 stations, 400 MHz BW, dual pols 400 MHz BW, dual pols ~ 200 nodes (8 cores each)~ 200 nodes (8 cores each)
GPU accelaration feasibleGPU accelaration feasible
Mix and Match of FPGAs and Mix and Match of FPGAs and CPUs/GPUsCPUs/GPUs
Software backend for special Software backend for special applications :applications :
Spectral zoom modes Spectral zoom modes Raw voltage recording Raw voltage recording
modesmodes Debugging mode for new Debugging mode for new
arrays arrays
Thank YouThank You