solar data analysis

1

Solar Data Analysis

F. Breitling, G. Mann, C. Vocks

3nd Workshop

of the

Solar Key Science Project of LOFAR

Astrophysikalisches Institut Potsdam

2010 July 05

2010 July 5 Solar Data Analysis F. Breitling

OverviewThe three LOFAR data types

1. Dynamic spectra

2. Single station imaging

3. Standard imaging

RCU: Receiver unit

RSP: Remote station processing board

TBB: Transient buffer boards


2

Dynamic spectraData

• Beam-formed data (currently)– Container format: Hirachical Data Format 5 (HDF5)

– Specification: Interface Control Document 3

lofar-usg-icd-003.pdf

• Dynamic spectrum data (maybe in future)– Container format: Hirachical Data Format 5 (HDF5)

– Specification: Interface Control Document 6

lofar-usg-icd-006.pdf

=> both formats are currently under heavy development

Ref: http://usg.lofar.org/wiki/doku.php?id=documents:lofar_data_products


3

http://usg.lofar.org/wiki/doku.php?id=documents:lofar_data_products

Dynamic spectraSoftware

• Dynspec package

• developed by Solar KSP (Frank Breitling)

• part of LOFAR User Software (http://usg.lofar.org)

• build on Data Access Layer (DAL)

• Python scripts

• Components – specview.py (can read Fit and HDF5 files)

– downsample.py (can also create Fit files)

– rebin.py


4

Dynamic spectraFirst solar observations

Date: 2009-09-11 10:08:52 (ID108)

2009-09-11 13:09:39 (ID109)

Duration: 1h

LOFAR stations: CS302-LBA

RS307-HBA

RS503-HBA

Measured param.: voltages of both polarizations

Subbands/station: 62

Down sampling: 128


5

Dynamic spectraFirst results


6

Dynamic spectraIdentified problems

– stripe of high intensity in CS302, shifting from subband 30-40 for obs108 to subband 50-60 for obs109

– intensity peaks and drops for subband 0-20 in obs109_CS302LBA

– intensity gradient in obs109_RS503HBA


7

Dynamic spectraNext steps

• New versions of the beam-formed and dynamic spectra data format has to be finalized

• Dynspec software can be adapted

• Discussion with Michael Wise and Lars Bähren about their further development plans

• Travel to ASTRON is already planed for July 12-16


8

Single station imagingData

• Visibilities from LCU correlator– 96 receiver units (RCUs) x 2 polarizations x 16 byte

– Format: Binary

– Specification: • Menno Norden <norden at astron.nl>• rspctl --help (running on LCU)

Reference: http://usg.lofar.org/forum/index.php?topic=202.0


9

mailto:[email protected]

http://usg.lofar.org/forum/index.php?topic=202.0

Single station imagingSoftware

• acm2skyimage.m

• developed by Stefan Wijnholds <wijnholds at astron.nl>

• runs with Matlab and Octave

• Extended components by Frank Breitling– SolarAzEl.m to add the Sun

– RADec2AzEl.m to add other astronomical objects

(also published as Matlab package at http://www.mathworks.com/matlabcentral/fileexchange/25617-right-ascensiondeclination-to-azimuthelevation)

– make-movies.sh


10

Single station imagingPotsdam commissioning


11

Single station imagingResults from Effelsberg


12

Single station datacombining with GURT


13

Standard imagingBasics

• Basic data analysis software– CASA (Common Astronomy Software Applications)

http://casa.nrao.edu/

• Data format– Measurements Sets (MS) from CASA

• Data processing: Standard Imaging Pipeline– The LOFAR wiki

http://www.lofar.org/wiki/doku.php?id=software:standard_imaging_pipeline

– The LOFAR Imaging Cookbookhttp://www.mpa-garching.mpg.de/~fdg/LOFAR_cookbook/LOFAR_Cookbook_v2.3.pdf


14

Standard imagingPipeline

Components

• OLAP: Online Application Processing (Blue Gene/P correlator)

• DP3/DPPP: Default Pre-Processing Pipeline (for flagging and averaging)

• BBS: BlackBoard Selfcal (calibration)

• CImager: Imager

Control

• via parset files


15

Standard imagingFirst solar observations

Date: 2010-06-09 10:17:00, 11:41:00, 13:40:00, 15:40:00

Duration: 4 x 10 Min

LOFAR stations: 20 CS +3 RS, i.e.

RS106HBA, RS306HBA, RS503HBA

[CS00{1-7}, CS0{21,30}, CS103] HBA{0,1}

Measured param.: complex visibilities, 2 polarizations

Frequency range: ~150-220 MHz

Subbands /station: 7


16

Standard imagingData Inspection with casaplotms

• a lot of RFI• flagging with

flagdata• creation of

NDPPP.parset reprocessing

with NDPPP


17

Standard imagingSky model

Calibration using BlackBoard Selfcal requires skymodel

File contents of 078xx.skymodel:

# (Name, Type, Ra, Dec, I, Q, U, V, ReferenceFrequency='60e6',

SpectralIndexDegree='0', SpectralIndex:0='0.0', MajorAxis, MinorAxis, Orientation) = format

# The above line defines the field order and is required.

TauA.neb, GAUSSIAN, 05:34:32.00, +22.00.52.00, 1888.55, 0.0, 0.0, 0.0,81.5e6, 0, -0.299, 420.0, 290.0, 0.0

3C123, POINT, 04:37:04.72, +29.40.15.6, 454.97

3C133, POINT, 05:02:58.58, +25.16.22.7, 44.67

3C131, POINT, 04:53:23.26, +31.29.26.4, 37.54

3C139.2, POINT, 05:24:26.92, +28.12.54.2, 35.88

3C141, POINT, 05:26:43.55, +32.50.02.0, 31.76

3C132, POINT, 04:56:43.00, +22.49.23.7, 24.39

4C29.17, POINT, 05:10:40.10, +29.09.00.5, 19.48

4C29.16, POINT, 05:09:29.50, +29.57.56.3, 17.43

3C138, POINT, 05:21:10.46, +16.38.26.0, 14.63

PKS0531, POINT, 05:34:44.65, +19.27.21.9, 14.52

Taken from “The VLA Low-Frequency Sky Survey” catalog (http://www.cv.nrao.edu/4mass/VLSSlist.shtml)


18

Standard imagingBBS calibration

After running the BBS calibration

amplitudes and phases are stable


19

Standard imagingmwimager

after calibration the master worker imager creates:

– PSF – dirty image– clean image


20

Standard imagingDirty image


21

Dirty image

not yet sucessful

Standard imagingNext steps

• investigation of 2s clock offset of CS002-CS007

(found by Olaf Wucknitz)

• BBS calibration without short baselines

(suggested by James Anderson)

• automatic flagging of RFI

• travel to ASTRON on July 12-16 and discuss with the experts Ger van Diepen and Ger de Bruyn

• take more observations, e.g. new data on July 03


22

Progress documentation


23


24

Solar Science Data CenterYearly data volume

for solar observations with LOFAR

(assuming a maximum observation time of 8h/d)

Observation timegranted to the SKSP

Data per year[TB/yr]

Data rate[MB/s]

33% (maximum) 104 100

5% 16 100

3% 9.5 100

Summary

• The SKSP is engaged in 3 types of data analysis

• Dynamic spectra can already be taken, next the new format has to be completed

• Single station imaging is important for commissioning, single station operations and for our colleagues from GURT

• Standard imaging is the newest data for the SKSP and will need a lot of attention in the future

• Ultimately we plan for a Solar Science Data Center and work is ongoing


25

References

• User Software Group: http://usg.lofar.org/• LOFAR User Forum: http://usg.lofar.org/forum/index.php• LOFAR Interface Control Documents (ICDs)

http://usg.lofar.org/wiki/doku.php?id=documents:lofar_data_products• Documentation for the Standard Imaging Pipeline

http://www.lofar.org/wiki/doku.php?id=software:standard_imaging_pipeline• The LOFAR Imaging Cookbook

http://www.mpa-garching.mpg.de/~fdg/LOFAR_cookbook/LOFAR_Cookbook_v2.3.pdf

• Frank Breitling <fbreitling at aip.de> http://www.aip.de/People/fbreitling/


26


27

Yearly observation time

at Exloo (core stations) altitude >10° above horizon

=> tobs=~8 h/d =10x106 s/y

(Maximum possible)

solar data analysis

Documents

solar data analysisf

breitlingsolar data

hirachical data format

pdfdynamic spectrum

dynamic spectra data

futurecontainer format

interface control document

receiver unitrsp