Swarna K Ghosh

On behalf of : UVIT team, SAC-ISRO team

ASTROSAT AO-1 Workshop @ IUCAA, 13/14-Jul-2016

UVIT Data Analysis Pipeline UVIT Data Analysis Pipeline

End-to-end overview :

UVIT --> Science stream data --> S/C Data Handling Unit --> Solid State Recorder --> .... Transmission to ground (+ House Keeping; LBT; Aux ...)

Ground reception --> Data Ingest Front End Processor (DIFEP)--> Raw data --> Level-0 --> Level-1 Level-1 data --> Data Analysis Pipeline Level-2 data --> (end-user / astronomer friendly) .....

Key requirements of Level-1 to Level-2

data processing pipeline :

Sky images in FUV / NUV / VIS, corrected for various instrument effects, spacecraft drifts, jitters, thermal effects, ...

Recovering angular resolution, & Absolute aspect

(developed by SAC-ISRO in collaboration with UVIT team)

For Photon Counting (PC : FUV, NUV) & Integration (IM : VIS) Modes -

reject affected data : drop-outs, parity error, cosmic rays, ...

Instrumental effects corrected for :

- response variation over FoV; bad pixels - temperature dependence of QE - temperature dependence of MCP gain (IM only) - distortion introduced by Detector assembly - distortion introduced by Optic assembly - systematic effects in extraction of photon location from event centroid (PC only) – dark, bias,

- thermal effects on inter-channel mis-alignment;

Spacecraft Drift :

normally - recovered from series of sky images in VIS channel taken in Integration Mode (IM) at ~ 1 image/sec; relative shifts / rotation extracted by comparing a set of detected bright stars;

if VIS channel not available - recovered from NUV channel images in PC mode;

Spacecraft Jitter :

recovered from Gyro signals after filtering & integration;

Major Processing Chains / Blocks :

DataIngest (CRC; GTI; TCT; …) uvtDetectStaruvtComputeDrift

Relative Aspect Series (Integration Mode) “RA_IM”

Relative Aspect Series(Photon Counting Mode)“RA_PC”

Sky Imaging (Integration Mode) “Level2_IM”

Sky Imaging (Photon Counting Mode) “Level2_PC”

gg

DriftJitter Thermal

uvtRelativeAspectIM (top)

Drift Jitter ThermaluvtRelativeAspectIM (bottom)

2363 lines C++

uvtLevel2PC (top)

uvtLevel2PC (bottom)

3880 lines C++

"uvtDetectStar" [2593 line C++ code]

- input parameters - First cut peaks : Pixels above threshold (mean + n * sigma) -> candidates - Refined peaks (highest in window) - Centroids -> star list - Match stars to Reference Frame - Sorting by intensity

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

"uvtComputeDrift" [2310 line C++ code]

- input parameters - read time-centroid tables - match stars in successive frames (x1, y1, Int1; x2, y2, Int2) - Find Shift & Rotation (from successive frames) - Filtering (time domain; polynomial fitting; order selectable) - Time series of Roll-Yaw-Pitch; & x-y-theta => AUX4

[uvit@orca ~]$ cd AS1UVTLevel2-1.5/as1/uvit/src

“src"

common uvtDetectDistCorr uvtOpticAssDistCorrL2*DataIngest uvtDetectDistCorrL2 uvtPixPadding*uvtDetectStar uvtQEMCPCorrImageFrameGen uvtFindWtdMean uvtRefFrameCaluvtAccEveryTsec uvtFlatFieldCorr uvtRegAvguvtCentroidBias uvtFrameIntegration uvtRelAspCaluvtCentroidCorr uvtFullFrameAst $uvtRelativeAspectIM*uvtComputeDrift $uvtLevel2IM $uvtRelativeAspectPCuvtComputeJitter $uvtLevel2PC uvtShiftRotuvtComputeThermal uvtMaskBadPix uvtSubDivisionuvtCosmicRayCorr uvtOpticAssDistCorr uvtUnitConversion[uvit@orca src]$

---------“common"

[uvit@orca common]$ ls

alglibinternal.cpp Database_backup.cpp fft.h spMatrix.oalglibinternal.h Database_backup.h fft.o stdafx.halglibmisc.cpp Database.cpp gti.cpp transform.cppalglibmisc.h Database.h gti.h transform.hap.cpp DataInfo.cpp linalg.cpp transform.h_01Jul2016ap.h DataInfo.h linalg.h transform.h_17Jun2016Attitude.cpp DataInfo.o macro_def.h transform.oAttitude.h Directory.cpp Makefile uvtUtils.cppcaldb_Handler.cpp Directory.h spGeneral.h uvtUtils.hcaldb_Handler.h Directory.o spMatrix.cpp uvtUtils.h~caldb_Handler.o fft.cpp spMatrix.h uvtUtils.o

[uvit@orca common]$

"””DataIngest"

[uvit@orca DataIngest]$ ls

DataIngest DataIngest.h GTIParams_bkp.txt lib_crc.c lib_crc.o main.oDataIngest.cpp DataIngest.o GTIParams.txt lib_crc.h main.cpp Makefile

.......................................................... "uvtDetectStar"

[uvit@orca uvtDetectStar]$ ls

main.cpp uvtDetectStar uvtDetectStar.hmain.o uvtDetectStar_17Jun16.cpp uvtDetectStar_loca_drk.cppMakefile uvtDetectStar.cpp uvtDetectStar.o[uvit@orca uvtDetectStar]$

---------------------------------------------------

"uvtComputeDrift"

[uvit@orca uvtComputeDrift]$ ls

main.cpp Makefile uvtComputeDrift_17Jun16.cpp uvtComputeDrift.hmain.o uvtComputeDrift uvtComputeDrift.cpp uvtComputeDrift.o[uvit@orca uvtComputeDrift]$

"uvtRelativeAspectIM"

[uvit@orca uvtRelativeAspectIM]$ ls

diff_local_dark_8jul_17jun uvtRelativeAspectIM.cppmain.cpp uvtRelativeAspectIM.cpp_17Jun2016main.o uvtRelativeAspectIM.hmain.o_04Jul16 uvtRelativeAspectIM.oMakefile uvtRelativeAspectIM.o_04Jul16uvtRelativeAspectIM uvtRelativeAspectIM_round_8jul16.cppuvtRelativeAspectIM_04Jul16[uvit@orca uvtRelativeAspectIM]$

-------------------------------------------------------------------- "uvtRelativeAspectPC"

[uvit@orca uvtRelativeAspectPC]$ ls

main.cpp Makefile uvtRelativeAspectPC uvtRelativeAspectPC.hmain.o uvtPcRaCommanArr uvtRelativeAspectPC.cpp uvtRelativeAspectPC.o[uvit@orca uvtRelativeAspectPC]$

“uvtLevel2IM"

[uvit@orca uvtLevel2IM]$ ls

main.cpp uvtLevel2IM.cpp_moved_for_RYP uvtLevel2IM_RYP_bypass_test.cppmain.o uvtLevel2IM.h uvtLevel2IM_V1.5Makefile uvtLevel2IM_moved_for_RYP uvtLevel2IM_V1.51.cppuvtLevel2IM uvtLevel2IM.o uvtLevel2IM_V1.5.cppuvtLevel2IM.cpp uvtLevel2IM.o_moved_for_RYP weight_gt_1_bug[uvit@orca uvtLevel2IM]$

............................................................ "uvtLevel2PC"

[uvit@orca uvtLevel2PC]$ ls

main.cpp uvtLevel2PC_tempo_v1_51.cppmain.o uvtLevel2PC_tempo_v1.52.cppMakefile uvtLevel2PC_tempo_v1_53.cppuvtLevel2PC uvtLevel2PC_time_shift.cppuvtLevel2PC_24Jun16.cpp uvtLevel2PC_v1_51.huvtLevel2PC.h uvtLevel2PC_v1_5.cppuvtLevel2PC.o[uvit@orca uvtLevel2PC]$

"CALDB"

[uvit@orca CALD_IIA_23sep15]$ ls

AV_PH_ENERGY CPU_SCALES FLAT_FIELDS_FILTER PSF_LONG_TIMEBACKGD_COUNT DARK FLAT_FIELDS_WAVELENGTH PSF_SHORT_TIMEBAD_PIXELS DARK_NOISE GYRO_CAL QE_TEMPBIAS_CORR DISTORTION LOCAL_DARK TELDEFcaldb_mod.txt EFF_AP PH_ENERGY THERMAL_ASPECTCPU_GAIN FILTER_SPECT_RESP PLATE_SCALE WINDOW_CORR

[uvit@orca CALD_IIA_23sep15]$

Calibration Database

Enter level directory path [/Data1/uvit/post_door/1011_12/ LEVL1AS1UVT20151204P01_998T02_9000000190_01011.tar_V1.0f9x] :

Enter caldb directory path [/Data1/uvit/CALDB2/CALD_IIA_23sep15/] :

Enter level2 directory [/Data1/uvit/post_door/1011_12/out/] : /Data1/uvit/post_door/1011_12/test/

Channel (NUV/FUV/VIS) [VIS] :

Enter y for drop frame, n for drop packet [y/n] : 'y'

Dark Subtraction Flag(y/n) : 'y'

GTI filtering to be done or not(y/n) : 'n' [later on should become 'y' when necessary]

Run Flat Field Correction module [y/n]? : 'y'

Required Output dimension after padding [Must be integer] : '600'

Number of frames to accumulate [Must be Integer] : '1'

Run Correction for Effects of temperature on QE and MCP [y/n]? : 'y'

Actual running : uvtRelativeAspectIM (updated for V1.5)(1/3)

Run Pixel Subdivision [y/n]? : 'n' [600 enough for VIS in RA_IM]* (if 'y' above) Enter the SubDivision Size : (600) / 1200 / 2400 /4800 /9600

Enter value for Algorithm for star peaks : '3'

Enter Threshold value for star peaks : '16'

Minimum limit for the number of the stars : '15'

Enter Refine Window Size : '15'

Enter Centroid Window Size : '3'

Generate Matched Stars files after Drift Computetion(y/n) : 'y'

Threshold value for Cosmic Ray Correction : '400000'

Enter the number of frames to be Discarded for reference frame calculation : '1'

Enter the value of Average Factor for reference frame calculation : '1'

Min Distance value for pixel matching(for Compute Drift) : '1'

Enter 0 for fitting data ,1 for applying filtering in drift calculation,2 for no fittingand no filtering : '2' Algorithm to be used for finding drift(1/2/3) : '1'

uvtRelativeAspectIM (2/3)

Clobber[y/n] : [y] :

Need 'theta' computation (y/n) : 'y'

History [y/n] : [y] :

Want To Write output of unitConvertion to Disk?[1 for yes/0 for No] : '1'

Want To Write output of badPixel to Disk?[1 for yes/0 for No] : '1'

Want To Write output of pixPadding to Disk?[1 for yes/0 for No] : '1'* Want To Write output of SubDivision to Disk?[1 for yes/0 for No] : '1'

Want To Write output of CosmicRay to Disk?[1 for yes/0 for No] : '1'

Want To Write output of AccEveryTsec to Disk?[1 for yes/0 for No] : '1'

Want To Write output of uvtFindStarCentroid to Disk?[1 for yes/0 for No] : '1'

Want To Write output of uvtDetectorDistortion to Disk?[1 for yes/0 for No] : '1'

Want To Write output of Optical Distortion to Disk?[1 for yes/0 for No] : '1'

uvtRelativeAspectIM (3/3)

Enter level directory path [/Data1/uvit/post_door/1011_12/ LEVL1AS1UVT20151204P01_998T02_9000000190_01011.tar_V1.0f9x] :

Enter caldb directory path [/Data1/uvit/CALDB2/CALD_IIA_23sep15/] :

Enter level2 directory [/Data1/uvit/post_door/1011_12/out/] : /Data1/uvit/post_door/1011_12/test/Channel (NUV/FUV/VIS) : NUV

Enter y for drop frame, n for drop packet (y/n) : 'n'

Parity check(1 for All three words)/(2 for only X,Y) : '1'

Enter threshold for Cosmic Ray Correction : '100' {test case}

Enter The value of Compare frames for the Cosmic Ray : '10'

GTI filtering to be done or not(y/n) : 'n'

Enter threshold for multiple photon event <0.0 - 255.0> : '100'

Enter Algorithm number for finding the stars and centroid(1 for SAC algorithm) : '1'

Actual running : uvtLevel2PC (updated for V1.5)(1/3)

Enter multiplication factor for RMS for peak(star) detection : '20'

Enter Minimum limit for the number of the stars : '15' {for test; can be reduced}

Enter refined window Size for star detection : '15'

Enter Centroid window Size for centroid detection : '3'

path of Database [/Data1/uvit/CATALOG1/vis_uv_star_catalog.db] :

Enter algorithm number to be used for finding drift(1/3) : '1'

Enter the number of frames to discard : '2'

Enter the number of frames to compute : '9000'

Enter frame size after frameIntegration : '4800'

Enter final frame size after registration and averaging : '4800'

Enter the absolute path for the RAS file [/Data1/uvit/post_door/1011_12/out/uvit/01011/uvtV/uvtV.02/uvtComputeDrift_1.2/AS1P01_998T02_9000000190uvtVIIIM00F4_l2_dr.fits] :

Clobber(y/n) : [y] :

History (y/n) : [y] :

uvtLevel2PC (2/3)

Want To Write output of unitConvertion to Disk?[1 for yes/0 for No] : '1'

Want To Write output of badPixel to Disk?[1 for yes/0 for No] : '1'

Want To Write output of FlatField To Disk?[1 for yes/0 for No] : '1'

Want To Write output of pixPadding to Disk?[1 for yes/0 for No] : '1'

Want To Write output of Centroid Correction to Disk?[1 for yes/0 for No] : '1'

Want To Write output of Centroid Bias to Disk?[1 for yes/0 for No] : '1'

Want To Write output of subDivision to Disk?[1 for yes/0 for No] : '1'

Want To Write output of CosmicRay to Disk?[1 for yes/0 for No] : '1'

Want To Write output of uvtQEMCPCorr to Disk?[1 for yes/0 for No] : '1'

Want To Write output of Frameintegration to Disk?[1 for yes/0 for No] : '1'

Want To Write output of uvtDetectorDistortion to Disk?[1 for yes/0 for No] : '1'

Want To Write output of Optical Distortion to Disk?[1 for yes/0 for No] : '1'

uvtLevel2PC (3/3)

[uvit@orca uvtV.01]$ pwdData1/uvit/post_door/1191/outRA_IM_dark_old_V_dett_dist/uvit/01191/uvtV/uvtV.01[uvit@orca uvtV.01]$ ltotal 84drwxrwxr-x. 2 uvit uvit 53248 Jul 11 10:38 uvtComputeDrift_1.5drwxrwxr-x. 3 uvit uvit 76 Jul 11 09:01 uvtRefFrameCal_1.5drwxrwxr-x. 3 uvit uvit 21 Jul 11 08:54 uvtOpticalDistortion_1.5drwxrwxr-x. 3 uvit uvit 21 Jul 11 08:54 uvtDetectDistortion_1.5drwxrwxr-x. 4 uvit uvit 32 Jul 11 08:54 uvtFindStarCentroid_1.5drwxrwxr-x. 4 uvit uvit 46 Jul 11 08:54 uvtPixPadding_1.5drwxrwxr-x. 4 uvit uvit 46 Jul 11 08:54 uvtQEMCPCorr_1.5drwxrwxr-x. 4 uvit uvit 46 Jul 11 08:54 uvtFlatFieldCorrection_1.5drwxrwxr-x. 4 uvit uvit 46 Jul 11 08:54 uvtCosmicRayCorrection_1.5drwxrwxr-x. 4 uvit uvit 46 Jul 11 08:54 uvtMaskBadPix_1.5drwxrwxr-x. 4 uvit uvit 46 Jul 11 08:54 uvtAccEveryTsec_1.5drwxrwxr-x. 3 uvit uvit 25 Jul 11 08:54 uvtUnitConvertion_1.5drwxrwxr-x. 4 uvit uvit 4096 Jul 11 08:54 DataIngest_1.5[uvit@orca uvtV.01]$

-rw-rw-r--. 1 uvit uvit 77760 Jul 11 09:01 AS1G04_007T02_9000000220uvtVIIIM00F4_l2_dr.fits-rw-rw-r--. 1 uvit uvit 8640 Jul 11 09:01 AS1G04_007T02_9000000220uvtVIIIM00F4_l2_dr.info

uvtRelativeAspectIM (outputs)

[uvit@orca uvtN.01]$ pwdData1/uvit/post_door/1191_15/outL2_PC_0011_t_norm_4800/uvit/01191/uvtN/uvtN.01[uvit@orca uvtN.01]$ ltotal 8drwxrwxr-x. 2 uvit uvit 4096 Jul 11 21:39 uvtRegAvg_1.5drwxrwxr-x. 2 uvit uvit 6 Jul 11 16:43 uvtFullFrameAst_1.5drwxrwxr-x. 4 uvit uvit 46 Jul 11 16:42 uvtFrameIntegration_1.5drwxrwxr-x. 2 uvit uvit 61 Jul 11 16:41 uvtShiftRot_1.5drwxrwxr-x. 2 uvit uvit 60 Jul 11 16:41 uvtOpticDistCorr_1.5drwxrwxr-x. 2 uvit uvit 60 Jul 11 16:41 uvtDetectDistCorr_1.5drwxrwxr-x. 2 uvit uvit 60 Jul 11 16:41 uvtUnitConvertion_1.5drwxrwxr-x. 2 uvit uvit 60 Jul 11 16:41 uvtQEMCPCorr_1.5drwxrwxr-x. 2 uvit uvit 60 Jul 11 16:39 uvtFlatFieldCorr_1.5drwxrwxr-x. 2 uvit uvit 60 Jul 11 16:39 uvtCosmicRayCorrection_1.5drwxrwxr-x. 2 uvit uvit 60 Jul 11 16:39 uvtSubDivision_1.5drwxrwxr-x. 2 uvit uvit 60 Jul 11 16:39 uvtPixPadding_1.5drwxrwxr-x. 2 uvit uvit 60 Jul 11 16:39 uvtMaskBadPix_1.5drwxrwxr-x. 3 uvit uvit 4096 Jul 11 16:39 DataIngest_1.5[uvit@orca uvtN.01]$-rw-rw-r--. 1 uvit uvit 92168640 Jul 11 16:43 AS1G04_007T02_9000000220uvtNIIPC00F7_l2_t0.0000_f1_exp_rg.fits-rw-rw-r--. 1 uvit uvit 92168640 Jul 11 16:43 AS1G04_007T02_9000000220uvtNIIPC00F7_l2_t0.0000_f1_sig_rg.fits

uvtLevel2PC (outputs)

Thank you !