ista 520/420 midterm aj jicha ryan jicha ian kaufer aj stangl roy zacharias “astrocats” - group...
DESCRIPTION
AstroCats Staff Project ManagersIan Kaufer, Roy Zacharias Chief ArchitectAJ Jicha System AdminsIan Kaufer, AJ Jicha Development/Test ManagerAJ Stangl Release ManagerAJ Stangl Chief ProgrammerRyan Jicha Technical WriterIan Kaufer, Roy ZachariasTRANSCRIPT
![Page 1: ISTA 520/420 Midterm AJ Jicha Ryan Jicha Ian Kaufer AJ Stangl Roy Zacharias “Astrocats” - Group 3](https://reader035.vdocuments.mx/reader035/viewer/2022062306/5a4d1b1b7f8b9ab0599936ca/html5/thumbnails/1.jpg)
ISTA 520/420Midterm
AJ JichaRyan JichaIan KauferAJ StanglRoy Zacharias
“Astrocats” - Group 3
![Page 2: ISTA 520/420 Midterm AJ Jicha Ryan Jicha Ian Kaufer AJ Stangl Roy Zacharias “Astrocats” - Group 3](https://reader035.vdocuments.mx/reader035/viewer/2022062306/5a4d1b1b7f8b9ab0599936ca/html5/thumbnails/2.jpg)
AgendaAstroCats Staff
Project Summary
Project Goals
Results
Data Processing Runtime
Obstacles
Future Goals
![Page 3: ISTA 520/420 Midterm AJ Jicha Ryan Jicha Ian Kaufer AJ Stangl Roy Zacharias “Astrocats” - Group 3](https://reader035.vdocuments.mx/reader035/viewer/2022062306/5a4d1b1b7f8b9ab0599936ca/html5/thumbnails/3.jpg)
AstroCats StaffProject Managers Ian Kaufer, Roy Zacharias
Chief Architect AJ Jicha
System Admins Ian Kaufer, AJ Jicha
Development/Test Manager AJ Stangl
Release Manager AJ Stangl
Chief Programmer Ryan Jicha
Technical Writer Ian Kaufer, Roy Zacharias
![Page 4: ISTA 520/420 Midterm AJ Jicha Ryan Jicha Ian Kaufer AJ Stangl Roy Zacharias “Astrocats” - Group 3](https://reader035.vdocuments.mx/reader035/viewer/2022062306/5a4d1b1b7f8b9ab0599936ca/html5/thumbnails/4.jpg)
Project Summary• Carl Herzenrother - requesting images processed using HPC for efficiency• Input: FITS files• Process:
• Python and Bash scripts• Generate modified FITS file• Generate Astrometrica config file• Linux, HPC, makeflow, Astrometrica, Astrometry.net
• Output:• Modified FITS image file• Astrometrica .cfg file for associated FITS image
![Page 5: ISTA 520/420 Midterm AJ Jicha Ryan Jicha Ian Kaufer AJ Stangl Roy Zacharias “Astrocats” - Group 3](https://reader035.vdocuments.mx/reader035/viewer/2022062306/5a4d1b1b7f8b9ab0599936ca/html5/thumbnails/5.jpg)
Project Goals
• Sort files with varying extensions (i.e. fit, fitz, .zip) into a common directory for processing
• Process each .fits file with solve field in a parallelized manner on HPC• Modify each new .fits file with calculated parameters from solve field (Field
center values)• Create Astrometrica configuration file with modified field rotation (PA) and
calculated focal length to match each new .fits file • Streamline process for easy replication by client on future datasets
![Page 6: ISTA 520/420 Midterm AJ Jicha Ryan Jicha Ian Kaufer AJ Stangl Roy Zacharias “Astrocats” - Group 3](https://reader035.vdocuments.mx/reader035/viewer/2022062306/5a4d1b1b7f8b9ab0599936ca/html5/thumbnails/6.jpg)
Project Work Flow
![Page 7: ISTA 520/420 Midterm AJ Jicha Ryan Jicha Ian Kaufer AJ Stangl Roy Zacharias “Astrocats” - Group 3](https://reader035.vdocuments.mx/reader035/viewer/2022062306/5a4d1b1b7f8b9ab0599936ca/html5/thumbnails/7.jpg)
Results• Scripts to process images and metadata functional• .makeflow script to process FITS images functional• Leveraging HPC/HTC and workers to process many tasks functional• Output files located in the following directories:
• /gsfs1/xdisk/ikaufer/midterm/midterm-Carl-Hergenrother/output/• Generate new FITS images functional• Generate .cfg files for FITS images functional• Generate list of unsolved/error files functional
![Page 8: ISTA 520/420 Midterm AJ Jicha Ryan Jicha Ian Kaufer AJ Stangl Roy Zacharias “Astrocats” - Group 3](https://reader035.vdocuments.mx/reader035/viewer/2022062306/5a4d1b1b7f8b9ab0599936ca/html5/thumbnails/8.jpg)
Data Processing Runtime• Benchmark:• 40 workers, 222 jobs: 2 hours 51 minutes
• Desired time:• 1,500 workers, 9,570 jobs: 2 hours 29 minutes
• Actual time:• 524 workers, 9,570 jobs: X hours Y minutes
![Page 9: ISTA 520/420 Midterm AJ Jicha Ryan Jicha Ian Kaufer AJ Stangl Roy Zacharias “Astrocats” - Group 3](https://reader035.vdocuments.mx/reader035/viewer/2022062306/5a4d1b1b7f8b9ab0599936ca/html5/thumbnails/9.jpg)
Obstacles• Issues running makeflow• .fits metadata in a single huge line• Three of the five group members did not have any standard HPC hours• Lack of uniform file extension naming scheme • Random .jpeg and .mp4 files• Spaces within file names
![Page 10: ISTA 520/420 Midterm AJ Jicha Ryan Jicha Ian Kaufer AJ Stangl Roy Zacharias “Astrocats” - Group 3](https://reader035.vdocuments.mx/reader035/viewer/2022062306/5a4d1b1b7f8b9ab0599936ca/html5/thumbnails/10.jpg)
Example Obstacle Findings
![Page 11: ISTA 520/420 Midterm AJ Jicha Ryan Jicha Ian Kaufer AJ Stangl Roy Zacharias “Astrocats” - Group 3](https://reader035.vdocuments.mx/reader035/viewer/2022062306/5a4d1b1b7f8b9ab0599936ca/html5/thumbnails/11.jpg)
Future Goals• Implement procedure to handle multiple pixel scale values
• Unify all parts of scripts into a single script for ease of use
![Page 12: ISTA 520/420 Midterm AJ Jicha Ryan Jicha Ian Kaufer AJ Stangl Roy Zacharias “Astrocats” - Group 3](https://reader035.vdocuments.mx/reader035/viewer/2022062306/5a4d1b1b7f8b9ab0599936ca/html5/thumbnails/12.jpg)
Questions?