Proof-of-concept gridded visibility stacking pipeline for deep spectral line interferometric imaging

Deep spectral line imaging is one of the greatest challenges for the SKA and its pathfinders, as the raw visibility
data-streams go beyond the practical capabilities of traditional long-term data storage and imaging techniques. The Deep Investigations of Neutral Gas Origins (DINGO) survey on the Australian Square Kilometre Array Pathfinder (ASKAP) is planning to observe for up to 2,500 hours per field for its deepest survey regions, likely spread over multiple years. As the corresponding raw datasets would be >100PB, and cannot be stored, the default pipeline will form daily images that are then averaged together to create the final deep image data products. This imaging method is severely limited for re-processing to improve final image quality. In particular, small systematic uncertainties, that may be unnoticeable on the daily images, could have a severe impact on the final data products. Therefore, compressed visibility storage is required for post-survey re-processing of the deep data. To meet this need, we developed an alternate DINGO pipeline in which visibilities are stored as a gridded data product. Gridded visibilities are sparse, so can be stored efficiently on a similar level as other compressed visibility formats, such as baseline-dependent averaged visibilities. This method is more than halves the long-term storage cost for the full ASKAP configuration (<30PB) and offers the lowest storage cost for a compact ASKAP configuration using only a subset of 30 antennas (∼5PB). Furthermore, gridding the data in this manner applies the correct kernels, whilst maintaining the ability to flag, reweight or even recalibrate the data. Thus, this approach addresses the greatest risks of the default daily imaging strategy. We present our proof-of-concept pipeline, and we demonstrate that our pipeline introduces no significant systematics as compared to accumulating the daily images. Furthermore, we report on our progress on imaging of DINGO pilot observations.

Theme – Data Processing Pipelines and Science-Ready Data