Calibrating Hubble Data in the Commercial Cloud
With the Hubble Space Telescope (HST), STScI and NASA pioneered ‘science ready data’ in the archive and made it possible for the community to go from observations to science as quickly as possible. The technologies required for calibrating observational data broadly breaks down into two main components: 1) the calibration pipeline that encodes the algorithms necessary for optimal extraction of data and 2) the supporting data management infrastructure used to run, manage, and monitor the pipelines and store the calibrated data products. In this talk I will present a project to deploy the HST calibration pipeline on commercial cloud infrastructure. I will discuss the trade-offs of using cloud-native vs. non-native services and changes we have had to make to our approach to processing in order to accommodate this workflow. I will also discuss new opportunities made available by the use of cloud services. Finally, I will compare costs between the two approaches.