Cost Management on Commercial Cloud Platforms
2020-11-11, 19:00–20:30, Times in UTC

Commercial cloud platforms are a powerful technology for astronomical research. The Event Horizon Telescope has processed much of its raw data on cloud platforms (Akiyama et al. 2019 - ApJL 875, L1; Kim et al. 2020; A & A, 640, A69 ) The Ice Cube neutrino experiment recently performed a similation experiment with 15,000 GPUs on three cloud platforms. Despite the benefits of cloud computing - such as on-demand scalability, and reduction of systems management overhead - confusion over how to manage costs remains for many one of the biggest barriers to entry, exacerbated by the rapid growth in services offered by commercial providers, and by the growth in the number of these providers. The confuses arises because storage, compute, and I/O are metered at separate rates, all of which can change without notice. As a rule, processing is very cheap, storage is more expensive, and downloading is very expensive. Thus an application that produces large image data sets for download will be far more expensive than an application that performs extensive processing on a small data set.

This BoF aims to quantify the above statement by presenting case studies of the costing of astronomy applications on commercal clouds, covering a range of processing scenarios, including:

  • Hosting the Rubin Observatory Interim Data Facility on a cloud platform.
  • Creating an all-sky mosaic of TESS survey images.
  • Summary of a cost management workshop at IPAC.
  • Launching Sci Server on a cloud platform.
  • Managing cloud services at STScI

​ Discussion of these and other cases are intended to answer the address the following questions:

  • What are the best practices that I can employ for estimating costs?
  • How do I pick the best platform for my application?
  • How do I take advantage of free or reduced costs services (educators or researchers credits; spot pricing; use an academic cloud...)?
  • What are the best practices for optimizing performance and reducing my costs?
  • What are the fiscal "black holes" that I can fall into?
  • Where can I find all this information?

Organizers: Bruce Berriman (Caltech/IPAC-NExScI); Gerard Lemson (JHU); William O'Mullane (Rubin Observatory); Ivelina Momcheva (STScI); Andreas Wicenic (ICRAR).


Theme – Cloud Computing at Different Scales