ChiVOData: A scalable data science platform for the heterogeneity of astronomical data in an HPC environment.
2020-11-08, 14:00–14:00, Posters

One of the limitations for any interactive astronomical data analysis platform is that in most cases there is a one-to-one relationship between observation and reduction pipeline. This limits the scalability and integration of these platforms with new data sources. On the other hand, these platforms must have the capacity to manage large volumes of data and offer the interfaces for the use of high-performance hardware. ChivoData is a new generation of astronomical data analysis platforms, built for the heterogeneity of astronomical data and its interoperability. It is based on microservices architecture, where each work environment is packaged with its libraries and software in an HPC container managed by Singularity. These containers are consumed as processing kernels by a jupyter notebook based platform. The latter allows the user to transparently use the HPC resources managed by SLURM. In addition, ChivoData is fully integrated with Luster file system, which allows the user to have their data centrally, only changing the processing engine according to the data they want to process. ChiVOData is a scalable full data science platform capable of supporting the heterogeneity of astronomical data in an HPC environment.


Theme – Science Platforms and Data Lakes, Cloud Computing at Different Scales, Data Interoperability