Community engagement in pipeline development using CWL, a case study for LOFAR

In modern experimental sciences, the increasing amount of data to process and the development of standardized procedures to analyze it, is driving the utilization of automatic processing pipelines. This is in particular true for radio interferometers where in order to create science ready data, a number of steps for data filtering, instrument calibration, and data transformation have to be performed on large datasets.
The capability to abstract the process, and increase the descriptive definition, of the pipelines seems to be key for maintaining future pipelines. This goal can be achieved using a standard pipeline definition language such as provided by the Common Workflow Language (CWL). As part of the effort towards realizing the European Open Science Cloud (EOSC) we have converted one of our standard LOFAR calibration pipelines to CWL. A fruitful and successful collaboration with the community maintainer of the pipeline has resulted in a demonstration of how CWL can improve supporting sustainable pipeline development and its community uptake. In this poster, we present the lessons learned and the benefits we found in the conversion process.


Theme – Data Processing Pipelines and Science-Ready Data, Open Source Software and Community Development in Astronomy