Publication on the Pulsar Network, integrated in workflow management systems

Description

Documentation of the design, deployment, and operationalization of the European Pulsar Network, developed within the EuroScienceGateway (ESG) project. This deliverable outlines how the Pulsar Network enables scalable, federated, and interoperable remote job execution across European Galaxy servers and compute infrastructures.

This record showcases the technical architecture, automation strategies, and monitoring solutions behind the distributed execution framework, supporting reproducible workflows and efficient resource sharing. The network connects 13 Pulsar endpoints across 10 countries, integrated with six national Galaxy servers and the European Galaxy server. Deployments span public clouds, institutional HPCs, and EOSC resources, unified under a secure, open-source infrastructure stack using Terraform, Ansible, RabbitMQ, CVMFS, and SABER.

The deliverable demonstrates how ESG addressed interoperability and scalability challenges through open infrastructure tooling, cross-institutional coordination, and continuous monitoring. It provides a replicable model for distributed compute resource integration and highlights Galaxy's extensibility in federated scientific computing.

Project: EuroScienceGateway, funded by the European Union’s Horizon Europe programme (HORIZON-INFRA-2021-EOSC-01) under grant agreement number 101057388.

Document: D3.2 Publication on the Pulsar Network, integrated in workflow management systems

Work Package: Work Package 3. Pulsar Network: Distributed heterogeneuos compute

Tasks:

• Task 3.1 Develop and maintain an Open Infrastructure-based deployment model for Pulsar endpoints
• Task 3.2 Add GA4GH Task Execution Service (TES) API to Pulsar
• Task 3.3 Build a European-wide network of Pulsar sites
• Task 3.4 Add TES support to WfExS (Workflow Execution Service)
• Task 3.5 Developing and maintaining national or domain-driven Galaxy servers

Lead Beneficiary: CNR

Contributing Beneficiaries: CNR, INFN, ALU-FR, CNRS, CESNET, UiO, UB, EPFL, AGH/AGH-UST, BSC, VIB, IISAS, TUBITAK, UNIMAN