Publication on the smart job scheduler implementation

Project: EuroScienceGateway was funded by the European Union programme Horizon Europe (HORIZON-INFRA-2021-EOSC-01-04) under grant agreement number 101057388 and by UK Research and Innovation (UKRI) under the UK government’s Horizon Europe funding guarantee grant number 10038963.

Document: D4.2 Publication on the smart job scheduler implementation

Work Package: Work Package 4. Building blocks for a sustainable operating model.

Task:

• Task 4.3 Implement a smart job-scheduling system across Europe

Lead Beneficiary: EGI

Contributing Beneficiary: ALU-FR, CESNET, EGI, UiO, and VIB

Executive Summary

Galaxy is currently using the Total Perspective Vortex (TPV) to schedule millions of jobs for hundred thousand users globally. While TPV has proven to be a robust meta-scheduling tool for Galaxy in the last years, there are areas of improvement that have been addressed in the EuroScienceGateway project:

• Gathering live usage metrics from across the distributed computing endpoints connected to Galaxy in order to distribute the load across all sites.
• Adding latitude and longitude attributes to data stores and computing endpoints to allocate jobs as close as possible to the location of the data.
• Visualizing job distribution across sites with an intuitive dashboard.

As a result the EuroScienceGateway project has developed two new tools:

• TPV Broker for the efficient meta-scheduling of jobs taking into account real-time usage metrics and data-locality information
• Galaxy Job Radar: a web dashboard to easily visualize the allocation of jobs across all sites

The EuroScienceGateway project has significantly improved the meta-scheduling of jobs for Galaxy, resulting in less waiting times for users to see their job completed and improving resource utilization across all sites.