Optimized meta-scheduling in Galaxy using TPV Broker

Abstract:

Description

Effective resource scheduling is critical in high-performance (HPC) and high-throughput computing (HTC) environments, where traditional scheduling systems struggle with resource contention, data locality, and fault tolerance. Meta-scheduling, which abstracts multiple schedulers for unified job allocation, addresses these challenges.

Galaxy, a widely used platform for data-intensive computational analysis, employs the \textit{Total Perspective Vortex (TPV)} system for resource scheduling. With over 550,000 users, Galaxy aims to optimize scheduling efficiency in large-scale environments. While TPV offers flexibility, its decision-making can be enhanced by incorporating real-time resource availability and job status.

This paper introduces the TPV Broker, a meta-scheduling framework that integrates real-time resource data to enable dynamic, data-aware scheduling. TPV Broker enhances scalability, resource utilization, and scheduling efficiency in Galaxy, offering potential for further improvements in distributed computing environments.

SEEK ID: https://workflowhub.eu/publications/55

DOI: 10.5281/zenodo.14936846

Teams: EuroScienceGateway

Publication type: Unpublished

Publisher: Zenodo

Citation: https://doi.org/10.5281/zenodo.14936846

Date Published: 1st Feb 2025

URL: https://doi.org/10.5281/zenodo.14936846

Registered Mode: imported from a bibtex file

Authors: Abdulrahman Azab, Paul De Geest, Sanjay Kumar Srikakulam, Tomáš Vondra, Mira Kuntz, Björn Grüning

Citation
Azab, A., De Geest, P., Srikakulam, S. K., Vondra, T., Kuntz, M., & Grüning, B. (2025). Optimized meta-scheduling in Galaxy using TPV Broker (Version 1.0). Zenodo. https://doi.org/10.5281/ZENODO.14936846
Activity

Views: 68

Created: 3rd Oct 2025 at 17:15

help Tags

This item has not yet been tagged.

help Attributions

None

Powered by
(v.1.17.0-main)
Copyright © 2008 - 2025 The University of Manchester and HITS gGmbH