eFlows4HPC project aims at providing workflow software stack and an additional set of services to enable the integration of HPC simulations and modelling with big data analytics and machine learning in scientific and industrial applications. The project is also developing the HPC Workflows as a Service (HPCWaaS) methodology that aims at providing tools to simplify the development, deployment, execution and reuse of workflows. The project demonstrates its advances through three application Pillars with high industrial and social relevance: manufacturing, climate and urgent computing for natural hazards; these applications will help to prove how the realization of forthcoming efficient HPC and data-centric applications can be developed with new workflow technologies.
Web page: https://eflows4hpc.eu
Funding codes:- 955558
- MCIN/AEI/10.13039/501100011033 and the European Union NextGenerationEU/PRTR (PCI2021-121957)
This project has received funding from the European High-Performance Computing Joint Undertaking (JU) under grant agreement No 955558. The JU receives support from the European Union’s Horizon 2020 research and innovation programme and Spain, Germany, France, Italy, Poland, Switzerland, Norway.
Related items
Teams: COMPSs Tutorials
Organizations: Universitat Politècnica de Catalunya - BarcelonaTech (UPC)
Teams: COMPSs Tutorials
Organizations: Barcelona Supercomputing Center (BSC-CNS)
Teams: COMPSs Tutorials
Organizations: Istituto Nazionale di Geofisica e Vulcanologia (INGV)
Expertise: Computer Science
Teams: COMPSs Tutorials
Organizations: Barcelona Supercomputing Center (BSC-CNS)
Teams: COMPSs Tutorials
Organizations: Istituto Nazionale di Geofisica e Vulcanologia (INGV)
Teams: COMPSs Tutorials
Organizations: Barcelona Supercomputing Center (BSC-CNS)
https://orcid.org/0009-0005-9339-628XTeams: COMPSs Tutorials
Organizations: Barcelona Supercomputing Center (BSC-CNS)
https://orcid.org/0000-0002-9583-9022Teams: COMPSs Tutorials
Organizations: Indian Institute of Technology (BHU) Varanasi
Teams: COMPSs Tutorials
Organizations: Istituto Nazionale di Geofisica e Vulcanologia (INGV)
Teams: COMPSs Tutorials
Organizations: Universitat Politècnica de Catalunya - BarcelonaTech (UPC)
Team created to publish applications during COMPSs Tutorials, and share them among participants.
Space: eFlows4HPC
Public web page: https://www.bsc.es/education/training/bsc-training/bsc-training-course-programming-distributed-computing-platforms-compss/
Organisms: Not specified
Distributed computing aims to offer tools and mechanisms that enable the sharing, selection, and aggregation of a wide variety of geographically distributed computational resources in a transparent way. The research done in this team is based on the past expertise of the group, and on extending it towards the aspects of distributed computing that can benefit from this expertise. The team at BSC has a strong focus on programming models and resource management and scheduling in distributed computing ...
Space: eFlows4HPC
Public web page: https://www.bsc.es/discover-bsc/organisation/scientific-structure/workflows-and-distributed-computing
Organisms: Not specified
This team is to publish workflows executed from WPs that are not the three main pillars of the project.
Space: eFlows4HPC
Public web page: https://eflows4hpc.eu/
Organisms: Not specified
It explores the modelling of natural catastrophes – in particular, earthquakes and their associated tsunamis shortly after such an event is recorded.
Space: eFlows4HPC
Public web page: https://eflows4hpc.eu/pillars/
Organisms: Not specified
It develops innovative adaptive workflows for climate and for the study of Tropical Cyclones (TC) in the context of the CMIP6 experiment, including in-situ analytics.
Space: eFlows4HPC
Public web page: https://eflows4hpc.eu/pillars/
Organisms: Not specified
It focuses on the construction of DigitalTwins for the prototyping of complex manufactured objects integrating state-of-the-art adaptive solvers with machine learning and data-mining, contributing to the Industry 4.0 vision.
Space: eFlows4HPC
Public web page: https://eflows4hpc.eu/pillars/
Organisms: Not specified
Project that aims to create the NeuroPlat portal for neurodrug design
Space: eFlows4HPC
Public web page: https://www.upf.edu/web/cech
Organisms: Not specified
Abstract (Expand)
Authors: Raul Sirvent, Javier Conejero, Francesc Lordan, Jorge Ejarque, Laura Rodriguez-Navas, Jose M. Fernandez, Salvador Capella-Gutierrez, Rosa M. Badia
Date Published: 1st Nov 2022
Publication Type: Proceedings
DOI: 10.1109/WORKS56498.2022.00006
Citation: 2022 IEEE/ACM Workshop on Workflows in Support of Large-Scale Science (WORKS),pp.1-9,IEEE
Provenance registration is becoming more and more important, as we increase the size and number of experiments performed using computers. In particular, when provenance is recorded in HPC environments, it must be efficient and scalable. In this paper, we propose a provenance registration method for scientific workflows, efficient enough to run in supercomputers (thus, it could run in other ...
Creator: Raül Sirvent
Submitter: Raül Sirvent
Session during the Innovative HPC workflows for industry (https://eflows4hpc.eu/event/innovative-hpc-workflows-for-industry/) that describes how Workflow Provenance is recorded with COMPSs: the background on the tools used, how the recording has been designed, and how to use it and inspect metadata.
Creator: Raül Sirvent
Submitter: Raül Sirvent
Name: PhysioNet kNN Kfold Contact Person: support-compss@bsc.es Access Level: public License Agreement: Apache2 Platform: COMPSs Machine: MareNostrum5
Kfold to evaluate kNN accuracy on PhysioNet dataset (https://b2drop.bsc.es/index.php/s/8Q8MefXX2rrzaWs). This application used dislib-0.9.0
Name: PhysioNet CascadeCSVM Kfold Contact Person: support-compss@bsc.es Access Level: public License Agreement: Apache2 Platform: COMPSs Machine: MareNostrum5
Kfold to evaluate CascadeCSVM accuracy on PhysioNet dataset (https://b2drop.bsc.es/index.php/s/8Q8MefXX2rrzaWs). This application used dislib-0.9.0
Name: PhysioNet RF Kfold Contact Person: support-compss@bsc.es Access Level: public License Agreement: Apache2 Platform: COMPSs Machine: MareNostrum5
Kfold to evaluate RandomForest accuracy on PhysioNet dataset (https://b2drop.bsc.es/index.php/s/8Q8MefXX2rrzaWs). This application used dislib-0.9.0
Name: Random Forest Contact Person: support-compss@bsc.es Access Level: public License Agreement: Apache2 Platform: COMPSs Machine: MareNostrum4 This is an example of Random Forest algorithm from dislib. To show the usage, the code generates a synthetical input matrix. The results are printed by screen. This application used dislib-0.9.0
Name: GridSearchCV Contact Person: support-compss@bsc.es Access Level: public License Agreement: Apache2 Platform: COMPSs Machine: MareNostrum5
GridSearch of kNN algorithm for the iris.csv dataset (https://gist.githubusercontent.com/netj/8836201/raw/6f9306ad21398ea43cba4f7d537619d0e07d5ae3/iris.csv). This application used dislib-0.9.0
Name: GridSearchCV Contact Person: support-compss@bsc.es Access Level: public License Agreement: Apache2 Platform: COMPSs Machine: MareNostrum5
GridSearch of kNN algorithm for the iris.csv dataset (https://gist.githubusercontent.com/netj/8836201/raw/6f9306ad21398ea43cba4f7d537619d0e07d5ae3/iris.csv). This application used dislib-0.9.0
Name: Matrix Multiplication Contact Person: support-compss@bsc.es Access Level: public License Agreement: Apache2 Platform: COMPSs
Description
Matrix multiplication is a binary operation that takes a pair of matrices and produces another matrix.
If A is an n×m matrix and B is an m×p matrix, the result AB of their multiplication is an n×p matrix defined only if the number of columns m in A is equal to the number of rows m in B. When multiplying A and B, the elements of the ...
Name: Matrix Multiplication Contact Person: support-compss@bsc.es Access Level: public License Agreement: Apache2 Platform: COMPSs
Description
Matrix multiplication is a binary operation that takes a pair of matrices and produces another matrix.
If A is an n×m matrix and B is an m×p matrix, the result AB of their multiplication is an n×p matrix defined only if the number of columns m in A is equal to the number of rows m in B. When multiplying A and B, the elements of the ...
Lysozyme in water full COMPSs application
Lysozyme in water full COMPSs application, using dataset_small
Lysozyme in water full COMPSs application
Name: KMeans Contact Person: support-compss@bsc.es Access Level: public License Agreement: Apache2 Platform: COMPSs Machine: MareNostrum5
KMEans for clustering the housing.csv dataset (https://github.com/sonarsushant/California-House-Price-Prediction/blob/master/housing.csv). This application used dislib-0.9.0
Name: TruncatedSVD (Randomized SVD) Contact Person: support-compss@bsc.es Access Level: public License Agreement: Apache2 Platform: COMPSs Machine: MareNostrum5
TruncatedSVD (Randomized SVD) for computing just 456 singular values out of a (4.5M x 850) size matrix. The input matrix represents a CFD transient simulation of air moving past a cylinder. This application used dislib-0.9.0
Name: SparseLU Contact Person: support-compss@bsc.es Access Level: public License Agreement: Apache2 Platform: COMPSs
Description
The Sparse LU application computes an LU matrix factorization on a sparse blocked matrix. The matrix size (number of blocks) and the block size are parameters of the application.
As the algorithm progresses, the area of the matrix that is accessed is smaller; concretely, at each iteration, the 0th row and column of the current matrix are discarded. ...
COMPSs Matrix Multiplication, out-of-core using files. Hypermatrix size used 2x2 blocks (MSIZE=2), block size used 2x2 elements (BSIZE=2)
Monte Carlo Pi Estimation Program Description
This program is a Monte Carlo simulation designed to estimate the value of Pi using PyCOMPSs.
Tasks in the Program
- Count Points in Circle Task (
count_points_in_circle
):
- Generates random points within a square with side length 1.
- Counts points falling within the inscribed circle (x^2 + y^2 <= 1).
- Input: Number of points to generate (num_points)
- Output: Tuple containing count of points within the circle and list of generated ...
Name: Matrix multiplication with Files, reproducibility example, without data persistence Contact Person: support-compss@bsc.es Access Level: public License Agreement: Apache2 Platform: COMPSs
Description
Matrix multiplication is a binary operation that takes a pair of matrices and produces another matrix.
If A is an n×m matrix and B is an m×p matrix, the result AB of their multiplication is an n×p matrix defined only if the number of columns m in A is equal to the number ...
Name: Matrix multiplication with Files, reproducibility example Contact Person: support-compss@bsc.es Access Level: public License Agreement: Apache2 Platform: COMPSs
Description
Matrix multiplication is a binary operation that takes a pair of matrices and produces another matrix.
If A is an n×m matrix and B is an m×p matrix, the result AB of their multiplication is an n×p matrix defined only if the number of columns m in A is equal to the number of rows m in B. When multiplying ...
Name: Incrementation and Fibonacci Access Level: public License Agreement: Apache2 Platform: COMPSs
Description
Brief Overview: Demonstrates COMPSs task parallelism with increment and Fibonacci computations. Helps to understand COMPSs.
Detailed Description:
- Performs multiple increments of input values in parallel using COMPSs.
- Concurrently calculates Fibonacci numbers using recursive COMPSs tasks.
- Demonstrates task synchronization via
compss_wait_on
.
Execution
...
Type: COMPSs
Creators: Ashish Bhawel, Ashish Bhawel, Uploading this Workflow under the guidance of Raül Sirvent.
Submitter: Ashish Bhawel