Research conducted by Goethe University Frankfurt.

Our primary goal has been to efficiently integrate dynamic caching through XRootD/XCache into key workflows, leveraging the high-bandwidth WAN connection between Goethe HLR and GSI.

Currently, we’ve made significant progress in implementing hash-based file placement and replication mechanisms, which offer several benefits:

• Efficient data storage. Hashing data files and generating unique values determine their storage location within an HPC cluster.
• Geographical flexibility. This system enables data to be stored across different locations, enhancing both accessibility and redundancy.
• Data availability. The hash values also identify replica nodes, ensuring that data remains accessible even in the event of local failures.
• Accelerated workflows. A geographically distributed, hash-based system provides quick and reliable data access, which can significantly speed up data analysis workflows in FIDIUM.

In addition, we’ve successfully tested all basic operations on data lake prototypes, including:

• Authentication, using OIDC-token-based authentication for secure access.
• File management by implementing file placement and replication, along with tests using davix for file management.
• Final setup, by completing the XRootD-based setup in close collaboration with the GSI group.