Research conducted by the University of Freiburg.

During our work on WP-1 (Exploration and efficient integration of opportunistic resources) and WP-2 (Accounting and controlling of heterogeneous resources), we have achieved significant milestones in developing tools for the integration of heterogeneous resources:

• Automatic start-stop mechanism. Start-stop mechanism has been developed. 
  In this mechanism, COBalD keeps one drone running for immediate job start. Unnecessary starts are prevented if no pending jobs are available in the overlay batch system (OBS).
• Plugins for accounting system. AUDITOR Plugin in TARDIS is used as an accounting system interface, and it retrieves drone information and triggers actions based on drone state changes.
• Design and implementation. Design, conception, and technology for the accounting system has been selected, and accounting prototype has been implemented.

AUDITOR accounting ecosystem overview

The AUDITOR ecosystem efficiently accounts for and controls heterogeneous resources through the following components:

• Core component. This component accepts, stores, and provides data. It was implemented in Rust, and is accessible via REST.
  The data is stored in a PostgreSQL database. The component is stateless, thus causing no data loss and being suitable for high availability setups.
  The component is available as RPM or Docker container with client libraries written in Rust and Python.

• Data collectors:

  • TARDIS collector gathers drone information.
  • 2 types of SLURM Collectors gather SLURM job data via CLI commands.
  • HTCondor collector gathers information about HTCondor jobs.
  • In the future, adding collectors for OpenStack, Kubernetes, and more is planned.

• Plugins:

  • Priority plugin computes and updates priorities on a batch cluster.
  • APEL accounting plugin (in progress) accounts for individual sites behind COBalD/TARDIS and reports data to WLCG accounting service (APEL).
  • Developing Utilization report for analyzing resource usage, potential savings, and CO2 footprint is planned for the future.

At the moment, 8 contributors from 3 universities are working on AUDITOR, 8 releases have been made, and the project can boast of continuous improvements and extensive documentation.

Future work

For the FIDIUM project extension, the following is planned:

• As we continue to explore and look for a way to integrate opportunistic resources efficiently, the development of Utilization Report Plugin is planned. The plugin will be able to analyze the proportion of the requested and consumed resources of a local user, and send a weekly report with possible savings and CO2 footprint.
• In the field of accounting and controlling of heterogeneous resources, development of a stress test collector/plugin is planned.
  The stress test collector/plugin will be capable of stress testing and performance measurements, contain a collector for testing uploads to AUDITOR, and a plugin for testing data access at AUDITOR.
  Basing on the results of the stress test analysis, further development and optimization of the AUDITOR database is also planned.