Project profile

Spain, France, Sweden, Germany, Italy, Poland
01/04/2021 - 31/03/2024

Main Goal

Create an active I/O stack that dynamically adjusts computation and storage requirements through intelligent global coordination, leveraging elasticity of computation and I/O, and the scheduling of storage resources along all levels of the storage hierarchy. At the same time, it delivers quality-of-service (QoS), energy efficiency, and resilience for accessing extremely large data sets in very heterogeneous computing and storage environments.

Specific goals

  1. Enable the efficient use of new storage tiers through application of HPC-style scheduling and a distributed control mechanism that, based on global monitoring, can dynamically adapt storage allocations to changing application demands.
  2. Increase application throughput of HPC systems by leveraging the performance advantage of fast, node-local storage tiers through novel, European ad-hoc storage systems, and facilities for in-transit/in-situ processing.
  3. Balance computation and data transfers by providing elastic mechanisms to dynamically adjust the ratio between the allocations of compute and storage resources.
  4. Reduce I/O interference by globally minimising data transfers between storage tiers in a coordinated way while conveying and enforcing end-to-end QoS requirements.
  5. Provide tools to co-design applications and storage systems with the goal of minimizing data movement, targeting different HPC architectures.
  6. Increase power-efficiency in data management operations by reducing data movement and adopting low-power storage and CPU technologies.

Status / Highlights

Open source tools and libraries

  • Available here
  • Ad-hoc file systems:  GekkoFS., Hercules, Expand
  • Object store:  DataClay
  • CAPIO: Cross-Application Programmable I/O
  • Monitoring tools: Limitless, Barreleye, Tau plugins
  • Malleability tools: FlexMPI, Elastisim
  • ADMIRE framework

Other results

  • Co-designed two use cases
  • Open research datasets: available at Zenodo
  • Scientific publications: available at here
  • Webminars, workshop and talks: available here

Project coordinator

Universidad Carlos III De Madrid

 

Address: Spain

Website: https://www.uc3m.es/Home