Project profile
Spain, France, Sweden, Germany, Italy, Poland
01/04/2021 - 31/03/2024
Main Goal
Create an active I/O stack that dynamically adjusts computation and storage requirements through intelligent global coordination, leveraging elasticity of computation and I/O, and the scheduling of storage resources along all levels of the storage hierarchy. At the same time, it delivers quality-of-service (QoS), energy efficiency, and resilience for accessing extremely large data sets in very heterogeneous computing and storage environments.
Specific goals
- Enable the efficient use of new storage tiers through application of HPC-style scheduling and a distributed control mechanism that, based on global monitoring, can dynamically adapt storage allocations to changing application demands.
- Increase application throughput of HPC systems by leveraging the performance advantage of fast, node-local storage tiers through novel, European ad-hoc storage systems, and facilities for in-transit/in-situ processing.
- Balance computation and data transfers by providing elastic mechanisms to dynamically adjust the ratio between the allocations of compute and storage resources.
- Reduce I/O interference by globally minimising data transfers between storage tiers in a coordinated way while conveying and enforcing end-to-end QoS requirements.
- Provide tools to co-design applications and storage systems with the goal of minimizing data movement, targeting different HPC architectures.
- Increase power-efficiency in data management operations by reducing data movement and adopting low-power storage and CPU technologies.
Status / Highlights
Open source tools and libraries
- Available here
- Ad-hoc file systems: GekkoFS., Hercules, Expand
- Object store: DataClay
- CAPIO: Cross-Application Programmable I/O
- Monitoring tools: Limitless, Barreleye, Tau plugins
- Malleability tools: FlexMPI, Elastisim
- ADMIRE framework
Other results
Copyright © ADMIRE