The DEEP-ER project features a highly scalable, efficient, and user-friendly parallel I/O system based on the Fraunhofer parallel file system BeeGFS (formerly known as FhGFS). Additionally, it provides a low overhead, unified user-level checkpointing system and exploiting the multiple levels of non-volatile memory and storage added to the DEEP architecture.

 

Extensions to the Posix I/O API enable applications to efficiently use the file system and the different levels of the memory/storage subsystem. These extensions originate from BeeGFS itself, the parallel I/O library SIONlib, and Exascale10 – a novel I/O concept developed by the Exascale10 Workgroup.

 

Building on the capabilities of the DEEP-ER I/O system and the characteristics of the OmpSs programming model, a dual-approach resiliency concept combines a coarse-grained application-based multi-level checkpoint/restart mechanism with a less intrusive and more fine-grained scheme for task-controlled recovery from component failures.

 

Benchmarking is an essential element in evaluating the success of a hardware prototyping project. In the DEEP-ER project we use the JUBE benchmarking environment to assess the performance of the DEEP-ER system.

Highly-scalable, parallel I/O will be an important building block for future Exascale systems. In the DEEP-ER project, the I/O subsystem relies on 3 components: Fraunhofer's parallel file system BeeGFS, the parallel I/O library SIONlib, and Exascale10.

Read more ...

Software Overview DEEP-ERWithin the DEEP-ER project, developments are made in four key software areas:

  • Programming software
  • I/O software
  • Resiliency software
  • Benchmarking & Tools

The software used in DEEP-ER is mostly open source. The developments within the project have partially also been reflected in the release updates as it is for instance the case with SIONib or BeeGFS. Some software has been entirely developed within the project, e.g. the libNAM library which enables access to the Network Attached Memory (NAM).

Below you find an overview on the software used in the project and where to get more information, where to download the latest releases and to access the documentation.

  General Information Documentation Download

Programming Environment

OmpSs https://pm.bsc.es/ompss   https://pm.bsc.es/ompss-downloads
ParaStation MPI http://docs.par-tec.com/pdf/SPD-MPI2-5.0.4en.pdf http://docs.par-tec.com/html/psmpi-userguide/index.html http://www.par-tec.com/products/parastationv5.html

I/O Software

BeeGFS http://www.beegfs.com   http://www.beegfs.com/wiki/APIsOverview  http://www.beegfs.com/release/DEEP-ER/
SIONlib http://www.fz-juelich.de/jsc/sionlib https://apps.fz-juelich.de/jsc/sionlib/docu/index.html http://www.fz-juelich.de/ias/jsc/EN/Expertise/Support/Software/SIONlib/sionlib-download_node.html
E10 http://www.deep-er.eu/software/io/exascale10.html  not available not available

Resiliency

OmpSs https://pm.bsc.es/ompss   https://pm.bsc.es/ompss-downloads
SCR info to come    
libNAM info to come    

Benchmarking & Tools

JUBE http://www.fz-juelich.de/jsc/jube https://apps.fz-juelich.de/jsc/jube/jube2/docu/index.html http://www.fz-juelich.de/ias/jsc/EN/Expertise/Support/Software/JUBE/JUBE2/jube-download_node.html
Extrae https://tools.bsc.es/extrae https://tools.bsc.es/tools_manuals https://tools.bsc.es/downloads
Paraver https://tools.bsc.es/paraver https://tools.bsc.es/tools_manuals https://tools.bsc.es/downloads
Dimemas https://tools.bsc.es/dimemas https://tools.bsc.es/tools_manuals https://tools.bsc.es/downloads