Traditionally, scientific codes often use high-level data structures and multidimensional arrays that are afterwards translated into a linear set of bytes in a file. However, this translation destroys any spatial locality spreading adjacent data over the whole file and making any future access to it non-contiguous. Moreover, fine grain data domain partitioning strategies typically results in very small size I/O accesses from the applications. The combination of these two characteristics makes storage access to data very inefficient.
Currently available solutions to this problem, such as collective I/O, are very simplistic and not built for extreme scale. Hence, there is a need for parallel I/O mechanisms, which are able to overcome current collective I/O limitations. Exascale10 provides new collective I/O methods and extensions that integrate with the rest of the DEEP-ER I/O stack and are able to fully exploit the DEEP-ER architecture - in particular the BeeGFS cache layer.