Close

7th July 2017

OpenFOAM Parallel I/O

When an OpenFOAM simulation runs in parallel, the data for decomposed fields and mesh(es) has historically been stored in multiple files within separate directories for each processor. Processor directories are named processorN, where N is the processor number.  In July 2017, the new collated file format was introduced to OpenFOAM-dev in which the data for each decomposed field (and mesh) is collated into a single file that is written (and read) on the master processor. The files are stored in a single directory named processors.  The work was carried out by Mattijs Janssens, in collaboration with Henry Weller (see contributors).

The new format produces significantly fewer files — one per field, instead of N per field, where N is the total number of processors.  For large parallel cases, it avoids limits on the number of open files imposed by the operating system.  The file writing can be threaded allowing the simulation to continue running while the data is being written to file — see below for details.  NFS (Network File System) is not needed when using the collated format and, additionally, there is a masterUncollated option to write data with the original uncollated format without NFS.

The controls for the file handling are in the OptimisationSwitches of the global etc/controlDict file:

OptimisationSwitches
{
    ...

    //- Parallel IO file handler
    // uncollated (default), collated or masterUncollated
    fileHandler uncollated;

    //- collated: thread buffer size for queued file writes.
    // If set to 0 or not sufficient for the file size threading is not used.
    // Default: 2e9
    maxThreadFileBufferSize 2e9;

    //- masterUncollated: non-blocking buffer size.
    // If the file exceeds this buffer size scheduled transfer is used.
    // Default: 2e9
    maxMasterFileBufferSize 2e9;
}

Selecting the File Handler

The fileHandler can be set for a specific simulation by:

  • over-riding the global OptimisationSwitches{fileHandler ...;} in the case controlDict file;
  • using the -fileHandler command line argument to the solver;
  • setting the FOAM_FILEHANDLER environment variable.

Updating Exisiting Files

A foamFormatConvert utility allows users to convert files between the collated and uncollated formats, e.g.

    mpirun -np 2 foamFormatConvert -parallel -fileHandler uncollated

An example case demonstrating the file handling methods is provided in: $FOAM_TUTORIALS/IO/fileHandling

Threading Support

Collated file handling runs faster with threading, especially on large cases.   But it requires threading support to be enabled in the underlying MPI.  Without it, the simulation will “hang” or crash.  For OpenMPI, threading support is not set by default prior to version 2, but is generally switched on from version 2 onwards.  For Ubuntu Linux, its openmpipackage has threading activated from version 17.04 (codename zesty) onwards, the first Ubuntu version that uses OpenMPI v2 (2.0.2).  The user can check whether OpenMPI is compiled with threading support by the following command

ompi_info -c | grep -oE "MPI_THREAD_MULTIPLE[^,]*"

When using the collated file handling, memory is allocated for the data in the thread.  maxThreadFileBufferSize sets the maximum size of memory that is allocated in bytes. If the data exceeds this size, the write does not use threading.

NOTE!!: if the user does not have threading enabled in their MPI, they should disable thread support for collated file handling by setting in the global etc/controlDict file:
maxThreadFileBufferSize 0;

For users who compile OpenMPIusing ThirdParty-dev,  threading has been enabled and you can update and recompile by:

cd $WM_THIRD_PARTY_DIR
git pull
rm -rf $MPI_ARCH_PATH
./Allwmake

When using the masterUncollated file handling, non-blocking MPI communication requires a sufficiently large memory buffer on the master node.  maxMasterFileBufferSize sets the maximum size of the buffer. If the data exceeds this size, the system uses scheduled communication.