OpenFOAM 2.3.0: Parallel Running

Mesh Decomposition and Reconstruction

The following improvements have been made to mesh decomposition and reconstruction utilities:

• reconstructPar, reconstructParMesh and decomposePar now decompose, reconstruct all cell sets, face sets and point sets; this behaviour can be disabled by executing with the -noSets command line option.
• The decomposeParDict file contains some optional decomposition constraints as shown below. All utilities using the decomposition framework, e.g. decomposePar, snappyHexMesh and redistributePar, now apply these constraints.
• redistributePar now properly reconstructs and redistributes all fields, including those with complex boundary conditions.

preserveFaceZones       (heater solid1 solid3);
preservePatches         (cyclic_half0 cyclic_half1);
singleProcessorFaceSets ((f0 -1));
preserveBaffles         true;

GAMG Processor Agglomeration

The GAMG solver uses an iterative, sparse linear solver at its coarsest level. At that level, there will be a small number of cells per processor, e.g. 10, but usually far more inter-processor faces. This matrix structure becomes hard to solve, requiring a large number of iterations on the coarsest level. Each iteration requires local and global parallel communication, reducing scaling on larger numbers of processors.

In this release, to improve performance, the GAMG solver can agglomerate the matrix onto fewer processors at the coarser levels. In the limit of agglomerating the matrix onto a single processor, there are no processor interfaces and parallel communication is not required. This agglomeration across processors is controlled by an optional processorAgglomerator keyword in the solver sub-dictionary in fvSolution:

p
{
    solver                GAMG;
    ...
    smoother              GaussSeidel;
    nCellsInCoarsestLevel 10;
    agglomerator          faceAreaPair;
    processorAgglomerator masterCoarsest; // coarsest level on processor 0
    ...                                   // (master)
}

By default, there is no processor agglomeration. Below are options for processor agglomeration:

• none: no agglomeration, but prints statistics.
• masterCoarsest: combines the coarsest level onto the master processor; this method can be effective for a low number of processors to avoid the global reductions in the coarse-level solver; its operation is similar to the directSolveCoarsest option except it uses an iterative solver instead of a direct solver.
• manual: user control of agglomeration at each level.

  processorAgglomerator manual;
  processorAgglomeration
  (
      (
          3 // at level 3
          (
              (0 1) // create coarse 0 from 0,1 (and moved onto 0)
              (3 2) // create coarse 1 from 2,3 (and moved onto 3)
          )
      )
      (
          6 // at level6
          (
              (0 1) // create coarse 0 from 0,1 (and moved onto 0)
          )
      )
  );
  

• procFaces: below cell limit, specified by the nAgglomeratingCells keyword, starts agglomerating processors according to the number of faces between the processors using the faceAreaPair agglomeration algorithm.

  processorAgglomerator procFaces;
  nAgglomeratingCells 400;
  

Statistics are printed when any agglomerator method is used. The important parameter for the coarsest level is the ratio of inter-processor faces to number of cells. Without agglomeration this can increase dramatically. An example of the statistics is shown below:

                        nCells   nFaces/nCells   nInterfaces  nIntFaces/nCells
Level  nProcs      avg     max     avg     max   avg     max       avg     max
-----  ------      ---     ---     ---     ---   ---     ---
    0       8   391627  411140   2.995   3.021     4       6   0.03627 0.04739
    1       8   193294  203396   3.766   4.006     4       6   0.06254 0.08112
    2       8    96153  101207   4.655   4.991     4       6   0.09734   0.128
    3       8    47486   50085   5.459   5.721     4       6    0.1435  0.1911
    4       8    23614   24908   6.024   6.195     4       6    0.2026  0.2699
    5       8    11642   12318    6.41   6.614     4       6    0.2819  0.3711
    6       8     5787    6120   6.645   7.176     4       6     0.381  0.4952
    7       8     2852    3023   6.741   7.569     4       6     0.515  0.6662
    8       8     1416    1501   6.808   7.818     4       6    0.7012  0.9038
    9       8      697     739   6.778   8.008     4       6    0.9588   1.255
   10       4      687     714   7.113   8.125     2       3    0.6413  0.8913
   11       2      674     702   7.203   7.739     1       1    0.5465  0.5688
   12       1      665     665   7.493   7.493     0       0         0       0
   13       1      327     327   7.416   7.416     0       0         0       0
   14       1      158     158   6.994   6.994     0       0         0       0
   15       1       75      75   5.973   5.973     0       0         0       0
   16       1       36      36   4.556   4.556     0       0         0       0

Source code

• libOpenFOAM library
  $FOAM_SRC/OpenFOAM/matrices/lduMatrix/solvers/GAMG/GAMGProcAgglomerations