Hooks Framework

The mpi-start Hooks Framework allow the extension of mpi-start features without changing the core functionality. Several hooks are included in the default distribution of mpi-start for dealing with file distribution and some MPI extensions. Site admins can check the local hooks description. Users probably are interested in developing their own hooks.

File distribution hooks

File distribution hooks are responsible for providing a common set of files prior to the execution of the application in all the hosts involved in that execution. Three steps are taken for file distribution:

• Detection of shared filesystem: mpi-start will try to detect if the current working directory is in a network file system (currently considered as such are: nfs, gfs, afs, smb, gpfs and lustre). If the detection is positive, the distribution of files is not performed. Detection can be totally skipped by setting: MPI_START_SHARED_FS to 0 or 1 (0 means that mpi-start will try distribution in the next step, and 1 that it won't).

• File distribution: in this step, mpi-start copies files from the current host to the other hosts involved in the execution. It uses the most suitable of the available distribution methods. Distribution methods are plugins that are detected at runtime by checking all the files with .filedist extension in the mpi-start etc directory.

• Clean-up: once the job is finished, mpi-start will try to clean-up in the remote hosts the files that were copied previously using the same distribution method as in the previous step. Clean-up removes entire directories, so it should not be used when running from important paths (e.g. home directory)!. The clean-up step can be deactivated by setting MPI_START_DISABLE_CLEANUP to "yes".

The file distribution method can be fixed by using the I2G_MPI_FILE_DIST variable.

File Distribution Method Plugins

A file distribution plugin must contain the following functions:

• check_distribution_method(): called during initialization to check if the distribution method is suitable. It returns a number, the lower the number it returns, the higher priority it will have. If the method is not suitable, then it should return 255 or larger.

• copy() perform the actual copy of files between hosts. Files are in a gzipped tarball pointed by the TARBALL variable.

• clean(): clean up any files once the execution is finished.

These distribution methods are included in mpi-start:

• ssh: uses scp to copy files, needs passwordless ssh properly configured.
• mpiexec: uses OSC mpiexec for copying, needs OSC mpiexec installed.
• cptoshared_area: copies files to a shared area that is not the current working directory. Needs the following variables:

  • MPI_SHARED_HOME: set to "yes".

  • MPI_SHARED_HOME_PATH: path of the shared area that will be used for execution.

• mpi_mt: uses mpi_mt for copying the files. Needs the mpi_mt binary to be available in all machines.

Extensions hooks

Extension hooks are local site hooks that come in the default mpi-start distribution. The following hooks are available. These hooks can be completely removed by deleting the affinity.hook, openmp.hook, mpitrace.hook, marmot.hook, or compiler.hook in the mpi-start configuration directory.

Affinity

The Affinity hook is enabled by setting the MPI_USE_AFFINITY variable to 1. When enabled (and the execution environment supports it), it will define the appropriate options for setting the processor affinity under the selected MPI implementation.

OpenMP

The OpenMP hook is enabled by setting the MPI_USE_OMP variable to 1. When enabled it will define the OMP_NUM_THREADS environment variable to the number of processors available per mpi process.

MPItrace

MPItrace is enabled by setting the I2G_USE_MPITRACE variable to 1. It adds to the execution the mpitrace utility, assuming it is installed at MPITRACE_INSTALLATION. Once the execution is finished, it gathers and creates the output files at the first host.

MARMOT

Marmot is a tool for analysing and checking MPI programs. This hook enables the use of the tool if the variable I2G_USE_MARMOT is set to 1. It also copies the analysis output to the first host.

Compiler

This hook sets environment variables MPI_MPI<COMPILER>, where COMPILER is one of CC, F90, F77, CXX, for C, FORTRAN 90, FORTRAN 77 and C++ compilers respectively. This variables should point to valid compilers for the current MPI implementation. The hook also fixes compiler flags (MPI_MPIxx_OPTS) to avoid problems with bad flag for the current processor architecture. This hook can be disabled by setting the environment variable MPI_COMPILER_HOOK to 0.

Local Site Hooks

Site admins can define their own hooks by:

• Creating new .hook files in the configuration directory, or

• modifying the mpi-start.hooks.local file.

The .hook files are executed in alphabetical order and the mpi-start.hooks.local will be executed after any other hook in the system are executed and the shared file system detection is performed. Each hook file contains the following functions:

• pre_run_hook (): it will be executed before the user hooks and the user application gets executed.

• post_run_hook (): it will be executed after the user application gets executed.

If any of these functions is not available, the hook will be ignored.

Developing User Hooks

Users can also customize the mpi-start behavior defining their own hooks by using the -pre and -post command line switches or by setting the I2G_MPI_PRE_RUN_HOOK and I2G_MPI_POST_RUN_HOOK environment variables

• -pre / I2G_MPI_PRE_RUN_HOOK: path of the file containing the pre-hook, in this file a function called pre_run_hook() must be available. This function will be called before the application execution. The pre-hook can be used, for example, to compile the executable itself or download data.

• -post / I2G_MPI_POST_RUN_HOOK: path of the file containing the post-hook, in this file a function called post_run_hook() must be available. This function will be called once the applications finishes its execution. The post-hook can be used to analyze results or to save the results on the grid.

Both pre and post hooks can be in the same file. Next sections contain some hook examples

Compilation

Pre-run hook can be used for generating the binaries of the application that will be run by mpi-start. The following sample shows a hook that compiles an application using the C MPI compiler, as defined by the compiler hook in the MPI_MPICC variable. It assumes that the source code is called like the application binary, but with a .c extension. Use of complex compilation commands like configure, make, etc is also possible. This code is only executed in the first host. The results of the compilation will be available to all hosts thanks to the file distribution mechanisms.

   1 #!/bin/sh
   2 
   3 # This function will be called before the execution of MPI application
   4 pre_run_hook () {
   5 
   6   # Compile the program.
   7   echo "Compiling ${I2G_MPI_APPLICATION}"
   8   $MPI_MPICC $MPI_MPICC_OPTS -o ${I2G_MPI_APPLICATION} ${I2G_MPI_APPLICATION}.c
   9   if [ ! $? -eq 0 ]; then
  10     echo "Error compiling program.  Exiting..."
  11     return 1
  12   fi
  13   echo "Successfully compiled ${I2G_MPI_APPLICATION}"
  14   return 0
  15 }

Input Preprocessing

Some applications require some input preprocessing before the application gets executed. For example, gromacs has a grompp tool that prepares the input for the actual mdrun application. In the following example the grompp tool prepares the input for gromacs:

   1 #!/bin/sh
   2 
   3 pre_run_hook()
   4 {
   5    echo "pre_run_hook called"
   6 
   7    # Here comes the pre-mpirun actions of gromacs
   8    export PATH=$PATH:/$VO_COMPCHEM_SW_DIR/gromacs-3.3/bin
   9    grompp -v -f full -o full -c after_pr -p speptide -np $MPI_START_NP
  10 
  11    return 0
  12 }

Note the use of the MPI_START_NP variable to get the number of processors. See the developer section for a list of internal mpi-start variables.

Output Gathering

Applications that write output files in each of the hosts involved in the execution may need to fetch all those files to transfer them back to the user once the execution is finished. The following example copies all the mydata.* files to the first host. It uses the mpi_start_foreach_host function of mpi-start that will call the first argument for each of the hosts passing the name of the host as parameter.

   1 # the first paramter is the name of a host in the
   2 my_copy () {
   3     CMD="scp . \$1:\$PWD/mydata.*"
   4     echo \$CMD
   5 }
   6 
   7 post_run_hook () {
   8     echo "post_run_hook called"
   9     if [ "x\$MPI_START_SHARED_FS" = "x0" ] ; then
  10         echo "gather output from remote hosts"
  11         mpi_start_foreach_host my_copy
  12     fi
  13     return 0
  14 }

Hooks Variable Summary

This section contains a summary of the variables that can modify the existing hook behaviour. They can be set using the -d command line switch.