= Hooks Framework =

<<TableOfContents>>

The mpi-start Hooks Framework allow the extension of mpi-start features without changing the core functionality. Several hooks are included in the default distribution of mpi-start for dealing with [[#File_distribution_hooks|file distribution]] and some [[#Extensions_hooks|MPI extensions]]. Site admins can check the [[#Local_site_hooks|local hooks]] description. Users probably are interested in [[#Developing_User_Hooks|developing their own hooks]].

== File distribution hooks ==

File distribution hooks are responsible for providing a common set of files prior to the execution of the application in all the hosts involved in that execution. Three steps are taken for file distribution:

 * Detection of shared filesystem: mpi-start will try to detect if the current working directory is in a network file system (currently considered as such are: nfs, gfs, afs, smb, gpfs and lustre). If the detection is positive, the distribution of files is not performed. Detection can be totally skipped by setting: `MPI_START_SHARED_FS` to 0 or 1 (0 means that mpi-start will try distribution in the next step, and 1 that it won't).

 * File distribution: in this step, mpi-start copies files from the current host to the other hosts involved in the execution. It uses the most suitable of the available distribution methods. Distribution methods are plugins that are detected at runtime by checking all the files with `.filedist` extension in the mpi-start `etc` directory.

 * Clean-up: once the job is finished, mpi-start will try to clean-up in the remote hosts the files that were copied previously using the same distribution method as in the previous step. '''Clean-up removes entire directories, so it should not be used when running from important paths (e.g. home directory)!'''. The clean-up step can be deactivated by setting `MPI_START_DISABLE_CLEANUP` to `"yes"`.

The file distribution method can be fixed by using the `I2G_MPI_FILE_DIST` variable.

=== File Distribution Method Plugins ===
A file distribution plugin must contain the following functions:

 * `check_distribution_method()`: called during initialization to check if the distribution method is suitable. It returns a number, the lower the number it returns, the higher priority it will have. If the method is not suitable, then it should return 255 or larger.
 * `copy()` perform the actual copy of files between hosts. Files are in a gzipped tarball pointed by the `TARBALL` variable.
 * `clean()`: clean up any files once the execution is finished.

These distribution methods are included in mpi-start:
 * ssh: uses scp to copy files, needs passwordless ssh properly configured.
 * mpiexec: uses OSC mpiexec for copying, needs OSC mpiexec installed.
 * cptoshared_area: copies files to a shared area that is not the current working directory. Needs the following variables:
   * `MPI_SHARED_HOME`: set to `"yes"`.
   * `MPI_SHARED_HOME_PATH`: path of the shared area that will be used for execution.
 * mpi_mt: uses mpi_mt for copying the files. Needs the mpi_mt binary to be available in all machines.

== Extensions hooks ==
Extension hooks are local site hooks that come in the default mpi-start distribution. The following hooks are available. These hooks can be completely removed by deleting the affinity.hook, openmp.hook, mpitrace.hook, marmot.hook, or compiler.hook in the mpi-start configuration directory.

=== Affinity ===
The Affinity hook is enabled by setting the `MPI_USE_AFFINITY` variable to `1`. When enabled (and the execution environment supports it), it will define the appropriate options for setting the processor affinity under the selected MPI implementation.
=== OpenMP ===
The OpenMP hook is enabled by setting the `MPI_USE_OMP` variable to `1`. When enabled it will define the `OMP_NUM_THREADS` environment variable to the number of processors available per mpi process.
=== MPItrace ===
MPItrace is enabled by setting the `I2G_USE_MPITRACE` variable to `1`. It adds to the execution the mpitrace utility, assuming it is installed at `MPITRACE_INSTALLATION`. Once the execution is finished, it gathers and creates the output files at the first host.
=== MARMOT ===
Marmot is a tool for analysing and checking MPI programs. This hook enables the use of the tool if the variable `I2G_USE_MARMOT` is set to `1`. It also copies the analysis output to the first host.
=== Compiler ===
This hook sets environment variables `MPI_MPI<COMPILER>`, where `COMPILER` is one of `CC`, `F90`, `F77`, `CXX`, for C, FORTRAN 90, FORTRAN 77 and C++ compilers respectively. This variables should point to valid compilers for the current MPI implementation. The hook also fixes compiler flags (`MPI_MPIxx_OPTS`) to avoid problems with bad flag for the current processor architecture. This hook can be disabled by setting the environment variable `MPI_COMPILER_HOOK` to `0`.

== Local Site Hooks ==


Site admins can define their own hooks by:
 * Creating new `.hook` files in the configuration directory, or
 * modifying the `mpi-start.hooks.local` file.

The `.hook` files are executed in alphabetical order and the `mpi-start.hooks.local` will be executed after any other hook in the system are executed and the shared file system detection is performed.
Each hook file contains the following functions:
 * `pre_run_hook ()`: it will be executed before the user hooks and the user application gets executed.
 * `post_run_hook ()`: it will be executed after the user application gets executed.
If any of these functions is not available, the hook will be ignored.

== Developing User Hooks ==
Users can also customize the mpi-start behavior defining their own hooks by using the `-pre` and `-post` command line switches or by setting the `I2G_MPI_PRE_RUN_HOOK` and `I2G_MPI_POST_RUN_HOOK` environment variables

 * `-pre` / `I2G_MPI_PRE_RUN_HOOK`: path of the file containing the pre-hook, in this file a function called `pre_run_hook()` must be available. This function will be called before the application execution. The pre-hook can be used, for example, to compile the executable itself or download data.

 * `-post` / `I2G_MPI_POST_RUN_HOOK`: path of the file containing the post-hook, in this file a function called `post_run_hook()` must be available. This function will be called once the applications finishes its execution. The post-hook can be used to analyze results or to save the results on the grid.

Both pre and post hooks can be in the same file. Next sections contain some hook examples

=== Compilation ===
Pre-run hook can be used for generating the binaries of the application that will be run by mpi-start. The following sample shows a hook that compiles an application using the C MPI compiler, as defined by the compiler hook in the `MPI_MPICC` variable. It assumes that the source code is called like the application binary, but with a `.c` extension. Use of complex compilation commands like configure, make, etc is also possible. This code is only executed in the first host. The results of the compilation will be available to all hosts thanks to the file distribution mechanisms.
{{{#!highlight sh
#!/bin/sh

# This function will be called before the execution of MPI application
pre_run_hook () {

  # Compile the program.
  echo "Compiling ${I2G_MPI_APPLICATION}"
  $MPI_MPICC $MPI_MPICC_OPTS -o ${I2G_MPI_APPLICATION} ${I2G_MPI_APPLICATION}.c
  if [ ! $? -eq 0 ]; then
    echo "Error compiling program.  Exiting..."
    return 1
  fi
  echo "Successfully compiled ${I2G_MPI_APPLICATION}"
  return 0
}
}}}


=== Input Preprocessing ===
Some applications require some input preprocessing before the application gets executed. For example, gromacs has a `grompp` tool that prepares the input for the actual `mdrun` application. In the following example the `grompp` tool prepares the input for gromacs:

{{{#!highlight sh
#!/bin/sh

pre_run_hook()
{
   echo "pre_run_hook called"

   # Here comes the pre-mpirun actions of gromacs
   export PATH=$PATH:/$VO_COMPCHEM_SW_DIR/gromacs-3.3/bin
   grompp -v -f full -o full -c after_pr -p speptide -np $MPI_START_NP

   return 0
}
}}}

Note the use of the `MPI_START_NP` variable to get the number of processors. See the developer section for a list of internal mpi-start variables.

=== Output Gathering ===
Applications that write output files in each of the hosts involved in the execution may need to fetch all those files to transfer them back to the user once the execution is finished. The following example copies all the `mydata.*` files to the first host. It uses the `mpi_start_foreach_host` function of mpi-start that will call the first argument for each of the hosts passing the name of the host as parameter.

{{{#!highlight sh
# the first paramter is the name of a host in the
my_copy () {
    CMD="scp . \$1:\$PWD/mydata.*"
    echo \$CMD
}

post_run_hook () {
    echo "post_run_hook called"
    if [ "x\$MPI_START_SHARED_FS" = "x0" ] ; then
        echo "gather output from remote hosts"
        mpi_start_foreach_host my_copy
    fi
    return 0
}
}}}

== Hooks Variable Summary ==

This section contains a summary of the variables that can modify the existing hook behaviour. They can be set using the `-d` command line switch.

|| '''Hook'''         ||  '''Variable'''        || '''Meaning''' ||
|| File Distribution  || `MPI_SHARED_FS`        || If undefined, mpi-start will try to detect a shared file system in the execution directory. If defined and equal to 1, mpi-start will assume that the execution directory is shared between all hosts and will not try to copy files. Any other value will make mpi-start assume that the execution directory is not shared. ||
|| File Distribution  || `I2G_MPI_FILE_DIST`    || Forces the use of a specific distribution method. ||
|| File Distribution  || `MPI_SHARED_HOME`      || If set to `"yes"`, mpi-start will use the path defined in `MPI_SHARED_HOME_PATH` for copying the files and executing the application. ||
|| File Distribution  || `MPI_SHARED_HOME_PATH` || Path to a shared directory. ||
|| File Distribution  || `MPI_START_DISABLE_CLEANUP` || If set to `"yes"`, mpi-start will not try to cleanup files after job execution. ||
|| OpenMP             || `MPI_USE_OMP`          || If set to 1, enable Open MP hook. ||
|| MPItrace           || `I2G_USE_MPITRACE`     || If set to 1, enable MPItrace hook.||
|| Marmot             || `I2G_USE_MARMOT`       || If set to 1, enable Marmot hook.||
|| Affinity           || `MPI_USE_AFFINITY`     || If set to 1, enable processor affinity hook. ||
|| Compiler           || `MPI_COMPILER_HOOK`    || If set to 0, disable compiler hook. ||