= Developer information =

This section gives an overview about the interfaces of different components of MPI-Start for developing new plugins.

== Variables ==

All necessary information of the mpi-start program are provided via environment variables. See [[../UserDocumentation]] and [[../SiteConfiguration]] for a list of those variables.

== Logging and debugging ==

The `I2G_MPI_START_VERBOSE` variable can be set to 1 to turn on the additional output.
For debugging purpose the `I2G_MPI_START_DEBUG` variable can be set to 1 to enable
the debugging output.  The variable `I2G_MPI_START_TRACE` can be set to 1 to trace
every operation that is performed by mpi-start (goes to stderr).

The functions for logging messages to output are the following:
 * `error_msg`: always prints message
 * `debug_msg`: prints message if both `I2G_MPI_START_VERBOSE` and `I2G_MPI_START_DEBUG` are set
 * `info_msg` & `warn_msg`: print message if `I2G_MPI_START_VERBOSE` is set

The `dump_vars` function dumps the complete environment to the output.

== Scheduler plugins ==

A scheduler plugin is responsible for detection of a runtime system and
the definition of process distribution variables. A scheduler plugin must provide
the following functions and variables:

 * scheduler_available
  This function is called to detect if the job is running in the
  supported environment or not. If the supported environment is
  found 0 must be returned, else -1.

 * scheduler_get_machinefile
  After the successfully detection of a scheduler environment this
  function is called to set the all process distribution variables.

 * SCHEDULER_NAME
  A variable containing the name of the scheduler, it may be used
  later by the execution environment plugins to select the best
  way for starting the application.

The process distribution variables that the scheduler plugin must set are:
|| ''Variable'' || ''Meaning'' ||
|| '''MPI_START_MACHINEFILE''' || This variable points a machinefile in the standard format. In the  standard format there is one host per line. If multiple processes should run on a host this host must be inserted multiple times. ||
|| '''MPI_START_HOSTFILE''' || Variable pointing to a file with a list of available hosts with one host per line without host duplicates ||
|| '''MPI_START_HOST_SLOTS_FILE''' || Variable pointing to a file where for each line there is a hostname and the number of slots allocated in that host. ||
|| '''MPI_START_NHOSTS''' || Number of hosts allocated (number of lines in the MPI_START_HOSTFILE file) ||
|| '''MPI_START_NSLOTS''' || Number of slots allocated (number of lines in the MPI_START_MACHINEFILE file) ||
|| '''MPI_START_NSLOTS_PER_HOST''' || Number of slots allocated in each host. If the slots are not equally distributed, put here the maximum number of slots available on every host. ||

== Execution environment plugins ==

A Execution Environment plugin is responsible for setting up the environment
for the specified execution environment (MPI) version. It has to provide the
following functions:

 * mpi_start
  This function is called by the mpi-start core to start a parallel
  job. This function consists (at least) of the following steps:
   1. setup parallel environment if necessary (e.g. start daemons)
   2. activate the pre-run hooks (call mpi_start_pre_run_hook)
   3. call mpi_exec to start the parallel job
   4. activate the post-run hooks (call mpi_start_post_run_hook)
   5. shutdown parallel environment if necessary (e.g. shutdown daemons)
  A `generic_mpistart.sh` file contains this core functionality for development of new plugins.

 * mpi_exec
  This function actually perform the execution of the parallel job
  (mpirun/mpiexec). This functionality has been separated from the
  setup/shutdown phase to allow usage of the parallel environment in
  the pre-/post-run hooks.

Several internal variables may change the behavior of the execution environment:

|| ''Variable'' || ''Meaning'' ||
|| '''MPI_START_SCHEDULER''' || Name of the scheduler plugin loaded ||
|| '''MPI_START_MPI_PREFIX''' || Path of the execution environment ||

In order to provide a different environment for the user application, MPI-Start uses a wrapper that is executed as a different process. In this wrapper, the variables defined in the `MPI_START_ENV_VARIABLES` are exported to the environment. The use of the wrapper can be controlled with the functions:
 * `mpi_start_export_variable var_name [value]`  Adds the variable `var_name` to `MPI_START_ENV_VARIABLES` and to the wrapper script, if `value` is defined, set it also in the wrapper.
 * `mpi_start_execute_wrapper cmd_line`: Executes the specified command line in `cmd_line`. If `MPI_START_DO_NOT_USE_WRAPPER` is set to 1 it will not use a wrapper and therefore the environment variables may not be defined.

== Clean up ==

When mpi-start finishes its execution the `clean_up` function is called. This will remove any temporary files created by MPI-Start by default (e.g. host files) and any other file appended to the `MPI_START_CLEANUP_FILES` variable.

== Hooks ==

See [[../UserDocumentation/HooksFramework| Hooks Framework]]for a complete description of the hooks mechanism and how to develop new hooks.

== Internals ==

=== Global Configuration Variables ===
||'''Variable'''     || '''Default''' || '''Description'''        ||
||`MPI_START_DUMMY_SCHEDULER`    || 1 || Enables or disables the dummy scheduler. ||
||`I2G_MPI_START_KEEP_FILES`     || 0 || Enables or disables the removal of temporary files at the end of execution.||
||`I2G_MPI_START_FULL_TRACE`     || - || Enables or disables full trace of mpi-start. ||
||`MPI_START_DO_NOT_USE_WRAPPER` || - || Enables or disables the use of a wrapper for the executable ||
||`MPI_START_SOCKETS`            || - || Number of sockets in the host (if not defined, mpi-start tries to detect them). ||
||`MPI_START_COREPERSOCKET`      || - || Number of cores per socket in the host (if not defined, mpi-start tries to detect them). ||
||`MPI_START_COREPERSOCKET`      || - || Number of cores per socket in the host (if not defined, mpi-start tries to detect them). ||
||`I2G_MPI_START_ENABLE_TESTING` || - || If equal to `"TEST"`, then do not call the main function. Used for sourcing the mpi-start file.  ||

=== Scheduler plugin variables ===
||'''Variable'''              || '''Description'''        ||
||`MPI_START_SCHEDULER`       || Name of the scheduler. ||
||`MPI_START_HOSTFILE`        || File containing one line per slot available. ||
||`MPI_START_MACHINEFILE`     || File containing one line per host available. ||
||`MPI_START_HOST_SLOTS_FILE` || File containing one line with a name of host, and the number of slots available in that host. ||
||`MPI_START_NP`              || Total number of processors to use. ||
||`MPI_START_NPHOST`          || Number of processes per host, may be undefined if slot allocation is used. ||

=== MPI Execution Variables ===
|| '''Variable'''            || '''Description'''        ||
||`MPIEXEC`                            || Defined by each flavour, mpiexec executable ||
||`MPI_GLOBAL_PARAMS`                  || Global parameters for mpiexec ||
||`MPI_LOCAL_PARAMS`                   || Local parameters for mpiexec ||
||`MPI_START_SSH_AGENT`                || User (or system) specified ssh agent (used mostly by condor). ||
||`MPI_START_DISABLE_LRMS_INTEGRATION` || If set to `"yes"`, do not use any LRMS integration available in the MPI flavour. ||
||`MPI_MPICH2_DISABLE_HYDRA`           || If set to `1`, disable the use of hydra launcher for mpich2. ||
||`OSC_MPIEXEC`                        || Set to `1` if OSC mpiexec is found. ||
||`HYDRA_MPIEXEC`                      || Set to `1` if hydra mpiexec is found. ||
||`OPENMPI_VERSION_MAJOR`              || Set to Open MPI major version. ||
||`OPENMPI_VERSION_MINOR`              || Set to Open MPI minor version. ||
||`OPENMPI_VERSION_RELEASE`            || Set to Open MPI release version. ||