This section gives an overview about the interfaces of different components of MPI-Start for developing new plugins.
Variables
All necessary information of the mpi-start program are provided via environment variables. See ../UserDocumentation and ../SiteConfiguration for a list of those variables.
Logging and debugging
The I2G_MPI_START_VERBOSE variable can be set to 1 to turn on the additional output. For debugging purpose the I2G_MPI_START_DEBUG variable can be set to 1 to enable the debugging output. The variable I2G_MPI_START_TRACE can be set to 1 to trace every operation that is performed by mpi-start (goes to stderr).
The functions for logging messages to output are the following:
error_msg: always prints message
debug_msg: prints message if both I2G_MPI_START_VERBOSE and I2G_MPI_START_DEBUG are set
info_msg & warn_msg: print message if I2G_MPI_START_VERBOSE is set
The dump_vars function dumps the complete environment to the output.
Scheduler plugins
A scheduler plugin is responsible for detection of a runtime system and the definition of process distribution variables. A scheduler plugin must provide the following functions and variables:
- scheduler_available
- This function is called to detect if the job is running in the supported environment or not. If the supported environment is found 0 must be returned, else -1.
- scheduler_get_machinefile
- After the successfully detection of a scheduler environment this function is called to set the all process distribution variables.
- SCHEDULER_NAME
- A variable containing the name of the scheduler, it may be used later by the execution environment plugins to select the best way for starting the application.
The process distribution variables that the scheduler plugin must set are:
Variable |
Meaning |
MPI_START_MACHINEFILE |
This variable points a machinefile in the standard format. In the standard format there is one host per line. If multiple processes should run on a host this host must be inserted multiple times. |
MPI_START_HOSTFILE |
Variable pointing to a file with a list of available hosts with one host per line without host duplicates |
MPI_START_HOST_SLOTS_FILE |
Variable pointing to a file where for each line there is a hostname and the number of slots allocated in that host. |
MPI_START_NHOSTS |
Number of hosts allocated (number of lines in the MPI_START_HOSTFILE file) |
MPI_START_NSLOTS |
Number of slots allocated (number of lines in the MPI_START_MACHINEFILE file) |
MPI_START_NSLOTS_PER_HOST |
Number of slots allocated in each host. If the slots are not equally distributed, put here the maximum number of slots available on every host. |
Execution environment plugins
A Execution Environment plugin is responsible for setting up the environment for the specified execution environment (MPI) version. It has to provide the following functions:
- mpi_start
- This function is called by the mpi-start core to start a parallel job. This function consists (at least) of the following steps:
- setup parallel environment if necessary (e.g. start daemons)
- activate the pre-run hooks (call mpi_start_pre_run_hook)
- call mpi_exec to start the parallel job
- activate the post-run hooks (call mpi_start_post_run_hook)
- shutdown parallel environment if necessary (e.g. shutdown daemons)
A generic_mpistart.sh file contains this core functionality for development of new plugins.
- This function is called by the mpi-start core to start a parallel job. This function consists (at least) of the following steps:
- mpi_exec
- This function actually perform the execution of the parallel job (mpirun/mpiexec). This functionality has been separated from the setup/shutdown phase to allow usage of the parallel environment in the pre-/post-run hooks.
Several internal variables may change the behavior of the execution environment:
Variable |
Meaning |
MPI_START_SCHEDULER |
Name of the scheduler plugin loaded |
MPI_START_MPI_PREFIX |
Path of the execution environment |
In order to provide a different environment for the user application, MPI-Start uses a wrapper that is executed as a different process. In this wrapper, the variables defined in the MPI_START_ENV_VARIABLES are exported to the environment. The use of the wrapper can be controlled with the functions:
mpi_start_export_variable var_name [value] Adds the variable var_name to MPI_START_ENV_VARIABLES and to the wrapper script, if value is defined, set it also in the wrapper.
mpi_start_execute_wrapper cmd_line: Executes the specified command line in cmd_line. If MPI_START_DO_NOT_USE_WRAPPER is set to 1 it will not use a wrapper and therefore the environment variables may not be defined.
Clean up
When mpi-start finishes its execution the clean_up function is called. This will remove any temporary files created by MPI-Start by default (e.g. host files) and any other file appended to the MPI_START_CLEANUP_FILES variable.
Hooks
See Hooks Frameworkfor a complete description of the hooks mechanism and how to develop new hooks.
Internals
Global Configuration Variables
Variable |
Default |
Description |
MPI_START_DUMMY_SCHEDULER |
1 |
Enables or disables the dummy scheduler. |
I2G_MPI_START_KEEP_FILES |
0 |
Enables or disables the removal of temporary files at the end of execution. |
I2G_MPI_START_FULL_TRACE |
- |
Enables or disables full trace of mpi-start. |
MPI_START_DO_NOT_USE_WRAPPER |
- |
Enables or disables the use of a wrapper for the executable |
MPI_START_SOCKETS |
- |
Number of sockets in the host (if not defined, mpi-start tries to detect them). |
MPI_START_COREPERSOCKET |
- |
Number of cores per socket in the host (if not defined, mpi-start tries to detect them). |
MPI_START_COREPERSOCKET |
- |
Number of cores per socket in the host (if not defined, mpi-start tries to detect them). |
I2G_MPI_START_ENABLE_TESTING |
- |
If equal to "TEST", then do not call the main function. Used for sourcing the mpi-start file. |
Scheduler plugin variables
Variable |
Description |
MPI_START_SCHEDULER |
Name of the scheduler. |
MPI_START_HOSTFILE |
File containing one line per slot available. |
MPI_START_MACHINEFILE |
File containing one line per host available. |
MPI_START_HOST_SLOTS_FILE |
File containing one line with a name of host, and the number of slots available in that host. |
MPI_START_NP |
Total number of processors to use. |
MPI_START_NPHOST |
Number of processes per host, may be undefined if slot allocation is used. |
MPI Execution Variables
Variable |
Description |
MPIEXEC |
Defined by each flavour, mpiexec executable |
MPI_GLOBAL_PARAMS |
Global parameters for mpiexec |
MPI_LOCAL_PARAMS |
Local parameters for mpiexec |
MPI_START_SSH_AGENT |
User (or system) specified ssh agent (used mostly by condor). |
MPI_START_DISABLE_LRMS_INTEGRATION |
If set to "yes", do not use any LRMS integration available in the MPI flavour. |
MPI_MPICH2_DISABLE_HYDRA |
If set to 1, disable the use of hydra launcher for mpich2. |
OSC_MPIEXEC |
Set to 1 if OSC mpiexec is found. |
HYDRA_MPIEXEC |
Set to 1 if hydra mpiexec is found. |
OPENMPI_VERSION_MAJOR |
Set to Open MPI major version. |
OPENMPI_VERSION_MINOR |
Set to Open MPI minor version. |
OPENMPI_VERSION_RELEASE |
Set to Open MPI release version. |