Middleware/MpiStart/MpiUtils352012-02-17 14:34:36enol342012-01-25 11:01:09enol332012-01-25 11:00:43enol322012-01-20 10:47:30enol312011-11-24 15:47:34enol302011-07-18 10:38:00enol292011-06-28 08:00:43enol282011-06-28 07:59:55enol272011-06-28 07:55:41enol262011-06-28 07:55:15enol252011-06-16 14:02:45enolIn sync with pdf documentation.242011-06-16 13:48:00enol232011-06-16 13:46:24enol222011-06-16 13:37:06enol212011-06-16 13:35:19enol202011-06-14 14:09:08enol192011-06-14 14:07:43enol182011-06-14 14:06:20enol172011-06-13 07:20:03enol162011-06-13 07:19:21enol152011-06-13 07:17:00enol142011-06-07 10:04:56enol132011-06-07 10:02:04enol122011-05-19 14:09:25enol112011-05-19 14:08:02enol102011-03-30 16:02:03enol92011-03-02 14:39:40enol82011-03-02 14:38:29enol72011-02-18 08:34:20enol62011-02-18 08:33:59enol52011-02-18 08:33:40enol42011-02-17 11:46:40enol32011-02-17 10:28:44enol22011-02-17 10:16:21enol12011-02-16 15:09:10enolFunctional DescriptionMPI-Utils is a meta package that provide mpi-start and a yaim mpi module in order to ease the installation of MPI support in the nodes. InstallationMPI-Utils is a meta package that depends on MPI-Start and the yaim mpi module for configuration of the MPI support in the CE and WN. Administrators must install a MPI implementation and configure it at the site. Most Linux distributions provide ready to use packages for Open MPI and MPICH implmentations. MPI-Utils can be installed from the EMI repositories, and should be as easy as: or for EMI-2 and onwards: In the WN, a MPI implementation must be installed also. Open MPI is recommended (the devel package allows users to compile their applications): devel packages and compilers The devel packages of the MPI packages do not include the compiler as dependency! You should install it also if you want to support the compilation of MPI applications (e.g. gcc, gcc-gfortran, gcc-g++) ConfigurationConfiguration is necessary on both the CE and WNs in order to support and advertise MPI correctly. This is performed by the yaim MPI module which should be run on both types of nodes. WN ConfigurationThe yaim plugin in the WN prepares the environment for the correct execution of mpi-start. Each of the MPI flavours supported by the site must be specified setting the variable MPI_<FLAVOUR>_ENABLE
to "yes"
. For example, to enable Open MPI, add the following: Optionally, if you are using a non OS provided MPI implementation, you can define the location and version with MPI_<FLAVOUR>_VERSION
and MPI_<FLAVOUR>_PATH
. Do not use these variables if you are using the OS provided MPI implementations. For example for Open MPI version 1.3, installed at /opt/openmpi-1.3: MPI flavours that use a particular mpiexec for starting the jobs (e.g. OSC mpiexec for PBS/Torque system) may also provide in the MPI_<FLAVOUR>_MPIEXEC
the path to the binary. Do not use this variable if you are not using a different mpiexec from the one provided by the MPI implementation. Additionally, you may specify a default MPI flavour to use if non is selected for execution, with the MPI_DEFAULT_FLAVOUR
. If no default flavour is specified, the first one defined in your site-info.def will be considered as default. If you provide a shared filesystem for the execution of the applications, but it is not the path where the jobs are started, then set the variable MPI_SHARED_HOME
to "yes"
and the variable MPI_SHARED_HOME_PATH
to the the location of the shared filesystem. Do not use this variable if the application starts its execution in a shared directory (e.g. shared home), this situation should be automatically detected. If you use ssh host based authentication, set the variable MPI_SSH_HOST_BASED_AUTH
to "yes"
. SSH configuration The yaim plugin DOES NOT configure passwordless ssh between the Worker Nodes. It must be configured manually by the site admin. The MPI_SSH_HOST_BASED_AUTH
variable just sets some environment variables for the execution of the jobs. Lastly, if your use a non default location for mpi-start, set its location with the MPI_MPI_START
variable. The complete list of configuration variables for the WN is shown in the next table: Variable Mandatory Description
MPI_<FLAVOUR>_ENABLE
YES set to "yes"
if you want to enable the <flavour> |
MPI_<FLAVOUR>_VERSION
NO set to the supported version of the <flavour>, usually is automatically detected |
MPI_<FLAVOUR>_PATH
NO set to the path of supported version of the <flavour>, usually is automatically detected by the yaim WN plugin |
MPI_<FLAVOUR>_MPIEXEC
NO If you are using OSC mpiexec (only in PBS/Torque sites), set this to the location of the mpiexec program, e.g. "/usr/bin/mpiexec"
|
MPI_DEFAULT_FLAVOUR
NO Set it to the default flavour for your site, if undefined, the first defined flavour will be used |
MPI_SHARED_HOME
NO set this to "yes"
if you have a shared home area between WNs. |
MPI_SHARED_HOME_PATH
NO location of the shared area for execution of MPI applications |
MPI_SSH_HOST_BASED_AUTH
NO set it to "yes"
if you have SSH based authentication between WNs |
MPI_MPI_START
NO Location of mpi-start if not installed in standard location (/usr/bin/mpi-start
)
The profile for a worker node is MPI_WN. Use it along any other profiles you may need for your WN. ]]>CE ConfigurationAs with the WN, individual flavours of MPI are enabled by setting the MPI_<FLAVOUR>_ENABLE
associated variable to "yes"
. The version of the MPI implementation must also be specified with the variable MPI_<FLAVOUR>_VERSION
, e.g. for configuring Open MPI version 1.3:
Possible flavours are: OPENMPI for Open MPI MPICH for MPICH-1 MPICH2 for MPICH-2 LAM for LAM-MPI The use of shared homes should be announced also by setting the MPI_SHARED_HOME
to "yes"
. If you are using PBS/Torque, you can set the variable MPI_SUBMIT_FILTER
to "yes"
in order to enable the submission of parallel jobs in your system. The submit filter assumes that your Worker Nodes are correctly configured to publish in their status the ncpus
variable with the number of available slots. If that's not true in your case, you may edit the file /var/torque/submit_filter
in line 71 to fit your pbsnodes output. An example for using the np
value is commented out in the file. The complete list of configuration variables for the CE is shown in the next table: Variable Mandatory Description
MPI_<FLAVOUR>_ENABLE
YES set to "yes"
if you want to enable the <flavour> |
MPI_<FLAVOUR>_VERSION
YES set to the supported version of the <flavour>, usually is automatically detected |
MPI_START_VERSION
NO set to the available mpi-start version. If not set, the yaim plugin will try to figure out the version by checking if mpi-start is installed. |
MPI_SHARED_HOME
NO set this to "yes"
if you have a shared home area between WNs. |
MPI_SUBMIT_FILTER
NO Set this to "yes"
to configure the submit filter for torque batch system that enables the submission of parallel jobs. The configuration assumes that torque path is /var/torque
or TORQUE_VAR_DIR
variable if defined.
The profile for configuring the CE is MPI_CE. ]]>MPI_CE and other yaim profiles The MPI_CE
profile should be the first in the yaim configuration, otherwise the Glue variables will not be properly defined. This restriction may be removed in future versions. mpi-start version The yaim plugin will publish in the tags the mpi-start version if mpi-start is installed at the CE. If not installed you should define the MPI_START_VERSION
with the version available at the WNs. Batch systemBatch system and MPI The batch system may need extra configuration for the submission of MPI jobs. In PBS, you may use the automatic creation of the submit filter with the MPI_SUBMIT_FILTER
variable. In the case of SGE you need to configure a parallel environment. Check the documentation of your batch system for any further details. Submit filter for PBS/Torqueglite-yaim-mpi <= 1.1.10In the case of using MPI_SUBMIT_FILTER
to automatically create the submit filter in Torque/PBS, it assumes that the pbsnodes -a output has the "ncpus=" field in the status line correctly set. If not, please change the submit filter like shown in this diff: Reconfiguration Any changes to the submit filter will be overwritten if yaim is re-run. glite-yaim-mpi >= 1.1.11The default behaviour of the submit filter has changed in this version to use the "np=xx" parameter of the pbsnodes command output. Check the patch shown in the previous section for the changes applied. Example configurationHere is an example configuration (with both CEs and WN variables!):
TestingYou can do some basic tests by logging in on a WN as a pool user and running the following: You should see something like this: You can also try submitting a job to your site, please read MPI-Start user documentation