Functional Description

MPI-Utils is a meta package that provide mpi-start and a yaim mpi module in order to ease the installation of MPI support in the nodes.

Installation

MPI-Utils is a meta package that depends on MPI-Start and the yaim mpi module for configuration of the MPI support in the CE and WN. Administrators must install a MPI implementation and configure it at the site. Most Linux distributions provide ready to use packages for Open MPI and MPICH implmentations.

MPI-Utils can be installed from the EMI repositories, and should be as easy as:

yum install glite-mpi

In the WN, a MPI implementation must be installed also. Open MPI is recommended (the devel package allows users to compile their applications):

yum install openmpi openmpi-devel

devel packages and compilers

The devel packages of the MPI packages do not include the compiler as dependency! You should install it also if you want to support the compilation of MPI applications (e.g. gcc, gcc-gfortran, gcc-g++)

Configuration

Configuration is necessary on both the CE and WNs in order to support and advertise MPI correctly (see Site Configuration for details). This is performed by the yaim MPI module which should be run on both types of nodes.

Configure MPI in site-info.def

Individual "flavours" of MPI are enabled by setting the associated variable to "yes". For example, to enable Open MPI, add the following:

   1 MPI_OPENMPI_ENABLE="yes"

The version of the MPI implementation must also be specified for the CE configuration with the variable MPI_<FLAVOUR>_VERSION, for Open MPI version 1.3:

   1 MPI_OPENMPI_VERSION="1.3"

You may also set the path to non default locations for a flavour of MPI with the variable MPI_<FLAVOUR>_PATH, but should not be used unless necessary:

   1 MPI_OPENMPI_PATH="/opt/openmpi-1.3/"

Possible flavours are:

OPENMPI, for Open MPI
MPICH for MPICH-1
MPICH2 for MPICH-2
LAM for LAM-MPI

The variables are:

Variable	Mandatory	Value
`MPI_<FLAVOUR>_ENABLE`	YES	set to "yes" if you want to enable the <flavour>
`MPI_<FLAVOUR>_VERSION`	YES	set to the supported version of the <flavour>
`MPI_<FLAVOUR>_PATH`	NO	set to the path of supported version of the <flavour>, usually is automatically detected by the yaim WN plugin
`MPI_SHARED_HOME`	NO	set this to "yes" if you have a shared home area between WNs.
`MPI_SSH_HOST_BASED_AUTH`	NO	set it to "yes" if you have SSH based authentication between WNs
`MPI_<FLAVOUR>_MPIEXEC`	NO	If you are using OSC mpiexec (only in PBS/Torque sites), set this to the location of the mpiexec program, e.g. "/usr/bin/mpiexec"
`MPI_MPI_START`	NO	Location of mpi-start if not installed in standard location /opt/mpi-start/bin/mpi-start
`MPI_DEFAULT_FLAVOUR`	NO	Set it to the default flavour for your site, if undefined, the first defined flavour will be used
`MPI_SUBMIT_FILTER`	NO	Set this to "yes" to configure the submit filter for torque batch system that enables the submission of parallel jobs

Here is an example configuration:

   1 #----------------------------------
   2 # MPI-related configuration:
   3 #----------------------------------
   4 # Several MPI implementations (or "flavours") are available.
   5 # If you do NOT want a flavour to be installed/configured, set its variable
   6 # to "no". Else, set it to "yes" (default). If you want to use an
   7 # already installed version of an implementation, set its "_PATH" and
   8 # "_VERSION" variables to match your setup (examples below).
   9 #
  10 # NOTE 1: the CE_RUNTIMEENV will be automatically updated in the file
  11 # functions/config_mpi, so that the CE advertises the MPI implementations
  12 # you choose here - you do NOT have to change it manually in this file.
  13 # It will become something like this:
  14 #
  15 #   CE_RUNTIMEENV="$CE_RUNTIMEENV
  16 #              MPI_MPICH
  17 #              MPI_MPICH2
  18 #              MPI_OPENMPI
  19 #              MPI_LAM"
  20 #
  21 # NOTE 2: it is currently NOT possible to configure multiple concurrent
  22 # versions of the same implementations (e.g. MPICH-1.2.3 and MPICH-1.2.7)
  23 # using YAIM. Customize "/opt/glite/yaim/functions/config_mpi" file
  24 # to do so.
  25 
  26 MPI_MPICH_ENABLE="yes"
  27 MPI_MPICH_VERSION="1.2.7p1"
  28 
  29 MPI_MPICH2_ENABLE="yes"
  30 MPI_MPICH2_VERSION="1.0.4"
  31 
  32 MPI_OPENMPI_ENABLE="yes"
  33 MPI_OPENMPI_VERSION="1.1"
  34 
  35 MPI_LAM_ENABLE="yes"
  36 MPI_LAM_VERSION="7.1.2"
  37 
  38 # set Open MPI as default flavour
  39 MPI_DEFAULT_FLAVOUR=OPENMPI
  40 
  41 # Enable the configuration of parallel jobs in torque
  42 MPI_SUBMIT_FILTER="yes"
  43 
  44 
  45 #---
  46 # Example for using an already installed version of MPI.
  47 # Setting "_PATH" and "_VERSION" variables will prevent YAIM
  48 # from downloading and installing the gLite-provided packages.
  49 # Just fill in the path to its current installation (e.g. "/usr")
  50 # and which version it is (e.g. "6.5.9").
  51 #---
  52 MPI_MPICH_PATH="/opt/mpich-1.2.7p1/"
  53 MPI_MPICH2_PATH="/opt/mpich2-1.0.4/"
  54 
  55 # If you do NOT provide a shared home, set $MPI_SHARED_HOME to "no" (default).
  56 #
  57 MPI_SHARED_HOME="yes"
  58 
  59 #
  60 # If you do NOT have SSH Hostbased Authentication between your WNs, set the below
  61 # variable to "no" (default). Else, set it to "yes".
  62 #
  63 MPI_SSH_HOST_BASED_AUTH="no"
  64 
  65 
  66 # If you use Torque as batch system, you may want to let the yaim plugin
  67 # configure a submit filter for you. Uncomment the following line to do so
  68 # MPI_SUBMIT_FILTER="yes"
  69 
  70 #
  71 # If you provide an 'mpiexec' for MPICH or MPICH2, please state the full path to
  72 # that file here (http://www.osc.edu/~pw/mpiexec/index.php). Else, leave empty.
  73 #
  74 #MPI_MPICH_MPIEXEC="/usr/bin/mpiexec"
  75 MPI_MPICH_MPIEXEC="/usr/bin/mpiexec"

Configure CE

The profile for configuring the CE is MPI_CE.

/opt/glite/yaim/bin/yaim -c -s site-info.def  -n MPI_CE -n <other_ce_profiles>

Batch system and MPI

The batch system may need extra configuration for the submission of MPI jobs. In PBS, you may use the automatic creation of the submit filter with the MPI_SUBMIT_FILTER variable. In the case of SGE you need to configure a parallel environment. Check the documentation of your batch system for any further details.

Configure WN

For a worker node:

/opt/glite/yaim/bin/yaim -c -s site-info.def  -n MPI_WN -n <other_WN_profiles>

Testing

You can do some basic tests by logging in on a WN as a pool user and running the following:

[dte056@cagnode48 dte056]$ env|grep MPI_

You should see something like this:

MPI_MPICC_OPTS=-m32
MPI_SSH_HOST_BASED_AUTH=yes
MPI_OPENMPI_PATH=/opt/openmpi/1.1
MPI_LAM_VERSION=7.1.2
MPI_MPICXX_OPTS=-m32
MPI_LAM_PATH=/usr
MPI_OPENMPI_VERSION=1.1
MPI_MPIF77_OPTS=-m32
MPI_MPICH_VERSION=1.2.7
MPI_MPIEXEC_PATH=/opt/mpiexec-0.80
MPI_MPICH2_PATH=/opt/mpich2-1.0.4
MPI_MPICH2_VERSION=1.0.4
I2G_MPI_START=/opt/mpi-start/bin/mpi-start
MPI_MPICH_PATH=/opt/mpich-1.2.7p1

You can also try submitting a job to your site, please read MPI-Start user documentation

eciencia: Middleware/MpiStart/MpiUtils (last edited 2012-02-17 14:34:36 by enol)

-  ⇤ ← Revision 2 as of 2011-02-17 10:16:21 → 
  Size: 5841
  Editor: enol
  Comment:
+   ← Revision 16 as of 2011-06-13 07:19:21 → ⇥
  Size: 7329
  Editor: enol
  Comment:
-Deletions are marked like this.
+Additions are marked like this.
 Line 1:
-= MPI-Utils =
-Line 5:
+Line 3:
-== yaim Configuration ==
+= Functional Description =
MPI-Utils is a meta package that provide [[../ | mpi-start]] and a yaim mpi module in order to ease the installation of MPI support in the nodes.
-Line 7:
+Line 6:
-=== Overview ===
Configuration is necessary on both the CE and WNs in order to support and advertise MPI correctly (see SiteConfig for details). This is performed by the gLite YAIM module glite-yaim-mpi which should be run on both the CE and WNs.
+= Installation =
MPI-Utils is a meta package that depends on MPI-Start and the yaim mpi module for configuration of the MPI support in the CE and WN. Administrators must install a MPI implementation and configure it at the site. Most Linux distributions provide ready to use packages for Open MPI and MPICH implmentations.
-Line 10:
+Line 9:
-=== Install packages ===
The YAIM module for MPI configuration is now available as part of the gLite distribution (http://glitesoft.cern.ch/EGEE/gLite/R3.1/glite-MPI_utils/sl4/i386/RPMS.release/glite-yaim-mpi-0.1.6-3.noarch.rpm).
If you install the meta-package glite-MPI_utils-3.1.1-0.i386.rpm it will pull in this YAIM module as well as other packages needed for MPI.

To install this using YUM, add a file (e.g. glite-MPI_utils.repo) with the following contents to your YUM repositories directory (/etc/yum/repos.d):
+MPI-Utils can be installed from the EMI repositories, and should be as easy as:
-Line 17:
+Line 12:
-[glite-MPI_utils]
name=glite 3.1 MPI
enabled=1
gpgcheck=0
baseurl=http://glitesoft.cern.ch/EGEE/gLite/R3.1/glite-MPI_utils/sl4/i386/
+yum install glite-mpi
-Line 24:
+Line 15:
-Then run this:
+In the WN, a MPI implementation must be installed also. Open MPI is recommended (the devel package allows users to compile their applications):
-Line 27:
+Line 18:
-yum install glite-MPI_utils
+yum install openmpi openmpi-devel
-Line 30:
+Line 21:
-=== Configure MPI in site-info.def ===
Individual "flavours" of MPI are enabled by setting the associated variable to "yes". For example, to enable MPICH, add the following:
{{{#!highlight sh
MPI_MPICH_ENABLE="yes"
+{{{#!wiki caution
'''devel packages and compilers'''

The devel packages of the MPI packages do not include the compiler as dependency! You should install it also if you want to support the compilation of MPI applications (e.g. gcc, gcc-gfortran, gcc-g++)
-Line 36:
+Line 27:
-You may set the path and version for a flavour of MPI as follows:
+= Configuration =

Configuration is necessary on both the CE and WNs in order to support and advertise MPI correctly (see [[../SiteConfiguration | Site Configuration]] for details). This is performed by the yaim MPI module which should be run on both types of nodes.

== Configure MPI in site-info.def ==

Individual "flavours" of MPI are enabled by setting the associated variable to "yes". For example, to enable Open MPI, add the following:
-Line 38:
+Line 36:
-MPI_MPICH_PATH="/opt/mpich-1.2.7p1/"
MPI_MPICH_VERSION="1.2.7p1"
+MPI_OPENMPI_ENABLE="yes"
-Line 42:
+Line 39:
-The remaining variables are:
|| {{{MPI_SHARED_HOME}}} || set this to "yes" if you have a shared home area between WNs.||
|| {{{MPI_SSH_HOST_BASED_AUTH}}} || ||
|| {{{MPI_<FLAVOUR>_MPIEXEC}}} || If you are using OSC mpiexec with mpich, set this to the location of the mpiexec program, e.g. "/usr/bin/mpiexec" ||
|| {{{MPI_MPI_START}}} || Location of mpi-start if not installed in standard location /opt/i2g/bin/mpi-start ||
+The version of the MPI implementation must also be specified for the CE configuration with the variable MPI_<FLAVOUR>_VERSION, for Open MPI version 1.3:
{{{#!highlight sh
MPI_OPENMPI_VERSION="1.3"
}}}

You may also set the path to non default locations for a flavour of MPI with the variable MPI_<FLAVOUR>_PATH, but should not be used unless necessary:
{{{#!highlight sh
MPI_OPENMPI_PATH="/opt/openmpi-1.3/"
}}}

Possible flavours are:
 * OPENMPI, for Open MPI
 * MPICH for MPICH-1
 * MPICH2 for MPICH-2
 * LAM for LAM-MPI

The variables are:

|| Variable         || Mandatory || Value ||
|| `MPI_<FLAVOUR>_ENABLE` || YES || set to "yes" if you want to enable the <flavour> ||
|| `MPI_<FLAVOUR>_VERSION` || YES || set to the supported version of the <flavour> ||
|| `MPI_<FLAVOUR>_PATH` || NO || set to the path of supported version of the <flavour>, usually is automatically detected by the yaim WN plugin ||
|| `MPI_SHARED_HOME` || NO || set this to "yes" if you have a shared home area between WNs. ||
|| `MPI_SSH_HOST_BASED_AUTH` || NO || set it to "yes" if you have SSH based authentication between WNs ||
|| `MPI_<FLAVOUR>_MPIEXEC` || NO || If you are using OSC mpiexec (only in PBS/Torque sites), set this to the location of the mpiexec program, e.g. "/usr/bin/mpiexec" ||
|| `MPI_MPI_START` || NO ||Location of mpi-start if not installed in standard location /opt/mpi-start/bin/mpi-start ||
|| `MPI_DEFAULT_FLAVOUR` || NO || Set it to the default flavour for your site, if undefined, the first defined flavour will be used ||
|| `MPI_SUBMIT_FILTER` || NO || Set this to "yes" to configure the submit filter for torque batch system that enables the submission of parallel jobs ||
-Line 49:
+Line 70:
-Line 76:
+Line 98:
+MPI_MPICH_VERSION="1.2.7p1"
-Line 77:
+Line 101:
+MPI_MPICH2_VERSION="1.0.4"
-Line 78:
+Line 104:
+MPI_OPENMPI_VERSION="1.1"
-Line 79:
+Line 107:
+MPI_LAM_VERSION="7.1.2"

# set Open MPI as default flavour
MPI_DEFAULT_FLAVOUR=OPENMPI

# Enable the configuration of parallel jobs in torque
MPI_SUBMIT_FILTER="yes"
-Line 88:
+Line 124:
-MPI_MPICH_VERSION="1.2.7p1"
-Line 90:
+Line 125:
-MPI_MPICH2_VERSION="1.0.4"
MPI_OPENMPI_VERSION="1.1"
MPI_LAM_VERSION="7.1.2"
-Line 104:
+Line 136:
+# If you use Torque as batch system, you may want to let the yaim plugin
# configure a submit filter for you. Uncomment the following line to do so
# MPI_SUBMIT_FILTER="yes"
-Line 112:
+Line 149:
-You should also set the variable CE_BATCH_SYS to "pbs" not "torque":
+== Configure CE ==
-Line 114:
+Line 151:
-{{{#!highlight sh
# Jobmanager specific settings

# If WNs and CE share /home, set this:
JOB_MANAGER=pbs

# If WNs and CE do not use the same shared /home (although WNs may use shared home), set this:
#JOB_MANAGER=lcgpbs

# If you are supporting MPI, set this:
CE_BATCH_SYS=pbs
+The profile for configuring the CE is MPI_CE. 
{{{
/opt/glite/yaim/bin/yaim -c -s site-info.def  -n MPI_CE -n <other_ce_profiles>
-Line 127:
+Line 156:
-This should cause the variable GlueCEInfoLRMSType to be set correctly in  /opt/lcg/var/gip/lcg-info-static-ce.conf and /opt/lcg/var/gip/ldif/static-file-CE.ldif
+{{{#!wiki caution
'''Batch system and MPI'''
-Line 129:
+Line 159:
+The batch system may need extra configuration for the submission of MPI jobs. In PBS, you may use the automatic creation of the submit filter with the `MPI_SUBMIT_FILTER` variable. In the case of SGE you need to configure a parallel environment. Check the documentation of your batch system for any further details.
}}}
-Line 130:
+Line 162:
+== Configure WN ==
For a worker node: 
{{{
/opt/glite/yaim/bin/yaim -c -s site-info.def  -n MPI_WN -n <other_WN_profiles>
}}}
-Line 131:
+Line 168:
-=== Configure CE ===
{{{/opt/glite/yaim/bin/yaim -c -s site-info.def  -n MPI_CE}}}

=== Configure WN ===
For a Torque worker node:
{{{/opt/glite/yaim/bin/yaim -c -s site-info.def  -n MPI_WN -n glite-WN  -n TORQUE_client}}}

=== Testing ===
+== Testing ==
-Line 144:
+Line 174:
+You should see something like this:
-Line 145:
+Line 176:
-You should see something like this:
-Line 159:
+Line 189:
-I2G_MPI_START=/opt/i2g/bin/mpi-start
+I2G_MPI_START=/opt/mpi-start/bin/mpi-start
-Line 163:
+Line 193:
-You can also try submitting a job to your site using the instructions found via the page JobSubmission.

SAM tests are run daily to verify MPI functionality. You can see the results by going to the SAM tests homepage (https://lxn1181.cern.ch:8443/sam/sam.py) and selecting to display the test "mpi-all" for CEs in the dteam VO.
+You can also try submitting a job to your site, please read [[../UserDocumentation | MPI-Start user documentation]]

Quick Links

Search Wiki

Page Tools