= MPI-Utils =

<<TableOfContents>>

== yaim Configuration ==

=== Overview ===
Configuration is necessary on both the CE and WNs in order to support and advertise MPI correctly (see SiteConfig for details). This is performed by the gLite YAIM module glite-yaim-mpi which should be run on both the CE and WNs.

=== Install packages ===
The YAIM module for MPI configuration is now available as part of the gLite distribution (http://glitesoft.cern.ch/EGEE/gLite/R3.1/glite-MPI_utils/sl4/i386/RPMS.release/glite-yaim-mpi-0.1.6-3.noarch.rpm).
If you install the meta-package glite-MPI_utils-3.1.1-0.i386.rpm it will pull in this YAIM module as well as other packages needed for MPI.

To install this using YUM, add a file (e.g. glite-MPI_utils.repo) with the following contents to your YUM repositories directory (/etc/yum/repos.d):

{{{
[glite-MPI_utils]
name=glite 3.1 MPI
enabled=1
gpgcheck=0
baseurl=http://glitesoft.cern.ch/EGEE/gLite/R3.1/glite-MPI_utils/sl4/i386/
}}}

Then run this: 

{{{
yum install glite-MPI_utils
}}}

=== Configure MPI in site-info.def ===
Individual "flavours" of MPI are enabled by setting the associated variable to "yes". For example, to enable MPICH, add the following:
{{{#!highlight sh
MPI_MPICH_ENABLE="yes"
}}}

You may set the path and version for a flavour of MPI as follows:
{{{#!highlight sh
MPI_MPICH_PATH="/opt/mpich-1.2.7p1/"
MPI_MPICH_VERSION="1.2.7p1"
}}}

The remaining variables are:
|| {{{MPI_SHARED_HOME}}} || set this to "yes" if you have a shared home area between WNs.||
|| {{{MPI_SSH_HOST_BASED_AUTH}}} || ||
|| {{{MPI_<FLAVOUR>_MPIEXEC}}} || If you are using OSC mpiexec with mpich, set this to the location of the mpiexec program, e.g. "/usr/bin/mpiexec" ||
|| {{{MPI_MPI_START}}} || Location of mpi-start if not installed in standard location /opt/i2g/bin/mpi-start ||

Here is an example configuration:
{{{#!highlight sh
#----------------------------------
# MPI-related configuration:
#----------------------------------
# Several MPI implementations (or "flavours") are available.
# If you do NOT want a flavour to be installed/configured, set its variable
# to "no". Else, set it to "yes" (default). If you want to use an
# already installed version of an implementation, set its "_PATH" and
# "_VERSION" variables to match your setup (examples below).
#
# NOTE 1: the CE_RUNTIMEENV will be automatically updated in the file
# functions/config_mpi, so that the CE advertises the MPI implementations
# you choose here - you do NOT have to change it manually in this file.
# It will become something like this:
#
#   CE_RUNTIMEENV="$CE_RUNTIMEENV
#              MPI_MPICH
#              MPI_MPICH2
#              MPI_OPENMPI
#              MPI_LAM"
#
# NOTE 2: it is currently NOT possible to configure multiple concurrent
# versions of the same implementations (e.g. MPICH-1.2.3 and MPICH-1.2.7)
# using YAIM. Customize "/opt/glite/yaim/functions/config_mpi" file
# to do so.

MPI_MPICH_ENABLE="yes"
MPI_MPICH2_ENABLE="yes"
MPI_OPENMPI_ENABLE="yes"
MPI_LAM_ENABLE="yes"

#---
# Example for using an already installed version of MPI.
# Setting "_PATH" and "_VERSION" variables will prevent YAIM
# from downloading and installing the gLite-provided packages.
# Just fill in the path to its current installation (e.g. "/usr")
# and which version it is (e.g. "6.5.9").
#---
MPI_MPICH_PATH="/opt/mpich-1.2.7p1/"
MPI_MPICH_VERSION="1.2.7p1"
MPI_MPICH2_PATH="/opt/mpich2-1.0.4/"
MPI_MPICH2_VERSION="1.0.4"
MPI_OPENMPI_VERSION="1.1"
MPI_LAM_VERSION="7.1.2"

# If you do NOT provide a shared home, set $MPI_SHARED_HOME to "no" (default).
#
MPI_SHARED_HOME="yes"

#
# If you do NOT have SSH Hostbased Authentication between your WNs, set the below
# variable to "no" (default). Else, set it to "yes".
#
MPI_SSH_HOST_BASED_AUTH="no"

#
# If you provide an 'mpiexec' for MPICH or MPICH2, please state the full path to
# that file here (http://www.osc.edu/~pw/mpiexec/index.php). Else, leave empty.
#
#MPI_MPICH_MPIEXEC="/usr/bin/mpiexec"
MPI_MPICH_MPIEXEC="/usr/bin/mpiexec"
}}}

You should also set the variable CE_BATCH_SYS to "pbs" not "torque":

{{{#!highlight sh
# Jobmanager specific settings

# If WNs and CE share /home, set this:
JOB_MANAGER=pbs

# If WNs and CE do not use the same shared /home (although WNs may use shared home), set this:
#JOB_MANAGER=lcgpbs

# If you are supporting MPI, set this:
CE_BATCH_SYS=pbs
}}}

This should cause the variable GlueCEInfoLRMSType to be set correctly in  /opt/lcg/var/gip/lcg-info-static-ce.conf and /opt/lcg/var/gip/ldif/static-file-CE.ldif



=== Configure CE ===
{{{/opt/glite/yaim/bin/yaim -c -s site-info.def  -n MPI_CE}}}

=== Configure WN ===
For a Torque worker node:
{{{/opt/glite/yaim/bin/yaim -c -s site-info.def  -n MPI_WN -n glite-WN  -n TORQUE_client}}}

=== Testing ===
You can do some basic tests by logging in on a WN as a pool user and running the following:

{{{
[dte056@cagnode48 dte056]$ env|grep MPI_
}}}

You should see something like this:
{{{
MPI_MPICC_OPTS=-m32
MPI_SSH_HOST_BASED_AUTH=yes
MPI_OPENMPI_PATH=/opt/openmpi/1.1
MPI_LAM_VERSION=7.1.2
MPI_MPICXX_OPTS=-m32
MPI_LAM_PATH=/usr
MPI_OPENMPI_VERSION=1.1
MPI_MPIF77_OPTS=-m32
MPI_MPICH_VERSION=1.2.7
MPI_MPIEXEC_PATH=/opt/mpiexec-0.80
MPI_MPICH2_PATH=/opt/mpich2-1.0.4
MPI_MPICH2_VERSION=1.0.4
I2G_MPI_START=/opt/i2g/bin/mpi-start
MPI_MPICH_PATH=/opt/mpich-1.2.7p1
}}}

You can also try submitting a job to your site using the instructions found via the page JobSubmission.

SAM tests are run daily to verify MPI functionality. You can see the results by going to the SAM tests homepage (https://lxn1181.cern.ch:8443/sam/sam.py) and selecting to display the test "mpi-all" for CEs in the dteam VO.