= Multicore and MPI-Start =
MPI-Start uses by default the exact number of processes as slots allocated by the batch system, independently of the node they are located. While this behavior is fine for simple MPI applications, hybrid applications need better control of the location and number of processes to use while running.

== MPI-Start 1.0.4 ==
Since mpi-start 1.0.4, users are able to select the following placement policies:

 * use a process per allocated slot (default),
 * specify the total number of processes to start, or
 * specify the number of processes to start at each node

Being the last one the most appropriate for hybrid applications. The typical use case would be starting 1 single process per node and at each node start as many threads as allocated slots are available within that node.

=== Total Number of Processes ===
In order to specify the total number of processes to use, mpi-start provides the `-np` option or the `I2G_MPI_NP` environment variable. Both methods are equivalent.

For example to start an application using exactly 12 processes (independently of the allocated slots):

{{{#!highlight sh

export I2G_MPI_APPLICATION=myapp
export I2G_MPI_APPLICATION_ARGS=arg1 arg2
export I2G_MPI_TYPE=openmpi
export I2G_MPI_NP=12

$I2G_MPI_START
}}}
or using the newer syntax:

{{{
mpi-start -t openmpi -np 12 myapp arg1 arg2
}}}
The location of the processes is determined by the MPI implementation that you are using. Normally a round-robin policy is followed!

=== Processes per Host ===
{{{#!wiki caution
'''Exclusive Execution'''

It is convenient to run your applications without sharing the Worker Nodes with other applications. Otherwise they can suffer for bad performance. In order to do so, in EMI CREAM you can use `WholeNodes=True` in your JDL.
}}}
As with the total number of processes, both command line switches and environment may be used. In this case there are two different options:

 * Specify the number of processes per host, with the `-npnode` command line switch or the `I2G_MPI_PER_NODE` variable.
 * Use the `-pnode` or `I2G_MPI_SINGLE_PROCESS` to start only 1 process per host (This is equivalent to `-npnode 1` or `I2G_MPI_PER_NODE=1`

For example, to start an application using a 2 processes per node (independently of the allocated slots):

{{{#!highlight sh

export I2G_MPI_APPLICATION=myapp
export I2G_MPI_APPLICATION_ARGS=arg1 arg2
export I2G_MPI_TYPE=openmpi
export I2G_MPI_PER_NODE=2

$I2G_MPI_START
}}}
or:

{{{
mpi-start -t openmpi -npnode 2 myapp arg1 arg2
}}}
In the case of using a single processes per host, the Open MP hook (defining `MPI_USE_OMP=1`) may be also used in order to let MPI-Start configure the number of OpenMP threads that should be started:

{{{#!highlight sh

export I2G_MPI_APPLICATION=myapp
export I2G_MPI_APPLICATION_ARGS=arg1 arg2
export I2G_MPI_TYPE=openmpi
export I2G_MPI_SINGLE_PROCESS=1
export MPI_USE_OMP=1

$I2G_MPI_START
}}}
or:

{{{
mpi-start -t openmpi -pnode -d MPI_USE_OMP=1 myapp arg1 arg2
}}}

This is the typical case of hybrid MPI/OpenMP application. In the figure, an application executing in two different hosts with 2-quadcore CPUs is shown. Each host will have a MPI process (named p0 and p1) and one thread for each of the CPU cores available (from t0 to t7).

{{attachment:pnode.png}}

MPI-Start will define the OpenMP threads to 8 and start only one process at each host.

== MPI-Start 1.0.5 ==
MPI-Start 1.0.5 introduces more ways of controlling the processes placement. The previous options are supported, and 2 new more are available:

 * Start one process per CPU socket (threads would be started per each core in the CPU), with the `-psocket` option
 * Start one process per core (no threads), with the `-pcore` option.