#pragma section-numbers 2

= MPI-Start as EMI-ES ParallelEnvironment backend =

<<TableOfContents>>

== UNICORE ==

The UNICORE EMI-ES !ParallelEnvironment is a complete implementation of the specification. The configuration is done via xml files where the mapping from the job specification to the final executable is specified.

For each mpi-flavor that needs to be supported by mpi-start an entry in the idb must be included. For example for OpenMPI:
{{{#!highlight xml

<ee:ExecutionEnvironment xmlns:ee="http://www.unicore.eu/unicore/jsdl-extensions">
    <ee:Name>OpenMPI</ee:Name>
    <ee:Version>1.5.4</ee:Version>
    <ee:Description>Runs OpenMPI job using mpi-start to start the job</ee:Description>
    <ee:ExecutableName>/usr/bin/mpi-start</ee:ExecutableName>
    <ee:CommandlineTemplate>#EXECUTABLE -t openmpi #ARGS #USERCOMMAND #USERARGS</ee:CommandlineTemplate>
    <ee:Argument>
      <ee:Name>Output</ee:Name>
      <ee:IncarnatedValue>-o </ee:IncarnatedValue>
      <ee:ArgumentMetadata>
        <ee:Type>string</ee:Type>
        <ee:Description>Write the job output to a FILE instead of to the standard output stream</ee:Description>
      </ee:ArgumentMetadata>
    </ee:Argument>
    <ee:Argument>
      <ee:Name>PostHook</ee:Name>
      <ee:IncarnatedValue>-post </ee:IncarnatedValue>
      <ee:ArgumentMetadata>
        <ee:Type>string</ee:Type>
        <ee:Description>Use the file as post hook</ee:Description>
      </ee:ArgumentMetadata>
    </ee:Argument>
    <ee:Argument>
      <ee:Name>PreHook</ee:Name>
      <ee:IncarnatedValue>-pre </ee:IncarnatedValue>
      <ee:ArgumentMetadata>
        <ee:Type>string</ee:Type>
        <ee:Description>Use the file as pre hook</ee:Description>
      </ee:ArgumentMetadata>
    </ee:Argument>
    <ee:Argument>
      <ee:Name>Error</ee:Name>
      <ee:IncarnatedValue>-e </ee:IncarnatedValue>
      <ee:ArgumentMetadata>
        <ee:Type>string</ee:Type>
        <ee:Description>Write the job error to a FILE instead of to the standard error stream</ee:Description>
      </ee:ArgumentMetadata>
    </ee:Argument>
    <ee:Option> 
      <ee:Name>Verbose</ee:Name> 
      <ee:IncarnatedValue>-v</ee:IncarnatedValue> 
      <ee:OptionMetadata> 
        <ee:Description>Enable verbose mode</ee:Description> 
      </ee:OptionMetadata> 
    </ee:Option> 
    <ee:Option> 
      <ee:Name>PerNode</ee:Name> 
      <ee:IncarnatedValue>-pnode</ee:IncarnatedValue> 
      <ee:OptionMetadata> 
        <ee:Description>Start one process per available node</ee:Description> 
      </ee:OptionMetadata> 
    </ee:Option> 
    <ee:Option> 
      <ee:Name>PerSocket</ee:Name> 
      <ee:IncarnatedValue>-psocket</ee:IncarnatedValue> 
      <ee:OptionMetadata> 
        <ee:Description>Start one process per available socket</ee:Description> 
      </ee:OptionMetadata> 
    </ee:Option> 
    <ee:Option> 
      <ee:Name>PerCore</ee:Name> 
      <ee:IncarnatedValue>-pcore</ee:IncarnatedValue> 
      <ee:OptionMetadata> 
        <ee:Description>Start one process per available core</ee:Description> 
      </ee:OptionMetadata> 
    </ee:Option> 
</ee:ExecutionEnvironment>
}}}

== ARC ==

ARC supports MPI jobs through the !RunTimeEnvironments. There are two options for integrating them with mpi-start
 1. Explicit mpi-start usage by users.
 1. mpi-start as backend of parallel jobs

=== Explicit mpi-start ===

In this use case, the only requirements is that mpi-start is available at the nodes that will execute the job. There is no need for a RuntimeEnvironment and the user must set as executable for the jobs `mpi-start`. A minimal job description is shown below:

{{{#!highlight xml

<ActivityDescription xmlns="http://www.eu-emi.eu/es/2010/12/adl">
    <ActivityIdentification>
        <Name>MPI Test job</Name>
        <Description>A test job showing the features of EMI-ES</Description>
        <Type>single</Type>
    </ActivityIdentification>
    <Application>
        <Executable>
            <Path>/usr/bin/mpi-start</Path>
            <Argument>-t</Argument>
            <Argument>openmpi</Argument>
            <Argument>-pre</Argument>
            <Argument>pre.sh</Argument>
            <Argument>cpi</Argument>
        </Executable>
        <Error>std.err</Error>
        <Output>std.out</Output>
    </Application>
    <Resources>
        <SlotRequirement>
            <NumberOfSlots>2</NumberOfSlots>
            <SlotsPerHost>2</SlotsPerHost>
            <ExclusiveExecution>true</ExclusiveExecution>
        </SlotRequirement>
    </Resources>

    <DataStaging>
        <InputFile><Name>pre.sh</Name><Source><URI>pre.sh</URI></Source></InputFile>
        <InputFile><Name>cpi.c</Name><Source><URI>cpi.c</URI></Source>
        </InputFile>
</DataStaging>
</ActivityDescription>
}}}

Note that the `-t openmpi` option is passed  explicitly to indicate to mpi-start which implementation to use.

=== mpi-start backend ===

In this case, a !RuntimeEnvironment prepares the execution with mpi-start, hiding from the user the details of the implementation. Any additional arguments to mpi-start must be passed as environment variables.

A sample runtime environment for running !OpenMPI applications would be:

{{{#!highlight sh
#!/bin/sh

export I2G_MPI_TYPE=openmpi

case "$1" in
0)
        joboption_args="mpi-start -- $joboption_args"
;;
1)
;;
2)
;;
*)
        return 1
;;
esac
}}}

The job description would look as follows:
{{{#!highlight xml
<ActivityDescription xmlns="http://www.eu-emi.eu/es/2010/12/adl">
    <ActivityIdentification>
        <Name>test job</Name>
        <Description>A test job showing the features of EMI-ES</Description>
        <Type>single</Type>
        <Annotation>test</Annotation>
    </ActivityIdentification>
    <Application>
        <Executable><Path>cpi</Path></Executable>
        <Error>std.err</Error>
        <Output>std.out</Output>
        <Environment><Name>I2G_MPI_PRE_RUN_HOOK</Name><Value>pre.sh</Value></Environment>
    </Application>
    <Resources>
        <SlotRequirement>
            <NumberOfSlots>2</NumberOfSlots>
            <SlotsPerHost>2</SlotsPerHost>
            <ExclusiveExecution>true</ExclusiveExecution>
        </SlotRequirement>
        <RuntimeEnvironment><Name>OPENMPI</Name></RuntimeEnvironment>
    </Resources>
    <DataStaging>
        <InputFile> <Name>pre.sh</Name> <Source><URI>pre.sh</URI></Source> </InputFile>
        <InputFile> <Name>cpi.c</Name> <Source><URI>cpi.c</URI></Source> </InputFile>
        <!-- Fake input file -->
        <InputFile> <Name>cpi</Name> <Source><URI>cpi</URI></Source> </InputFile>
    </DataStaging>
</ActivityDescription>
}}}

== CREAM ==

Cream does not support the !ParallelEnvironment. Jobs may be sumbmitted using `/usr/bin/mpi-start` as executable. See the previous ARC example for details.