MPI

OpenMPI

If you submit your jobs with sbatch and your job loads an OpenMPI module, you can issue your mpirun commands without an option describing the resources to be used (such as -np, -H, -npersocket, etc.). OpenMPI has indeed been configured to retrieve this information directly from Slurm.

In order to optimize the performance of your jobs, it is therefore advisable to configure the options for booking resources with Slurm. If you do not give a sufficiently precise definition of the reserved resources, your tasks may be dispatched to different nodes (via the queue cpucourt and if the cluster is widely used) and therefore you will lose performance. It is therefore recommended to use options that specify the reservation, such as:

--nodes=1  
--ntasks=4
--ntasks-per-node=4 
--ntasks-per-socket=4 

Here, we reserve 4 cores on the same node and the same processor. For information, the computation nodes (queues cpucourt and cpulong) are equipped with 2 processors (socket), each having 20 cores.

Here is an example of a job declaration with Slurm where the code will run on 2 nodes (with 10 cores used on each socket):

#!/bin/bash
#SBATCH --job-name=check_mpi
#SBATCH --output=check_mpi.txt

#SBATCH --time=10:00
#SBATCH --account=project_name

#SBATCH --ntasks=40 
#SBATCH --ntasks-per-node=20
#SBATCH --nodes=2
#SBATCH --ntasks-per-socket=10

module purge
module load openmpi/4.1.2

mpirun program.mpi

MVAPICH

It is recommended to use module mvapich2/2.3.3-mlnx (loaded after gnu7). Here is a job example:

#!/bin/bash
#SBATCH --job-name=myJob
#SBATCH --nodes=5
#SBATCH --ntasks=100
#SBATCH --ntasks-per-node=20
#SBATCH --account=my_account
#SBATCH --time=01:00:00

 module purge
 module load gnu7/7.3.0
 module load mvapich2/2.3.3-mlnx

 SLURM_HOSTLIST=$(scontrol show hostnames|paste -d, -s)
 export MV2_ENABLE_AFFINITY=0
 export MV2_CPU_BINDING_LEVEL=numanode
 export MV2_SHOW_CPU_BINDING=1
 export MV2_USE_SHARED_MEM=on

 mpiexec.hydra -genvall -hosts ${SLURM_HOSTLIST} -n ${SLURM_NTASKS} -ppn ${SLURM_NTASKS_PER_NODE} /home/user/myExecutable

Intel MPI

Job example:

#!/bin/bash
#SBATCH --job-name=test_mpi
#SBATCH --output=mpi.txt
#SBATCH --ntasks=4
#SBATCH --time=10:00
#SBATCH --account=projectname
#SBATCH --constraint=intel

module purge
module load intel

mpirun -bootstrap slurm -n $SLURM_NTASKS /home/username/jobs/my_code.mpi

Click here to read the official Intel MPI documentation.