MPI

OpenMPI

If you submit your jobs with sbatch and your job loads an OpenMPI module, you can issue your mpirun commands without an option describing the resources to be used (such as -np, -H, -npersocket, etc.). OpenMPI has indeed been configured to retrieve this information directly from Slurm.

In order to optimize the performance of your jobs, it is therefore advisable to configure the options for booking resources with Slurm. If you do not give a sufficiently precise definition of the reserved resources, your tasks may be dispatched to different nodes (via the queue cpucourt and if the cluster is widely used) and therefore you will lose performance. It is therefore recommended to use options that specify the reservation, such as:

--nodes=1  
--ntasks=4
--ntasks-per-node=4
--ntasks-per-socket=4

Here, we reserve 4 cores on the same node and the same processor. For information, the computation nodes (queues cpucourt and cpulong) are equipped with 2 processors (socket), each having 20 cores.

Here is an example of a job declaration with Slurm where the code will run on 2 nodes (with 10 cores used on each socket):

#!/bin/bash
#SBATCH --job-name=check_mpi
#SBATCH --output=check_mpi.txt

#SBATCH --time=10:00
#SBATCH --account=project_name

#SBATCH --ntasks=40
#SBATCH --ntasks-per-node=20
#SBATCH --nodes=2
#SBATCH --ntasks-per-socket=10

module purge
module load openmpi/4.0.4

mpirun program.mpi

MVAPICH

It is recommended to use module mvapich2/2.3.3-mlnx (loaded after gnu7). Here is a job example:

#!/bin/bash
#SBATCH --job-name=myJob
#SBATCH --nodes=5
#SBATCH --ntasks=100
#SBATCH --ntasks-per-node=20
#SBATCH --account=my_account
#SBATCH --time=01:00:00

module purge
module load gnu7/7.3.0
module load mvapich2/2.3.3-mlnx

SLURM_HOSTLIST=$(scontrol show hostnames|paste -d, -s)
export MV2_ENABLE_AFFINITY=0
export MV2_CPU_BINDING_LEVEL=numanode
export MV2_SHOW_CPU_BINDING=1
export MV2_USE_SHARED_MEM=on

mpiexec.hydra -genvall -hosts ${SLURM_HOSTLIST} -n ${SLURM_NTASKS} -ppn ${SLURM_NTASKS_PER_NODE} /home/user/myExecutable

Intel MPI

To launch a job that uses an Intel command, you must use the licenses option in your Slurm job:

#SBATCH --licenses=intel:1

If there is no free license when submitting your job, it will be kept pending until a license is available. To check how many licenses are currently available, use:

scontrol show lic

Job example:

#!/bin/bash
#SBATCH --job-name=test_mpi
#SBATCH --output=mpi.txt
#SBATCH --licenses=intel:1
#SBATCH --ntasks=4
#SBATCH --time=10:00
#SBATCH --account=projectname

module purge
module load intel

mpirun -bootstrap slurm -n 4 /home/username/jobs/my_code.mpi

Click here to read the official Intel MPI documentation.