Create your Miniconda environment (Python, R)

We strongly encourage users to create their own Python environment with the Miniconda distribution rather than using the centralized Python distributions available with the cluster modules. You can work with the latest updates faster, get performance improvements and avoid possible incompatibilities between different modules.

With Miniconda, you can create your own Python or R environment and install the packages you need there. You can even repeat the operation for different projects and thus have an environment specific to each.

Miniconda is a minimalist version of Anaconda and includes a basic Python and the Conda package manager. You can then install additional packages by following the procedure below.

Prerequisites

To use the miniconda commands, you must first load the miniconda module on the login node:

module load miniconda

Create the environment

Create the environment by giving it a path (here /workspace/$USER/my_env) and choosing its Python version (here 3.7):

conda create python=3.7 --prefix /workspace/$USER/my_env

You can also install it in another directory. For instance, you can share an environment with the other users of your project. To do so, you should install it in your shared directory in /workspace/your_group_name:

 conda create python=3.7 --prefix /workspace/your_group_name

Activate your environment

conda activate /workspace/$USER/my_env

Add a package

conda install package_name

Install R packages

You can also install the R packages you want in your Miniconda environment. To do this, you must first install R there, then the packages:

conda install -c r r r-essentials
R
install.packages("package_name", dependencies=TRUE)

Packages will be installed by default into /home/$USER/.conda/envs/r_env/lib/R/library

List your environments

conda env list

The asterisk in front of one of the listed environments indicates which one is currently in use.

Deactivate your environment

conda deactivate

Delete an environment

Here, we delete an environment in /workspace/$USER/my_env:

conda env remove -p /workspace/$USER/my_env

Job example

Put the following content into a file, named for example job.slurm

#!/bin/sh
#SBATCH --job-name=python_job_conda
#SBATCH --time=0-00:03:00
#SBATCH --account=project_name
#SBATCH --output=my_job_output.txt
#SBATCH --ntasks=1
#SBATCH --partition=cpucourt

 module purge
 module load miniconda

 conda activate /workspace/$USER/my_env
 python /workspace/$USER/jobs/my_code.py
 conda deactivate

Then submit this job by entering sbatch job.slurm on login-hpc.