We strongly encourage users to create their own Python environment with the Miniconda distribution rather than using the centralized Python distributions available with the cluster modules. You can work with the latest updates faster, get performance improvements and avoid possible incompatibilities between different modules.
With Miniconda, you can create your own Python or R environment and install the packages you need there. You can even repeat the operation for different projects and thus have an environment specific to each.
Miniconda is a minimalist version of Anaconda and includes a basic Python and the Conda package manager. You can then install additional packages by following the procedure below.
Prerequisites
To use the miniconda commands, you must first load the miniconda module on the login node:
module load miniconda
Create the environment
Create the environment by giving it a name (here my_env) and choosing its Python version (here 3.7):
conda create --name my_env python=3.7
The environments you create are added to the /home/$USER/.conda/envs folder.
You can also install it in another directory. For instance, you can share an environment with the other users of your project. To do so, you should install it in your shared directory in /workspace/your_group_name:
conda create python=3.7 --prefix /workspace/your_group_name
Activate your environment
# if you gave a name to your environment: conda activate my_env # if you created it with the prefix option: conda activate /workspace/your_group_name
Add a package
conda install package_name
Install R packages
You can also install the R packages you want in your Miniconda environment. To do this, you must first install R there, then the packages:
conda install -c r r r-essentials
R
install.packages("package_name", dependencies=TRUE)
Packages will be installed by default into /home/$USER/.conda/envs/r_env/lib/R/library
List your environments
conda env list
The asterisk in front of one of the listed environments indicates which one is currently in use.
Deactivate your environment
conda deactivate
Delete an environment
Here, we delete an environment called my_env:
conda env remove --name my_env # if you created it with the prefix option: conda env remove -p /workspace/my_folder
Job example
Put the following content into a file, named for example job.slurm
#!/bin/sh #SBATCH --job-name=python_job_conda #SBATCH --time=0-00:03:00 #SBATCH --account=project_name #SBATCH --output=my_job_output.txt #SBATCH --ntasks=1 #SBATCH --partition=cpucourt module purge module load miniconda conda activate /home/user1/.conda/envs/my_env python /home/user1/jobs/my_code.py conda deactivate
Then submit this job by entering sbatch job.slurm on login-hpc.