{"id":564,"date":"2020-05-13T11:35:16","date_gmt":"2020-05-13T09:35:16","guid":{"rendered":"http:\/\/calculs.univ-cotedazur.fr\/?page_id=564"},"modified":"2025-09-10T16:27:54","modified_gmt":"2025-09-10T14:27:54","slug":"submit-a-job","status":"publish","type":"page","link":"https:\/\/calculs.univ-cotedazur.fr\/?page_id=564&lang=en","title":{"rendered":"Submit a job"},"content":{"rendered":"\n<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_11 counter-hierarchy counter-decimal ez-toc-grey\">\n<nav><ul class=\"ez-toc-list ez-toc-list-level-1\"><li class=\"ez-toc-page-1 ez-toc-heading-level-2\"><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/calculs.univ-cotedazur.fr\/?page_id=564&amp;lang=en#IMPORTANT_NOTICE\" title=\"IMPORTANT NOTICE\">IMPORTANT NOTICE<\/a><\/li><li class=\"ez-toc-page-1 ez-toc-heading-level-2\"><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/calculs.univ-cotedazur.fr\/?page_id=564&amp;lang=en#Configure_your_resource_reservation\" title=\"Configure your resource reservation\">Configure your resource reservation<\/a><\/li><li class=\"ez-toc-page-1 ez-toc-heading-level-2\"><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/calculs.univ-cotedazur.fr\/?page_id=564&amp;lang=en#Submit_a_job\" title=\"Submit a job\">Submit a job<\/a><ul class=\"ez-toc-list-level-3\"><li class=\"ez-toc-heading-level-3\"><a class=\"ez-toc-link ez-toc-heading-4\" href=\"https:\/\/calculs.univ-cotedazur.fr\/?page_id=564&amp;lang=en#In_a_script\" title=\"In a script\">In a script<\/a><ul class=\"ez-toc-list-level-4\"><li class=\"ez-toc-heading-level-4\"><a class=\"ez-toc-link ez-toc-heading-5\" href=\"https:\/\/calculs.univ-cotedazur.fr\/?page_id=564&amp;lang=en#OpenMP_jobs\" title=\"OpenMP jobs\">OpenMP jobs<\/a><\/li><li class=\"ez-toc-page-1 ez-toc-heading-level-4\"><a class=\"ez-toc-link ez-toc-heading-6\" href=\"https:\/\/calculs.univ-cotedazur.fr\/?page_id=564&amp;lang=en#MPI_jobs\" title=\"MPI jobs\">MPI jobs<\/a><\/li><li class=\"ez-toc-page-1 ez-toc-heading-level-4\"><a class=\"ez-toc-link ez-toc-heading-7\" href=\"https:\/\/calculs.univ-cotedazur.fr\/?page_id=564&amp;lang=en#Python_jobs\" title=\"Python jobs\">Python jobs<\/a><\/li><li class=\"ez-toc-page-1 ez-toc-heading-level-4\"><a class=\"ez-toc-link ez-toc-heading-8\" href=\"https:\/\/calculs.univ-cotedazur.fr\/?page_id=564&amp;lang=en#GPU\" title=\"GPU\">GPU<\/a><\/li><\/ul><\/li><li class=\"ez-toc-page-1 ez-toc-heading-level-3\"><a class=\"ez-toc-link ez-toc-heading-9\" href=\"https:\/\/calculs.univ-cotedazur.fr\/?page_id=564&amp;lang=en#Interactive_mode\" title=\"Interactive mode\">Interactive mode<\/a><ul class=\"ez-toc-list-level-4\"><li class=\"ez-toc-heading-level-4\"><a class=\"ez-toc-link ez-toc-heading-10\" href=\"https:\/\/calculs.univ-cotedazur.fr\/?page_id=564&amp;lang=en#Jupyter\" title=\"Jupyter\">Jupyter<\/a><\/li><\/ul><\/li><li class=\"ez-toc-page-1 ez-toc-heading-level-3\"><a class=\"ez-toc-link ez-toc-heading-11\" href=\"https:\/\/calculs.univ-cotedazur.fr\/?page_id=564&amp;lang=en#Visualization_node\" title=\"Visualization node\">Visualization node<\/a><\/li><li class=\"ez-toc-page-1 ez-toc-heading-level-3\"><a class=\"ez-toc-link ez-toc-heading-12\" href=\"https:\/\/calculs.univ-cotedazur.fr\/?page_id=564&amp;lang=en#Exclusive_mode\" title=\"Exclusive mode\">Exclusive mode<\/a><\/li><li class=\"ez-toc-page-1 ez-toc-heading-level-3\"><a class=\"ez-toc-link ez-toc-heading-13\" href=\"https:\/\/calculs.univ-cotedazur.fr\/?page_id=564&amp;lang=en#Dependencies_between_jobs\" title=\"Dependencies between jobs\">Dependencies between jobs<\/a><\/li><\/ul><\/li><\/ul><\/nav><\/div>\n<h2><span class=\"ez-toc-section\" id=\"IMPORTANT_NOTICE\"><\/span>IMPORTANT NOTICE<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p class=\"has-vivid-red-color has-text-color has-medium-font-size\"><img loading=\"lazy\" width=\"48\" height=\"48\" class=\"wp-image-2732\" style=\"width: 48px;\" src=\"http:\/\/calculs.univ-cotedazur.fr\/wp-content\/uploads\/2024\/04\/icons8-warning.gif\" alt=\"\"><strong>You must not execute your code\/software directly on the login-hpc machine.<\/strong><img loading=\"lazy\" width=\"48\" height=\"48\" class=\"wp-image-2732\" style=\"width: 48px;\" src=\"http:\/\/calculs.univ-cotedazur.fr\/wp-content\/uploads\/2024\/04\/icons8-warning.gif\" alt=\"\"><\/p>\n\n\n\n<p class=\"has-black-color has-text-color\">You must work on a compute node. To do so, you must first request to use one or several machines via a job scheduler named Slurm.<\/p>\n\n\n\n<p class=\"has-black-color has-text-color\">Your request can be included into a file. Then you need to launch this command: sbatch name_of_your_file on login-hpc to request an allocation of resources that will execute your code. <\/p>\n\n\n\n<p>For short duration executions, please refer to the interactive mode section.<\/p>\n\n\n\n<p class=\"has-black-color has-text-color\">You can find more details in the sections below.<\/p>\n\n\n\n<h2><span class=\"ez-toc-section\" id=\"Configure_your_resource_reservation\"><\/span>Configure your resource reservation<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>Reservations on the cluster are made via the&nbsp;<a rel=\"noreferrer noopener\" aria-label=\" (s\u2019ouvre dans un nouvel onglet)\" href=\"https:\/\/slurm.schedmd.com\/\" target=\"_blank\">Slurm Scheduler<\/a>, which puts the submitted jobs in queue until resources are allocated. <\/p>\n\n\n\n<p><strong>Important<\/strong>: <strong>When submitting a job, it is required to mention the following information:<\/strong><\/p>\n\n\n\n<ul><li><strong>account=<\/strong> the name of the group\/project assigned to you by the scientific committee (received by e-mail)<\/li><li><strong>partition=<\/strong> the queue on which your job will be sent. By default, if no partition is specified, the job is sent to cpucourt. Make sure to choose the right partition according to the needs of your job (time, resources). See the description of the partitions <a href=\"https:\/\/calculs.univ-cotedazur.fr\/?page_id=530&amp;lang=en\" data-type=\"page\" data-id=\"530\">here<\/a>.<\/li><li><strong>time=<\/strong> a walltime after which your job will be stopped by Slurm (with time =dd-hh:mm:ss). The walltime of your job must be less than or equal to the maximum walltime defined for the partition on which you submit your job (<a href=\"https:\/\/calculs.univ-cotedazur.fr\/?page_id=530&amp;lang=en\">view the limits here<\/a>). <\/li><li>the resources you wish to reserve (by default, 1 core). In particular, you can use:<ul><li><strong>-n ou ntasks<\/strong>= for the number of tasks to be run. By default, Slurm allocates one core per task. If your job is sequential (no parallelization), ntasks should be equal to 1. Give a value greater than 1 for the MPI jobs. <\/li><li><strong>cpus-per-task=<\/strong> to be used with OpenMP jobs. Use this option to specify the desired number of threads (to be associated with ntasks=1). Slurm allocates one core per cpu.<\/li><li><strong>ntasks-per-node= <\/strong>number of tasks to be performed on the same node.<\/li><li><strong>ntasks-per-socket=<\/strong> number of tasks to be performed on the same processor. As a reminder, the nodes of the cpucourt and cpulong queues have 2 processors of 20 cores each. <\/li><li><strong>-N ou nodes<\/strong>= for the number of nodes. If not specified, allocation will be based on other resource options. If you only specify the number of nodes, with the cpucourt queue (which is not exclusive) this will reserve by default 1 core per reserved node. As the cpulong queue is exclusive, if you reserve 1 node, you reserve all of its cores, even if you do not use them all<\/li><li><strong>constraint<\/strong>=intel or amd for the type of CPU you want to use (<a href=\"https:\/\/calculs.univ-cotedazur.fr\/?page_id=2106&amp;lang=en\" data-type=\"page\" data-id=\"2106\">more details here<\/a>)<\/li><\/ul><\/li><\/ul>\n\n\n\n<p><strong>Warning:<\/strong> even if your job has not finished executing once the walltime has been reached, your job will be automatically stopped. It is therefore strongly recommended that you carefully estimate the execution time of your job and introduce checkpoints in your code. <\/p>\n\n\n\n<h2><span class=\"ez-toc-section\" id=\"Submit_a_job\"><\/span>Submit a job<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<h3><span class=\"ez-toc-section\" id=\"In_a_script\"><\/span>In a script<span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<p>To submit your job: <strong>sbatch<\/strong> <strong>file_containing_your_script<\/strong><\/p>\n\n\n\n<p>The full list of options when submitting a job can be found <a href=\"https:\/\/slurm.schedmd.com\/sbatch.html\" target=\"_blank\" rel=\"noreferrer noopener\" aria-label=\"here. (s\u2019ouvre dans un nouvel onglet)\">here.<\/a><\/p>\n\n\n\n<p>Example of script named \u00ab\u00a0myJob\u00a0\u00bb which allows to execute 5 tasks (here a task will take 1 core). Resources will be reserved for a maximum of 2 hours and the job will be sent to the short job queue (cpucourt).<\/p>\n\n\n\n<pre class=\"bash alert\">#!\/bin\/bash<br><br>#SBATCH --job-name=myJob<br>#SBATCH --output=output.txt<br>\n#SBATCH --ntasks=5<br>#SBATCH --time=0-02:00:00\n#SBATCH --account=your_project_name<br>#SBATCH --partition=cpucourt<br>\n#SBATCH --mail-user=...@univ-cotedazur.fr<br>#SBATCH --mail-type=BEGIN,END,FAIL<br><br>module purge<br>module load ...<br><br>my commands to run the job ...<br><\/pre>\n\n\n\n<p>Although optional, the commands&nbsp;<em>mail-user<\/em>&nbsp;et&nbsp;<em>mail-type<\/em>&nbsp;allow you to receive email notifications about the status of your job.  <\/p>\n\n\n\n<p>The modules allow you to dynamically modify your environment variables needed to run your code (essentially PATH, LD_LIBRARY_PATH or MAN_PATH), depending on the module you are loading. For more information on using the modules, <a href=\"https:\/\/calculs.univ-cotedazur.fr\/?page_id=535&amp;lang=en\">click here.<\/a>  The complete list of modules installed on the cluster is available <a href=\"https:\/\/calculs.univ-cotedazur.fr\/?page_id=545&amp;lang=en\">here<\/a>.  <\/p>\n\n\n\n<h4><span class=\"ez-toc-section\" id=\"OpenMP_jobs\"><\/span>OpenMP jobs<span class=\"ez-toc-section-end\"><\/span><\/h4>\n\n\n\n<p>With OpenMP, parallelization can only be done between the different threads of a single node. It is therefore recommended to set ntasks to 1 so that the code execution is started only once. The number of desired threads is to be defined with cpus-per-tasks, which corresponds to the number of cores that will be allocated on this node. Here is an example:<\/p>\n\n\n\n<pre class=\"bash alert\">#!\/bin\/bash<br><br>#SBATCH --job-name=myJob\n#SBATCH --nodes=1\n#SBATCH --ntasks=1<br>#SBATCH --cpus-per-task=12<br>#SBATCH --time=0-02:00:00\n#SBATCH --account=projectname<br>#SBATCH --partition=cpucourt<br><br>export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK\n<\/pre>\n\n\n\n<h4><span class=\"ez-toc-section\" id=\"MPI_jobs\"><\/span>MPI jobs<span class=\"ez-toc-section-end\"><\/span><\/h4>\n\n\n\n<p><a href=\"https:\/\/calculs.univ-cotedazur.fr\/?page_id=584&amp;lang=en\">See this page.<\/a><\/p>\n\n\n\n<h4><span class=\"ez-toc-section\" id=\"Python_jobs\"><\/span>Python jobs<span class=\"ez-toc-section-end\"><\/span><\/h4>\n\n\n\n<p><a href=\"https:\/\/calculs.univ-cotedazur.fr\/?page_id=575&amp;lang=en\">See this page<\/a> to create your Miniconda environment.<\/p>\n\n\n\n<h4><span class=\"ez-toc-section\" id=\"GPU\"><\/span>GPU<span class=\"ez-toc-section-end\"><\/span><\/h4>\n\n\n\n<p>For jobs using the GPU node, you must add the two options below, <em>gpus <\/em>being the number of GPU cards to reserve (between 1 and 6).<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>#SBATCH --gpus=1\n#SBATCH --partition=gpu<\/code><\/pre>\n\n\n\n<p><strong>Important: in some codes using the cuda:0 notation you must replace this notation by cuda:$CUDA_VISIBLE_DEVICES to make your job run on the assigned graphics card(s).<\/strong><\/p>\n\n\n\n<p>If you want to specify the GPU card type (H100, A100 or V100):<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">#SBATCH --gpus=h100:1\nor\n#SBATCH --gpus=a100:1\nor\n#SBATCH --gpus=v100:1<\/pre>\n\n\n\n<p>To define the number of usable cores per GPU card, you can add the following option:<\/p>\n\n\n\n<pre class=\"wp-block-verse\"><code>#SBATCH <\/code>--cpus-per-gpu=<\/pre>\n\n\n\n<h3><span class=\"ez-toc-section\" id=\"Interactive_mode\"><\/span>Interactive mode<span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<p>The interactive mode is an alternative to the sbatch mode.<\/p>\n\n\n\n<p>This allows you to launch an interactive shell on a compute node so that you can directly work on that node. It is especially well suited for the following situations:<\/p>\n\n\n\n<ul><li>you want to launch short duration executions<\/li><li>you want to compile a code<\/li><li>you want to untar a file<\/li><li>your software requires user interaction<\/li><\/ul>\n\n\n\n<p><strong>This mode is perfect for server executions like Jupyter.<\/strong><\/p>\n\n\n\n<p>To launch an interactive bash shell for one hour on the <em>cpucourt <\/em>partition:<\/p>\n\n\n\n<pre class=\"bash alert\">srun -A my_account_Slurm -p cpucourt -t 01:00:00 --pty bash -i<\/pre>\n\n\n\n<p>In this example, there is no -N or -n option, thus Slurm reserves only one core.<\/p>\n\n\n\n<h4><span class=\"ez-toc-section\" id=\"Jupyter\"><\/span>Jupyter<span class=\"ez-toc-section-end\"><\/span><\/h4>\n\n\n\n<p><a href=\"https:\/\/calculs.univ-cotedazur.fr\/?page_id=2288&amp;lang=en\" data-type=\"page\" data-id=\"2288\">See this page.<\/a><\/p>\n\n\n\n<h3><span class=\"ez-toc-section\" id=\"Visualization_node\"><\/span>Visualization node<span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<p><a href=\"https:\/\/calculs.univ-cotedazur.fr\/?page_id=1813&amp;lang=en\" data-type=\"page\" data-id=\"1813\">See this page.<\/a><\/p>\n\n\n\n<h3><span class=\"ez-toc-section\" id=\"Exclusive_mode\"><\/span>Exclusive mode<span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<p><strong>By default, Azzurra uses the Slurm shared mode<\/strong>: you reserve a certain number of cores on one or more nodes. Other jobs than yours can run on the remaining cores of this node(s).<\/p>\n\n\n\n<p><strong>When using the exclusive mode, you reserve all the cores on one or several nodes.<\/strong> No other job will run at the same time as yours on these node(s). However, even if your job does not use all the reserved cores, Slurm considers your job has consumed the elapsed time of the job multiplied by the total amount of reserved cores.<\/p>\n\n\n\n<p>Add this option to use the exclusive mode:<\/p>\n\n\n\n<pre class=\"wp-block-verse\">#SBATCH --exclusive (in a batch)\nor\n--exclusive (command line)<\/pre>\n\n\n\n<h3><span class=\"ez-toc-section\" id=\"Dependencies_between_jobs\"><\/span>Dependencies between jobs<span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<p>If you need to submit a series of jobs that need to execute one after another, you can include dependencies. <\/p>\n\n\n\n<p>In the example below, an initial job is sent. When it&rsquo;s completed, a new job will start. This is an example with 5 dependencies. You can adjust it depending on your needs. Replace code.sh by the code you want to launch et adjust the slurm options (account, partition, etc.).<\/p>\n\n\n\n<pre class=\"wp-block-verse\">#!\/bin\/sh\n id=$(sbatch --parsable --account=your_account --partition=cpucourt --job-name=job-initial --ntasks=1 --output=outjob.txt code.sh)\n echo \"job 1 has jobid $id\"\n for n in {1..5}; do\n     id=$(sbatch --parsable --account=your_account --partition=cpucourt --depend=afterany:$id --job-name=iteration-$n --ntasks=1 --output=output-iter-$n.slurmout code.sh);\n     echo \"job $n has jobid $id\"\n done<\/pre>\n","protected":false},"excerpt":{"rendered":"<p>IMPORTANT NOTICE You must not execute your code\/software directly on the login-hpc machine. You must work on a compute node. To do so, you must first request to use one or several machines via a job scheduler named Slurm. Your &hellip; <\/p>\n","protected":false},"author":1,"featured_media":0,"parent":552,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":[],"_links":{"self":[{"href":"https:\/\/calculs.univ-cotedazur.fr\/index.php?rest_route=\/wp\/v2\/pages\/564"}],"collection":[{"href":"https:\/\/calculs.univ-cotedazur.fr\/index.php?rest_route=\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/calculs.univ-cotedazur.fr\/index.php?rest_route=\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/calculs.univ-cotedazur.fr\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/calculs.univ-cotedazur.fr\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=564"}],"version-history":[{"count":47,"href":"https:\/\/calculs.univ-cotedazur.fr\/index.php?rest_route=\/wp\/v2\/pages\/564\/revisions"}],"predecessor-version":[{"id":3128,"href":"https:\/\/calculs.univ-cotedazur.fr\/index.php?rest_route=\/wp\/v2\/pages\/564\/revisions\/3128"}],"up":[{"embeddable":true,"href":"https:\/\/calculs.univ-cotedazur.fr\/index.php?rest_route=\/wp\/v2\/pages\/552"}],"wp:attachment":[{"href":"https:\/\/calculs.univ-cotedazur.fr\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=564"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}