In order to use an HPC system interactively – i.e. whilst sat in front of the terminal interacting live with allocated resources – there is a simple two stage process in Slurm.<\/p>\n
Firstly, we must create an allocation – that is, an allocation of a certain amount of resources that we specify we need. This is done using the salloc<\/em> command, like this: Now that we are ‘inside’ an allocation, we can use the srun<\/em> command to execute against the allocated resources, for example: We could also launch an MPI job here if we wished. We would load the software modules<\/em> as we do in a batch script and call mpirun<\/em> in the same way. This can be useful during code debugging.<\/p>\n It is also possible to use srun<\/em> to launch an interactive shell process for some heavy processing on a compute node, for example: Once a resource allocation is granted as per the above, we can use srun<\/em> to provide X11 graphical forwarding all the way from the compute nodes to our desktop using srun –x11 <application>.<\/em><\/p>\n For example, to run an X terminal: Additionally, the –x11<\/em> argument can be augmented in this fashion –x11=[batch|first|last|all]<\/em> to the following effects:<\/p>\n <\/p>\n Job arrays operate in Slurm much as they do in other batch systems. They enable a potentially huge number of similar jobs to be launched very quickly and simply, with the value of a runtime-assigned array id<\/em> then being used to cause each particular job iteration to vary slightly what it does. Array jobs are declared using the –array<\/em> argument to sbatch<\/em>, which can (as with all arguments to sbatch<\/em>) be inside as job script as an #SBATCH<\/em> declaration or passed as a direct argument to sbatch<\/em>. There are a number of ways to declare: When a job array is running, the output of squeue<\/em> shows the parent task and the currently running iteration indexes: Each iteration in an array assumes its own job ID in Slurm. However, Slurm creates two new environment variables that can be used in the script in addition to SLURM_JOB_ID<\/em> storing the particular iteration’s job ID.<\/p>\n SLURM_ARRAY_JOB_ID<\/em> stores the value of the parent job submission – i.e. the ID reported in the output from sbatch<\/em> when submitted.<\/p>\n SLURM_ARRAY_TASK_ID <\/em>stores the value of the array index.<\/p>\n Additionally, when specifying a job’s STDOUT and STDERR files using the -o<\/em> and -e<\/em> directives to sbatch<\/em>, the reference %A <\/em>will take on the parent job ID and the reference %a<\/em> will take on the iteration index. In summary:<\/p>\n\n
\n[test.user@cstl001 imb]$ salloc -n 8 --ntasks-per-node=1\n salloc: Granted job allocation 134<\/pre>
\nNow that an allocation has been granted, we have access to those specified resources. Note that the resource specification we made in this case is exactly as the parameters passed for batch use was – so in this case we have asked for 8 tasks (processes) with them distributed at one per node.<\/p>\n
\n[test.user@cstl001 imb]$ srun hostname\ncst004\ncst003\ncst002\ncst008\ncst006\ncst007\ncst001\ncst005<\/pre>
\nThe above output shows how, by default, srun<\/em> executes a command on all allocated processors. Arguments can be passed to srun<\/em> to operate differently, for example:<\/p>\n
\nsrun -n 2 --pty bash<\/pre>
\nThis would move us to a shell on a compute node.<\/p>\n<\/h5>\n
X11 Forwarding Interactively<\/h3>\n
\nsrun --x11 xterm<\/pre>
\nNote that the user must have X11 forwarded to the login node for this to work – this can be checked by running xclock<\/em> at the command line.<\/p>\n\n
Job Arrays<\/h3>\n
Submission<\/h4>\n
\n[test.user@cstl001 hello_world]$ sbatch --array=0-64 sbatch_sub.sh<\/pre>
\n…declares an array with iteration indexes from 0 to 64.
\n[test.user@cstl001 hello_world]$ sbatch --array=0,4,8,12 sbatch_sub.sh<\/pre>
\n…declares an array with iteration indexes specifically identified as 0, 4, 8 and 12.
\n[test.user@cstl001 hello_world]$ sbatch --array=0-12:3 sbatch_sub.sh<\/pre>
\n…declares an array with iteration indexes from 0 to 12 with a stepping of 3, i.e. 0,3,6,9,12<\/p>\nMonitoring<\/h4>\n
\n[test.user@cstl001 hello_world]$ squeue\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 JOBID PARTITION\u00a0\u00a0\u00a0\u00a0 NAME\u00a0\u00a0\u00a0\u00a0 USER ST\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 TIME\u00a0 NODES NODELIST(REASON)\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 143_[6-64]\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 all\u00a0\u00a0\u00a0 hello test.use PD\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 0:00\u00a0\u00a0\u00a0\u00a0\u00a0 4 (Resources)\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 143_4\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 all\u00a0\u00a0\u00a0 hello test.use\u00a0 R\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 0:00\u00a0\u00a0\u00a0\u00a0\u00a0 4 cst[005-008]\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 143_5\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 all\u00a0\u00a0\u00a0 hello test.use\u00a0 R\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 0:00\u00a0\u00a0\u00a0\u00a0\u00a0 4 cst[005-008]\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 143_0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 all\u00a0\u00a0\u00a0 hello test.use\u00a0 R\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 0:03\u00a0\u00a0\u00a0\u00a0\u00a0 4 cst[001-004]\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 143_1\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 all\u00a0\u00a0\u00a0 hello test.use\u00a0 R\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 0:03\u00a0\u00a0\u00a0\u00a0\u00a0 4 cst[001-004]\n[test.user@cstl001 hello_world]$ squeue\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 JOBID PARTITION\u00a0\u00a0\u00a0\u00a0 NAME\u00a0\u00a0\u00a0\u00a0 USER ST\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 TIME\u00a0 NODES NODELIST(REASON)\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 143_[15-64]\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 all\u00a0\u00a0\u00a0 hello test.use PD\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 0:00\u00a0\u00a0\u00a0\u00a0\u00a0 4 (Resources)\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 143_14\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 all\u00a0\u00a0\u00a0 hello test.use\u00a0 R\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 0:00\u00a0\u00a0\u00a0\u00a0\u00a0 4 cst[001-004]\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 143_10\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 all\u00a0\u00a0\u00a0 hello test.use\u00a0 R\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 0:02\u00a0\u00a0\u00a0\u00a0\u00a0 4 cst[005-008]\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 143_11\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 all\u00a0\u00a0\u00a0 hello test.use\u00a0 R\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 0:02\u00a0\u00a0\u00a0\u00a0\u00a0 4 cst[005-008]\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 143_1\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 all\u00a0\u00a0\u00a0 hello test.use\u00a0 R\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 0:07\u00a0\u00a0\u00a0\u00a0\u00a0 4 cst[001-004]<\/pre><\/p>\n
IDs and Variables<\/h4>\n