Job arrays operate in Slurm much as they do in other batch systems. They enable a potentially huge number of similar jobs to be launched very quickly and simply, with the value of a runtime-assigned array id<\/em> then being used to cause each particular job iteration to vary slightly what it does. Array jobs are declared using the –array<\/em> argument to sbatch<\/em>, which can (as with all arguments to sbatch<\/em>) be inside as job script as an #SBATCH<\/em> declaration or passed as a direct argument to sbatch<\/em>. There are a number of ways to declare:<\/p>\n …declares an array with iteration indexes from 0 to 64.<\/p>\n …declares an array with iteration indexes specifically identified as 0, 4, 8 and 12.<\/p>\n …declares an array with iteration indexes from 0 to 12 with a stepping of 3, i.e. 0,3,6,9,12<\/p>\n When a job array is running, the output of squeue<\/em> shows the parent task and the currently running iteration indexes:<\/p>\n Each iteration in an array assumes its own job ID in Slurm. However, Slurm creates two new environment variables that can be used in the script in addition to SLURM_JOB_ID<\/em> storing the particular iteration’s job ID.<\/p>\n SLURM_ARRAY_JOB_ID<\/em> stores the value of the parent job submission – i.e. the ID reported in the output from sbatch<\/em> when submitted.<\/p>\n SLURM_ARRAY_TASK_ID <\/em>stores the value of the array index.<\/p>\n Additionally, when specifying a job’s STDOUT and STDERR files using the -o<\/em> and -e<\/em> directives to sbatch<\/em>, the reference %A <\/em>will take on the parent job ID and the reference %a<\/em> will take on the iteration index. In summary:<\/p>\n\n And so, with this example script:<\/p>\n We can submit the script:<\/p>\n Resulting in the following output files:<\/p>\n Each iteration of which contained variables as follows:<\/p>\n[test.user@cl1 hello_world]$ sbatch --array=0-64 sbatch_sub.sh<\/pre>\n
[test.user@cl1 hello_world]$ sbatch --array=0,4,8,12 sbatch_sub.sh<\/pre>\n
[test.user@cl1 hello_world]$ sbatch --array=0-12:3 sbatch_sub.sh<\/pre>\n
Monitoring<\/h4>\n
[test.user@cl1 hello_world]$ squeue JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 143_[6-64] all hello test.use PD 0:00 4 (Resources) 143_4 all hello test.use R 0:00 4 ccs[005-008] 143_5 all hello test.use R 0:00 4 ccs[005-008] 143_0 all hello test.use R 0:03 4 ccs[001-004] 143_1 all hello test.use R 0:03 4 ccs[001-004]\n[test.user@cl1 hello_world]$ squeue JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 143_[15-64] all hello test.use PD 0:00 4 (Resources) 143_14 all hello test.use R 0:00 4 ccs[001-004] 143_10 all hello test.use R 0:02 4 ccs[005-008] 143_11 all hello test.use R 0:02 4 ccs[005-008] 143_1 all hello test.use R 0:07 4 ccs[001-004]<\/pre>\n
IDs and Variables<\/h4>\n
\n\n
\n\t \nBASH Environment Variable<\/th> SBATCH Field Code<\/th> Description<\/th>\n<\/tr>\n<\/thead>\n \n\t $SLURM_JOB_ID<\/td> %J<\/td> Job identifier<\/td>\n<\/tr>\n \n\t $SLURM_ARRAY_JOB_ID<\/td> %A<\/td> Array parent job identifier<\/td>\n<\/tr>\n \n\t $SLURM_ARRAY_TASK_ID<\/td> %a<\/td> Array job iteration index<\/td>\n<\/tr>\n \n\t $SLURM_ARRAY_TASK_COUNT<\/td> <\/td> Number of indexes (tasks) in the job array<\/td>\n<\/tr>\n \n\t $SLURM_ARRAY_TASK_MAX<\/td> <\/td> Maximum array index<\/td>\n<\/tr>\n \n\t $SLURM_ARRAY_TASK_MIN<\/td> <\/td> Minimum array index<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n\n #!\/bin\/bash #SBATCH -J arraytest\n#SBATCH --array=0-4\n#SBATCH -o output-%A_%a-%J.o\n#SBATCH -n 1 echo SLURM_JOB_ID $SLURM_JOB_ID\necho SLURM_ARRAY_JOB_ID $SLURM_ARRAY_JOB_ID\necho SLURM_ARRAY_TASK_ID $SLURM_ARRAY_TASK_ID<\/pre>\n
[test.user@cl1 sbatch]$ sbatch array.sh Submitted batch job 231<\/pre>\n
output-231_0-232.o\noutput-231_1-233.o\noutput-231_2-234.o\noutput-231_3-235.o\noutput-231_4-231.o<\/pre>\n
output-231_0-232.o:\nSLURM_JOB_ID 232\nSLURM_ARRAY_JOB_ID 231\nSLURM_ARRAY_TASK_ID 0<\/pre>\n
output-231_1-233.o:\nSLURM_JOB_ID 233\nSLURM_ARRAY_JOB_ID 231\nSLURM_ARRAY_TASK_ID 1<\/pre>\n
output-231_2-234.o:\nSLURM_JOB_ID 234\nSLURM_ARRAY_JOB_ID 231\nSLURM_ARRAY_TASK_ID 2<\/pre>\n
output-231_3-235.o:\nSLURM_JOB_ID 235\nSLURM_ARRAY_JOB_ID 231\nSLURM_ARRAY_TASK_ID 3<\/pre>\n
output-231_4-231.o:\nSLURM_JOB_ID 231\nSLURM_ARRAY_JOB_ID 231\nSLURM_ARRAY_TASK_ID 4<\/pre>\n