Large numbers of serial jobs can become incredibly inefficient and troublesome on mixed-mode HPC systems. The SCW Slurm deployment limits the number of running & submitted jobs any single user may have.<\/p>\n
However, there are ways to submit multiple jobs:<\/p>\n
Shell process control<\/strong><\/p>\n Here is an example of submitting 2 processes on a single node: GNU Parallel and Slurm’s srun command<\/strong><\/p>\n Here is the example, commented, job submission file serial_batch.sh<\/em>:
\n#!\/bin\/bash\n#SBATCH --ntasks=32\n#SBATCH --ntasks-per-node=32\n#SBATCH -o example.log.%J\n#SBATCH -e example.err.%J\n#SBATCH -J example\n\n#set the partition, use compute if running in Swansea\n#SBATCH -p htc\n#SBATCH --time=1:00:00\n#SBATCH --exclusive\n\ntime my_exec < input1.csv > input1.log.$SLURM_JOBID &\ntime my_exec < input2.csv > input2.log.$SLURM_JOBID &\n# important to make sure the batch job won't exit before all the\n# simultaneous runs are completed.\nwait\n<\/pre>
\nThe my_exec<\/em> commands in this case would be multithreaded to use 32 cores between them.<\/p>\n
\n#!\/bin\/bash --login\n#SBATCH -n 40 #Number of processors in our pool\n#SBATCH -o output.%J #Job output\n#SBATCH -t 12:00:00 #Max wall time for entire job\n\n#change the partition to compute if running in Swansea\n#SBATCH -p htc #Use the High Throughput partition which is intended for serial jobs\n\nmodule purge\nmodule load hpcw\nmodule load parallel\n\n# Define srun arguments:\nsrun="srun -n1 -N1 --exclusive"\n# --exclusive ensures srun uses distinct CPUs for each job step\n# -N1 -n1 allocates a single core to each task\n\n# Define parallel arguments:\nparallel="parallel -N 1 --delay .2 -j $SLURM_NTASKS --joblog parallel_joblog --resume"\n# -N 1 is number of arguments to pass to each job\n# --delay .2 prevents overloading the controlling node on short jobs\n# -j $SLURM_NTASKS is the number of concurrent tasks parallel runs, so number of CPUs allocated\n# --joblog name parallel's log file of tasks it has run\n# --resume parallel can use a joblog and this to continue an interrupted run (job resubmitted)\n\n# Run the tasks:\n$parallel "$srun .\/runtask arg1:{1}" ::: {1..64}\n# in this case, we are running a script named runtask, and passing it a single argument\n# {1} is the first argument\n# parallel uses ::: to separate options. Here {1..64} is a shell expansion defining the values for\n# the first argument, but could be any shell command\n#\n# so parallel will run the runtask script for the numbers 1 through 64, with a max of 40 running \n# at any one time\n#\n# as an example, the first job will be run like this:\n# srun -N1 -n1 --exclusive .\/runtask arg1:1<\/pre>
\nSo, in the above we are requesting an allocation from Slurm of 12 processors, but we have 32 tasks to run. Parallel will execute the jobs as soon as space on our allocation becomes available (i.e. tasks finish). As this does not have the overhead of setting up a new full job, it is more efficient.<\/p>\n