{"id":38,"date":"2015-07-21T10:59:47","date_gmt":"2015-07-21T09:59:47","guid":{"rendered":"https:\/\/portal.supercomputing.wales\/?page_id=38"},"modified":"2018-09-21T12:05:13","modified_gmt":"2018-09-21T11:05:13","slug":"migrating-jobs","status":"publish","type":"page","link":"https:\/\/portal.supercomputing.wales\/index.php\/index\/slurm\/migrating-jobs\/","title":{"rendered":"More On Slurm Jobs & migration"},"content":{"rendered":"

Job Runtime Environment in Slurm<\/h3>\n

When a job is submitted, Slurm will store all environment variables in-place at submission time and replicate that environment on the first allocated node where the batch script actually runs.<\/p>\n

Additionally, at run time, Slurm will set a number of shell environment variables that relate to the job itself and can be used in the job run. The Slurm documentation’s manpage on\u00a0sbatch<\/em> provides an exhaustive guide, but we highlight some useful ones here.<\/p>\n

<\/h4>\n

#SBATCH Directives<\/h4>\n

In line with most batch schedulers, Slurm uses directives in submission scripts to specify job requirements and parameters for a job – the #SBATCH directives. Thus for an MPI task we might typically have:
\n

#SBATCH -p compute\n#SBATCH -o runout.%J\n#SBATCH -e runerr.%J\n#SBATCH --job-name=mpijob\n#SBATCH -n 80\n#SBATCH --tasks-per-node=40\n#SBATCH --exclusive\n#SBATCH -t 0-12:00\n#SBATCH --mem-per-cpu=4000<\/pre>
\nWalking through these:<\/p>\n\n\n\n\n\t\n\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t
Slurm #SBATCH directive<\/th>Description<\/th>\n<\/tr>\n<\/thead>\n
#SBATCH --partition=compute
\nor
\n#SBATCH -p compute<\/td>		In Slurm, jobs are submitted to 'partitions'.  Despite the naming difference, the concept is the same.<\/td>\n<\/tr>\n
#SBATCH --output=runout.%J
\nor
\n#SBATCH -o runout.%J<\/td>		File for STDOUT from the job run to be stored in.  The '%J' to Slurm is replaced with the job number.<\/td>\n<\/tr>\n
#SBATCH --error=runerr.%J
\nor
\n#SBATCH -e runerr.%J<\/td>		File for STDERR from the job run to be stored in. The '%J' to Slurm is replaced with the job number.<\/td>\n<\/tr>\n
#SBATCH --job-name=mpijob<\/td>Job name, useful for monitoring and setting up inter-job dependency.<\/td>\n<\/tr>\n
#SBATCH --ntasks=128
\nor
\n#SBATCH -n 128<\/td>		Number of processors required for job.<\/td>\n<\/tr>\n
#SBATCH --tasks-per-node=16<\/td>The number of processors (tasks) to run per node.<\/td>\n<\/tr>\n
#SBATCH --exclusive<\/td>Exclusive job allocation - i.e. no other users on allocated nodes.<\/td>\n<\/tr>\n
#SBATCH --time=0-12:00
\nor
\n#SBATCH -t 0-12:00<\/td>		Maximum runtime of job.  Note that it is beneficial to specify this and not leave it at the maximum as it will improve the chances of the scheduler 'back-filling' the job and running it earlier.<\/td>\n<\/tr>\n
#SBATCH --mem-per-cpu=4000 <\/td>Memory requirements of job.  Slurm's memory-based scheduling is more powerful than many schedulers.  <\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n\n
 <\/p>\n
Environment Variables<\/h4>\n
Once an allocation has been scheduled and a job script is started (on the first node of the allocation), Slurm sets a number of shell environment variables that can be used in the script at runtime. Below is a summary of some of the most useful:<\/p>\n\n\n\n\n\t\n\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t
Slurm<\/th>Description<\/th>\n<\/tr>\n<\/thead>\n
$SLURM_JOBID<\/td>Job ID.<\/td>\n<\/tr>\n
$SLURM_JOB_NODELIST<\/td>Nodes allocated to the job i.e. with at least once task on.<\/td>\n<\/tr>\n
$SLURM_ARRAY_TASK_ID<\/td>If an array job, then the task index.<\/td>\n<\/tr>\n
$SLURM_JOB_NAME<\/td>Job name.<\/td>\n<\/tr>\n
$SLURM_JOB_PARTITION<\/td>Partition that the job was submitted to.<\/td>\n<\/tr>\n
$SLURM_JOB_NUM_NODES<\/td>Number of nodes allocated to this job.  <\/td>\n<\/tr>\n
$SLURM_NTASKS<\/td>Number of tasks (processes) allocated to this job.<\/td>\n<\/tr>\n
$SLURM_NTASKS_PER_NODE
\n(Only set if the --ntasks-per-node option is specified)<\/td>		Number of tasks (processes) per node.<\/td>\n<\/tr>\n
$SLURM_SUBMIT_DIR<\/td>Directory in which job was submitted.<\/td>\n<\/tr>\n
$SLURM_SUBMIT_HOST<\/td>Host on which job was submitted.<\/td>\n<\/tr>\n
$SLURM_PROC_ID<\/td>The process (task) ID within the job. This will start from zero and go up to $SLURM_NTASKS-1.<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n\n
 <\/p>\n
System Queues & Partitions in Slurm<\/h3>\n
Please use the sinfo <\/em>command to see the names of the partitions (queues) to use in your job scripts. If not specified, the default partition will be used for job submissions. sinfo -s <\/em>will give a more succinct partition list. Please see the Hawk<\/a> and Sunbird<\/a> pages for a list of partitions and their descriptions on each system.<\/p>\n","protected":false},"excerpt":{"rendered":"
Job Runtime Environment in Slurm When a job is submitted, Slurm will store all environment variables in-place at submission time and replicate that environment on the first allocated node where the batch script actually runs. Additionally, at run time, Slurm will set a number of shell environment variables that relate to the job itself and […]<\/p>\n","protected":false},"author":1,"featured_media":0,"parent":33,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"page-nosidebar.php","meta":{"_lmt_disableupdate":"","_lmt_disable":"","footnotes":""},"class_list":["post-38","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/portal.supercomputing.wales\/index.php\/wp-json\/wp\/v2\/pages\/38","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/portal.supercomputing.wales\/index.php\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/portal.supercomputing.wales\/index.php\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/portal.supercomputing.wales\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/portal.supercomputing.wales\/index.php\/wp-json\/wp\/v2\/comments?post=38"}],"version-history":[{"count":30,"href":"https:\/\/portal.supercomputing.wales\/index.php\/wp-json\/wp\/v2\/pages\/38\/revisions"}],"predecessor-version":[{"id":873,"href":"https:\/\/portal.supercomputing.wales\/index.php\/wp-json\/wp\/v2\/pages\/38\/revisions\/873"}],"up":[{"embeddable":true,"href":"https:\/\/portal.supercomputing.wales\/index.php\/wp-json\/wp\/v2\/pages\/33"}],"wp:attachment":[{"href":"https:\/\/portal.supercomputing.wales\/index.php\/wp-json\/wp\/v2\/media?parent=38"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}