Slurm controls access to the GPUs on a node such that access is only granted when the resource is requested specifically. Slurm models GPUs as a Generic Resource (GRES), which is requested at job submission time via the following additional directive:<\/p>\n
#SBATCH --gres=gpu:2<\/pre>\nThis directive requires Slurm to allocate two GPUs per allocated node, to not use nodes without GPUs and to grant access. SCW GPU nodes have two GPUs each.<\/p>\n
Jobs must also be submitted to the desired GPU-enabled nodes queue:<\/p>\n
#SBATCH -p gpu # to request P100 GPUs<\/pre>Or\n#SBATCH -p gpu_v100 # to request V100 GPUs<\/pre>\nIt is then possible to use CUDA enabled applications or the CUDA toolkit modules themselves, modular environment examples being:<\/p>\n
module load CUDA\/9.1<\/pre>\nmodule load gromacs\/2018.2-single-gpu<\/pre>\nCUDA Versions & Hardware Differences<\/h3>\n
Multiple versions of the CUDA libraries are installed on SCW systems, as can always be seen by:<\/p>\n
[b.iss03c@cl1 ~]$ module avail CUDA ---- \/apps\/modules\/libraries ----\nCUDA\/10.0 CUDA\/10.1 CUDA\/11.2 CUDA\/11.3 CUDA\/11.4 CUDA\/8.0 CUDA\/9.0 CUDA\/9.1 CUDA\/9.2<\/pre>\nThe GPU nodes always run the latest nvidia driver to support the latest installed version of CUDA and also offer backwards-compatabilitity with prior versions.<\/p>\n
However, Pascal<\/em> generation nVidia Tesla cards (present in Hawk<\/strong>) which are supported in all installed versions of CUDA, but Volta<\/em> generation nVidia Tesla cards (present in Hawk<\/strong> and Sunbird<\/strong>) are only supported in CUDA 9+. Codes that require CUDA 8, such as Amber 16, will not run on the Volta<\/em> cards.<\/p>\n
Some important differences between Pascal<\/em> and Volta<\/em> nVidia Tesla cards:<\/p>\n\n
\n\n
\n\t \nCharacteristic<\/th> Volta<\/th> Pascal<\/th>\n<\/tr>\n<\/thead>\n \n\t Tensor Cores<\/td> 640<\/td> 0<\/td>\n<\/tr>\n \n\t Cuda Cores<\/td> 5120<\/td> 3584<\/td>\n<\/tr>\n \n\t Memory (GB)<\/td> 16<\/td> 16<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n\n Tensor cores are a new type of programmable core exclusive to GPUs based on the Volta architecture that run alongside standard CUDA cores. Tensor cores can accelerate mixed-precision matrix multiply and accumulate calculations in a single operation. This capability is specially significant for AI\/DL\/ML applications that rely on large matrix operations.<\/p>\n
GPU Compute Modes<\/h3>\n
nVidia GPU cards can be operated in a number of Compute Modes. In short the difference is whether multiple processes (and, theoretically, users) can access (share) a GPU or if a GPU is exclusively bound to a single process. It is typically application-specific whether one or the other mode is needed, so please pay particular attention to example job scripts. GPUs on SCW systems default to ‘shared’ mode.<\/p>\n
Users are able to set the Compute Mode of GPUs allocated to their job through a pair of helper scripts that should be called in a job script in the following manner:<\/p>\n
To set exclusive mode:<\/p>\n
clush -w $SLURM_NODELIST \"sudo \/apps\/slurm\/gpuset_3_exclusive\"<\/pre>\nAnd to set shared mode (although this is the default at the start of any job):<\/p>\n
clush -w $SLURM_NODELIST \"sudo \/apps\/slurm\/gpuset_0_shared\"<\/pre>\nTo query the Compute Mode:<\/p>\n
clush -w $SLURM_NODELIST \"nvidia-smi -q|grep Compute\"<\/pre>\nIn all cases above, sensible output will appear in the job output file.<\/p>\n
Additionally, as Slurm models the GPUs as a consumable resource that must be requested in their own right (i.e. not implicitly with processor\/node count), the default of the scheduler would be to not<\/em> allocate the same GPU to multiple users or jobs – it would take some manual work to achieve this.<\/p>
<\/p>\n","protected":false},"excerpt":{"rendered":"
GPU Usage Slurm controls access to the GPUs on a node such that access is only granted when the resource is requested specifically. Slurm models GPUs as a Generic Resource (GRES), which is requested at job submission time via the following additional directive: #SBATCH –gres=gpu:2 This directive requires Slurm to allocate two GPUs per allocated […]<\/p>\n","protected":false},"author":1,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"_lmt_disableupdate":"no","_lmt_disable":"","footnotes":""},"class_list":["post-830","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/portal.supercomputing.wales\/index.php\/wp-json\/wp\/v2\/pages\/830","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/portal.supercomputing.wales\/index.php\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/portal.supercomputing.wales\/index.php\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/portal.supercomputing.wales\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/portal.supercomputing.wales\/index.php\/wp-json\/wp\/v2\/comments?post=830"}],"version-history":[{"count":10,"href":"https:\/\/portal.supercomputing.wales\/index.php\/wp-json\/wp\/v2\/pages\/830\/revisions"}],"predecessor-version":[{"id":1308,"href":"https:\/\/portal.supercomputing.wales\/index.php\/wp-json\/wp\/v2\/pages\/830\/revisions\/1308"}],"wp:attachment":[{"href":"https:\/\/portal.supercomputing.wales\/index.php\/wp-json\/wp\/v2\/media?parent=830"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}