Frequently Asked Questions

On this page you will find frequently asked questions about Slurm on Supercomputing Wales. We will be continually updating this page.

1. How do I create a machines file for mpirun?
2. How do get a list of processors/nodes that have been allocated to my job run?
3. How do I perform graphical interface X11 forwarding in Slurm jobs?
4. Can I access allocated nodes interactively during job runs for debugging?
5. What queues are there now? ‘sinfo’ is confusing
6. When will my job(s) run?
7. Why do I see non-editable Bash functions in my ksh environment when running under Slurm?
8. I used to run many hundreds or thousands of small jobs on LSF, but Slurm limits the number I can have. What can I do?
9. When submitting a job, I receive an error about inefficient tasks-per-node specification. What do I do?
10. How do I acknowledge SCW in publications?
11. I cannot login using SFTP (WinSCP, Filezilla, etc.)?
12. What does an error message about execve from slurmstepd in job log files mean?
13. I receive errors at the end of many jobs about rmdir and “Device or resource busy” – what is broken?

1. How do I create a machines file for mpirun?

Intel MPI and OpenMPI will automatically pick up the numbers of processors and the allocated hosts when mpirun is called within a running job script allocation. This is performed through Slurm setting environment variables appropriately that mpirun picks up. However, if you require to use an MPI that does not support this mechanism, then an allocated machines file is easily created by using the srun command inside the job script. srun is a Slurm command that runs a specified shell command against all or a selection of allocated nodes. When specified without any arguments it will run on all allocated processors, hence we can create a machines file easily:

srun hostname &gt;&gt; machines_file

srun also supports a large selection of specification semantics for limiting the target nodes. See the Slurm documentation for further details.

2. How do get a list of processors/nodes that have been allocated to my job run?

See the answer to 1.

3. How do I perform graphical interface X11 forwarding in Slurm jobs?

See here.

4. Can I access allocated nodes interactively during job runs for debugging?

You are able to ssh to allocated machines whilst your allocation remains in-place (i.e. your job is running there). Users cannot ssh to machines they are not allocated to and will be forcibly logged out of any such sessions when their jobs finish.

5. What queues are there now? ‘sinfo’ is confusing

‘sinfo’ is a little verbose as it reports on the node status within the queues (partitions) in addition to the queue presence itself. Try to run ‘sinfo -s’ for a shorter & more succinct report, for example:

sinfo -s
PARTITION AVAIL TIMELIMIT NODES(A/I/O/T) NODELIST
compute* up 3-00:00:00 15/51/1/67 ccs[0023-0048,0071-0096,0120-0134]
highmem up 3-00:00:00 0/26/0/26 ccs[1001-1026]
gpu up 2-00:00:00 0/13/0/13 ccs[2001-2013]
htc up 3-00:00:00 0/26/0/26 ccs[3001-3026]
dev up 30:00 0/2/0/2 ccs[0135-0136]

6. When will my job(s) run?

Please see this section in the Running & Monitoring Guide.

7. Why do I see non-editable Bash functions in my ksh environment when running under Slurm?

Ksh is sometimes required for some software to run (e.g. The Unified Model) and with the recent Bash update to fix the Shellshock vulnerability has created a situation where ksh inherits the environment variable beginning with BASH_FUNC when ksh is started from within a bash shell but it is unable to clear or modify the variable. This causes problems if you read the environment and try and recreate it elsewhere (such as within a mpirun wrapper script). This is a known bug and is reported in https://bugzilla.redhat.com/show_bug.cgi?id=1147645.

8. I used to run many hundreds or thousands of small jobs on LSF, but Slurm limits the number I can have. What can I do?

The previous scheduler made it too easy for users to abuse the system and/or cause to operate inefficiently by submitting many many jobs. Slurm enforces a Quality of Service settings that limits the number of jobs any single user can have running and the number they can have submitted. Attempts to submit further jobs will result in errors. Through the use of GNU parallel and Slurm’s srun command, we can now create (with a little bit of script) a far more efficient configuration for handling such use-cases that protects the service and also will result in quicker results being obtained (due to less job scheduling overhead). Please see Batch Submission of Serial Jobs for Parallel Execution.

9. When submitting a job, I receive an error about inefficient tasks-per-node specification. What do I do?

The old HPCW and Raven systems each had 12 or 16 processor cores per node. The new SCW systems have a similar overall processor count, but consisting of nodes with 40 processor cores per node. Hence, the overal node count is significantly lower, which raises the risk that partially-populated nodes (i.e. those on which all processor cores are not allocated to jobs) are utilised exclusively (through the Slurm exclusive flag) and that large amounts of processor cores are unallocated as a result and thus overall system occupancy & effiency would be impacted. Many scripts migrating from the previous systems will specify tasks-per-node as 12 or 16, so this is a significant potential cause of this. This submission-time error alerts users to this, so you should follow its instruction – i.e. either update your jobs or, if you do know what you are doing, then set an appropriate environment variable to enable override of this protection mechanism.

10. How do I acknowledge SCW in publications?

It is invaluable to the SCW project that this happens, so thank you and please use this text:

We acknowledge the support of the Supercomputing Wales project, which is part-funded by the European Regional Development Fund (ERDF) via Welsh Government.

11. I cannot login using SFTP (WinSCP, Filezilla, etc.)?

First up, please check that you are able to SSH into the services using the same method (e.g. key or password) that you are using. If further issues, fallback to a password login via SSH and SFTP to test. If further issues after that, please submit a support ticket.

12. What does an error message about execve from slurmstepd in job log files mean?

At the end of jobs you may see a message like this:

slurmstepd: error: execve(): /var/log/slurm/spool_slurmd/job9999999/slurm_script: Not a directory.

This message can be safely ignored. It relates to a system level function and is only output after your batch script has completed as part of the job teardown.

13. I receive errors at the end of many jobs about rmdir and “Device or resource busy” – what is broken?

Errors like this:

error: task_p_post_term: rmdir(/dev/cpuset/slurm6933968/slurm6933968.0_0) failed Device or resource busy

…do not actually indicate a problem with your script or its operation, as this message comes from the slurm job cleanup stage which runs after your job script completes.

Further, it does not leave compute nodes in bad state, as it just relates to a timing issue that is rapidly tidied up afterwards.