Storage

Types of Storage

Super Computing Wales offers two types of storage: home directories and scratch.

The scratch disks are large, have large per-user quotas, are on fast storage but only intended for short term storage of temporary data. Please don’t leave data on the scratch disk once you are finished with it. If the disk fills up it will negatively impact on other users ability to run jobs.

Home directories have a default quota of 50GB per user, they are slower to access and are intended for longer term storage of data. However, please note that you should backup important data elsewhere as we do not formally offer an SCW backup service.

Checking Storage Quotas

Quotas are enabled on Supercomputing Wales home directories.

These limit the amount of storage space and number of files that a user or group can store in their respective home directory location.

To view your allocation, use the ‘myquota’ command, like this:

[a.n.other@cl1 ~]$ myquota
Disk quotas for group a.n.other (gid 1693): 
     Filesystem  blocks   quota   limit   grace   files   quota   limit   grace
cfsfs001-s05:/nfsshare/exports/space05
                   402G    500G    525G            192k   1000k   1050k

In this example, we see six items of data:

  • The first set of three items represent storage space:
    • ‘blocks’ is the amount of storage space utilised, in this case 402GB
    • ‘quota’ is the ‘soft limit’ assigned, in this case 500GB
    • ‘limit’ is the ‘hard limit’ assigned, in this case 525GB
  • The second set of three items represent file count:
    • ‘files’ is the number of files present, in this case 192000
    • ‘quota’ is the ‘soft limit’ assigned, in this case 1000000
    • ‘limit’ is the ‘hard limit’ assigned, in this case 1050000

A ‘soft limit’ can be exceeded for a period of maximum 7 days, after which utilisation must be reduced below the threshold or error will result. In this case, the ‘grace’ item in the ‘myquota’ output will display the number of days remaining in this state.

A ‘hard limit’ cannot be exceeded and attempts to do so will result in error.

Increases in quota allocations are possible on a case-by-case basis, please contact the Support Desk to request. However, it is worth noting that quotas are implemented to reduce the unfair use of the system by a few users and thus provide capacity for new users.

Shared Areas

We are able to create two types of shared area for multiple users to access:

Project Shared Areas are a way for the users of a particular project to share data. They are separate from user home directories and quotas as such.

Collaboration Shared Areas are also possible that cross project and institutional boundaries as separate areas from home directories for data sharing among multiple users.

Please contact Support to discuss either of these if this would be useful.