Slurm workload manager
Slurm commands are not the same as SGE commands, and options have different names. See here for a comparison of Torque, Slurm and SGE commands.
For more information about a command consult the man page, e.g.: man sbatch
As in SGE, slurm options can be specified on the command line or in active comments in the submission script. Slurm uses a different prefix for active comments - #SBATCH - which means you can include both SGE and slurm options in your scripts without conflict. The comparison table shows some commonly used options for Torque, Slurm and SGE.
Partitions
The equivalent of queues in SGE (or PBS/Torque) are called partitions in slurm. SGE queue names include ".q" at the end of the name, slurm partition names don't.
Available partitions on dogmatix currently are:
- smp
- the default partition including all SMP owned nodes except the asterix and ghost nodes.
- asterix
- equivalent to the low.q queue from asterix comprising nodes with 1-4 GPUs and 8 cores. Node a-1-0 has 96GB RAM, the others have 40-48GB.
Obelix had separate queues for different hardware but this isn't needed on dogmatix. If you want your job to run on a high memory node, specify the amount of memory you need with: --mem=memory. If you want your job to run on a specific type of hardware specify the hardware type as a constraint:
| Hardware | Constraint | Obelix |
| C6220 | --constraint=C6220
| -q low.q
|
| C8220 | --constraint=C8220
| -q low1.q
|
| FC430 | --constraint=FC430
| NEW |
| R720 | --constraint=R720
| -q small.q
|
| R610 | --constraint=R610
| -q small1.q
|
| R715 | --constraint=R715
| -q kanuka.q
|
Recommendations
Always specify a maximum run time (--time=[days-]hh:mm:ss). The default run time is unlimited to stop jobs being killed prematurely but the scheduler will favour jobs with a shorter maximum run time.
Always specify the memory you need (--mem=memory). The default memory allocation is small to ensure optimal use of resources and your job will stall or fail if it requires more memory than requested.
If you want your job to run on a specific type of hardware, specify the hardware to run on with: --constraint=hardware.
If your job doesn't use a lot of memory, specify an appropriate memory limit. Your job will likely run sooner and it keeps the large memory nodes free for jobs that really need large memory.
Data
Data partitions on dogmatix are the same as on obelix but home directories are new. You can find your obelix and asterix files in these partitions on dogmatix:
- /data/username
- the same as on obelix
- /data[2-6]/username
- the same as on obelix
- /obelix-home/username
- your home directory from obelix (read only)
- /asterix-home/username
- your home directory from asterix (read only)
- /asterix-data/username
- your data directory from asterix (read only)
For example you can copy files from your obelix home directory with:
cp /obelix-home/username/path path
Examples
Simple test script
For a short simple job requiring a single task on a single node:
#!/bin/bash #SBATCH --partition smp #SBATCH --nodes=1 #SBATCH --ntasks=1 #SBATCH --mem=2G #SBATCH --time=00:10:00 ...
Mpi large memory job
Runs on multiple large memory nodes using openmpi.
#!/bin/bash #SBATCH --partition smp #SBATCH --nodes=2 #SBATCH --ntasks-per-node=4 #SBATCH --mem=196G #SBATCH --time=05:00:00 mpirun ...
GPU job
Runs on an asterix node using a GPU.
#!/bin/bash #SBATCH --partition asterix #SBATCH --gres=gpu:1 #SBATCH --mem=1G #SBATCH --time=05:00:00 module load cuda ...
Intel Phi job
Runs on a node with an Intel Phi mic.
#!/bin/bash
#SBATCH --partition smp
#SBATCH --gres=mic:1
#SBATCH --time=01:00:00
ssh -x mic${OFFLOAD_DEVICES} "bash -l -c '...'"
Interactive job
- Command line
- :
srun --pty bash - Graphical
- :
srun --x11 --pty bash
NB To run matlab or mathematica jobs interactively, you'll need to request more than the default memory by adding --mem=memory to the srun command.
Asterix
The old asterix cluster nodes have been moved to the asterix partition on dogmatix. Asterix data has been copied to dogmatix and mounted under /asterix-home and /asterix-data. To copy your code from asterix to dogmatix, login to dogmatix and run:
cp /asterix-home/path path
cp /asterix-data/path path
To submit jobs to the asterix nodes add this to your submission script:
#SBATCH --partition asterix
You may need to increase the allocated memory to prevent your job from failing or stalling:
#SBATCH --mem=memory
The default CUDA version on dogmatix is 7.5 so CUDA jobs will need to be recompiled to use that version. Earlier versions are also available for compatitibilty with older code from asterix and ghost. You'll need to load the appropriate cuda module first:
module load cuda
module load cuda-3
module load cuda-4
module load cuda-6
Specify the number of GPUs required for your job with:
#SBATCH --gres=gpu:n
Specify the type of GPU you want to use (e.g. 2070 or 2050) with:
#SBATCH --constraint=2070
To run on the large memory node:
#SBATCH --mem=90G