Job submit commands with examples - teaching

Introduction

Slurm's main job submission commands are: sbatch, salloc, and srun. 

Note: Slurm does not automatically copy executable or data files to the nodes allocated to a job. The files must exist either on a local disk or in some global file system (e.g. NFS or CIFS). You can also, use sbcast command to transfer files to local storage on allocated nodes.

Command sbatch

Submit a job script for later execution.

Syntax

sbatch [options] Jobscript [args...]

Options

• Slurm batch job options are usually embedded in a job script prefixed by #SBATCH directives
• Slurm options can also be passed as command line options or by setting Slurm input environment variables
• options passed on the command line override corresponding options embedded in the job script or environment variable
• see Passing options to Slurm commands section below for more details

Job script

Usually a job script consists of two parts:

• Slurm directives section:
  • identified by #SBATCH directive at the beginning of each line in this section
  • where sbatch command line options are set. For reproducibility, use this section (instead of command line or environment variables) to pass sbatch options. For legibility, use long form options.
• Job commands section:
  • commands in this section are executed in the assigned node resources. It is written in scripting language identified by interpreter directive (e.g. #!/bin/bash).

Example

First create job script (e.g. job01.sh)

#!/bin/bash
#SBATCH --mail-user=userid@uwaterloo.ca
#SBATCH --mail-type=end,fail
#SBATCH --partition=gpu-k80
#SBATCH --job-name="test-job-01"
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=3
#SBATCH --time=00:10:00
#SBATCH --mem-per-cpu=1G
#SBATCH --gres=gpu:k80:1
#SBATCH --output=stdout-%j.log
#SBATCH --output=stderr-%j.log

# Your job commands
srun --label nvidia-smi --query-gpu=gpu_name \
     --format=csv,noheader

Then Submit the job script using sbatch command:

sbatch job01.sh

For more details on sbatch, read the man page (man sbatch) or visit the sbatch page on SchedMD website

Command salloc

• allocate resources (e.g. nodes), possibly with a set of constraints (e.g. number of processor per node) for later utilization
• use to allocate resources and spawn a shell, in which the srun command is used to launch parallel tasks

Syntax

salloc [options] [command [args...]]

Options

See Passing options to Slurm commands section below for more details.

Example

In this example we use salloc command to get allocation and create bash shell. Then use srun to execute nvidia-smi command.

$ salloc --partition=gpu-k80 --gres=gpu:k80:1 \
         --nodes=1 --ntasks-per-node=3 \
         /bin/bash
 salloc: Granted job allocation 168
 salloc: Waiting for resource configuration
 salloc: Nodes gpu-pt1-01 are ready for job

Now use srun to run --label nvidia-smi --query-gpu=gpu_name --format=csv,noheader command on resources allocated by salloc. The --label option will prepend lines of output with the remote task id

$ srun --label nvidia-smi \
       --query-gpu=gpu_name --format=csv,noheader
 2: Tesla K80
 1: Tesla K80
 0: Tesla K80

To exit salloc session, use exit command.

$ exit
 exit
 salloc: Relinquishing job allocation 168
 salloc: Job allocation 168 has been revoked.

For more details on salloc command read the man page (man salloc) or visit  salloc page on SchedMD website

Command srun

• Run a parallel job on cluster managed by Slurm. If run from command line directly, srun will first create a resource allocation in which to run the parallel job.
• srun can be run from within sbatch script or salloc shell

Syntax

srun [OPTIONS] executable [args...]

Options

See Passing options to Slurm commands section below for more details.

Example

$ srun --partition=gpu-k80 --gres=gpu:k80:1 \
       --nodes=1 --ntasks-per-node=3 \
       --label nvidia-smi --query-gpu=gpu_name \
       --format=csv,noheader

The output of the above example should looks like:

 2: Tesla K80
 0: Tesla K80
 1: Tesla K80

For more details on srun command read man pages (man srun) or visit  srun page on SchedMD website

Passing options to Slurm commands

• Command options can be passed in the following ways, listed in order of precedence:
  • On the command line
  • Input environment variables
  • In the job script (for sbatch command) prefixed by #SBATCH directive. 

The table below shows the most commonly-used options. All of these options can be used with sbatch command. srun and salloc commands support most of these options.
 

• below are some of the input environment variables for sbatch command and their equivalent command line options
• salloc also uses similar environment variables (SALLOC_*). Many of salloc environment variables have an sbatch equivalent.
  • for example, SBATCH_ACCOUNT environment variable is equivalent to SALLOC_ACCOUNT variable
• srun environment variables start with SLURM_* (e.g. SLURM_ACCOUNT)