Slurm Jobs Array¶

Job arrays are a convenient way to submit and manage large numbers of similar jobs quickly. Job arrays are particularly useful when running similar jobs, such as performing the same analysis with different inputs or parameters.

There are several ways to define job arrays, such as specifying the range of indices or providing a list of indices in a file. Slurm also offers various features to manage and track job arrays, such as options to simultaneously suspend, resume, or cancel all jobs in the array.

Note

50 is the maximum number of jobs allowed to be run at once per user-account.

Job Array Examples¶

The most basic configuration for a job array is as follows:

#!/bin/bash
#SBATCH --partition=short
#SBATCH --job-name=job-array-example
#SBATCH --output=out_array_%A_%a.out
#SBATCH --error=err_array_%A_%a.err
#SBATCH --array=1-6

This command runs the same script six times using Slurm job arrays. Each job array has two additional environment variable sets. SLURM_ARRAY_JOB_ID (%A) is set to the first job ID of the array, and SLURM_ARRAY_TASK_ID (%a) is set to the job array index value. Both the SLURM_ARRAY_JOB_ID (%A) and SLURM_ARRAY_TASK_ID (%a) are referenced when naming outputs in the example so they are not overwritten when a “task” (i.e., one of the executions of the script through the job array) finishes.

Tip

Generally, we want to pass the SLURM_ARRAY_TASK_ID as an argument for our script. If you are using R, you can retrieve the former using task_id <- Sys.getenv("SLURM_ARRAY_TASK_ID"). If you are using job arrays with Python, you can obtain the task ID using the following:

import sys
taskId = sys.getenv('SLURM_ARRAY_TASK_ID')

Resource Allotment in Job Arrays¶

When submitting an array and setting its size with many dimensions, please use the % symbol to indicate how many jobs run simultaneously. When you specify the memory, number of nodes, number of CPUs, or other specifications, they will be applied to each job for the array.

Note

All resource-related #SBATCH directives (such as --mem, --cpus-per-task, --nodes, and --time) apply to each individual task in the array, not to the array as a whole. This means that Slurm will allocate the specified amount of memory, CPU cores, nodes, and time to every single task in the array independently. The directive --mem=128G does not divide 128 GB among all tasks. Instead, each task in the array receives 128 GB of memory. If your array consists of 600 tasks, and 20 run concurrently, then at peak usage Slurm may allocate up to 20 × 128 GB = 2560 GB of memory (subject to availability and limits). The same logic applies to other resources like CPUs and nodes.

For example, the following code specifies an array of 600 jobs, with 20 jobs running at a time:

#!/bin/bash
#SBATCH --partition=short
#SBATCH --job-name=job-array-example
#SBATCH --output=out-array_%A_%a.out
#SBATCH --error=err-array_%A_%a.err
#SBATCH --array=1-600%20
#SBATCH --mem=128G
#SBATCH --nodes=2

Slurm will submit 20 jobs to run simultaneously with each job, represented by a task ID, using the allocated resources for the submission of 2 nodes and 128 GB of RAM.

Job arrays are used for embarrassingly parallel jobs. If your job executed at each job ID does not use any multi-threading libraries, you can use the following header to avoid wasting resources:

#!/bin/bash
#SBATCH --partition=short
#SBATCH --job-name=jarray-example
#SBATCH --output=out/array_%A_%a.out
#SBATCH --error=err/array_%A_%a.err
#SBATCH --array=1-600%50  # 50 is the maximum number
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
#SBATCH --mem-per-cpu=2G ## debug prior to know how much RAM

Job Arrays from Commmand Line¶

You can launch job arrays from the sbatch submission on the command line as follows:

sbatch --array=<indexes> [options] script_file

Indexes can be listed as 1-5, =1,2,3,5,8,13 (i.e., each index listed), or 1-200%5 (i.e., produce a 200 task job array with only 5 tasks active at any given time).