BioSLURM cluster user guide

This document describes use of the bioslurm SLURM cluster. In order to use the cluster, you have to be a member of a BioHPC "membership"-type credit account (please refer to BioHPC Cloud: User Guide to learn more about credit accounts). To check whether it is the case, log in to BioHPC Home using your BioHPC credentials and navigate to 'User -> Manage Credit Accounts'.

Cluster overview

The cluster name is bioslurm. It contains three CPU compute nodes and two GPU nodes:

Node	SLURM CPUs	Memory available to SLURM	Local scratch	Other resources	Partitions
cbsulm09	64	515434 MB	7500 GB	—	short, long, debug
cbsulm10	64	515434 MB	7500 GB	—	short, long, debug
cbsulm14	112	515312 MB	10500 GB	avx2 feature	short, long, debug
cbsugpu02	24	257290 MB	4500 GB	1 Tesla P100 GPU	gpu
cbsugpu03	32	257290 MB	10500 GB	2 Tesla P100 GPUs	gpu

On cbsugpu02, eight CPU threads are intentionally left outside SLURM for controller and system services.

Storage

Home directories

Home directories under /home are on networked storage. Do not run I/O-intensive computations directly in a home directory. Home is suitable for scripts, small configuration files, logs, and final results, but not for repeatedly reading or writing large working datasets.

Consumable local scratch

Every BioSLURM job receives a private directory on node-local scratch storage. Request the desired capacity in GB with the SLURM GRES option --gres=scratch:N. For example, the following requests 200 GB:

#SBATCH --gres=scratch:200

If --gres=scratch:N is omitted, the job receives 50 GB by default. Explicitly requesting scratch is nevertheless recommended because it documents the job's storage requirement and allows SLURM to schedule the job on a node with sufficient local space.

At job start, a directory is automatically created as:

/workdir/<userID>/<JobID>

The same directory is exposed inside the job through all of these environment variables:

$SLURM_TMPDIR
$JOB_SCRATCH
$TMPDIR
$TMP
$TEMP

SLURM_TMPDIR is the preferred name to use in scripts. The scratch directory is the only location on the compute node that is writable by jobs. User home directories, including /home2, remain accessible for reading and writing as usual, but they reside on network storage rather than on the local node.

Using local scratch efficiently

For best performance, jobs should perform most input/output operations in the node-local directory identified by $SLURM_TMPDIR. A typical workflow is:

#!/bin/bash -l
#SBATCH --gres=scratch:200

# Move to the node-local scratch directory
cd "$SLURM_TMPDIR"

# Copy input data from permanent storage
cp ~/project/input.fastq .

# Run the analysis
my_program input.fastq

# Copy results back before the job exits
cp results.txt ~/project/results/

This pattern keeps I/O-intensive temporary and intermediate files on local disks while preserving only the files that are needed after the job completes.

When several jobs use the same input: /home2

If multiple jobs need to read the same input data, placing that data in /home2/<userID> can avoid copying an identical large dataset into each job's private scratch directory. The /home2 filesystem is mounted on all BioSLURM compute nodes and is accessible from within jobs.

For example, a shared reference can remain on /home2 while each job writes temporary files and intermediate output to its own local scratch:

# Shared input/reference data
reference=/home2/$USER/genomes/hg38.fa

# Per-job local working directory
cd "$SLURM_TMPDIR"

my_program --reference "$reference" --input input.fastq

Prefer SLURM_TMPDIR	Prefer /home2
Temporary files	Shared input datasets
Intermediate results	Reference genomes and databases
Sort files and other heavy temporary I/O	Software or data that several jobs must read
Large I/O-intensive working output	Files that must remain after a job ends

As a general rule, perform computation and temporary I/O in $SLURM_TMPDIR. Copy input data into scratch when practical, and copy final results back to permanent storage before exit. Use /home2 when sharing the same input across jobs avoids needless duplication, while keeping in mind the limitations of network I/O.

Partitions and limits

Partition	Nodes	Maximum time	Default?	Purpose and special limits
short	cbsulm09, cbsulm10, cbsulm14	24 hours	Yes	Normal CPU jobs of up to one day.
long	cbsulm09, cbsulm10, cbsulm14	7 days	No	Long CPU jobs. Subject to the cluster-wide bsl_long_limit QOS cap described below.
gpu	cbsugpu02, cbsugpu03	24 hours	No	Jobs requiring Tesla P100 GPUs.
debug	cbsulm09, cbsulm10, cbsulm14	1 hour	No	Short testing and troubleshooting jobs; configured with a higher partition priority tier.

If --partition is omitted, the job goes to short. If --time is omitted, the applicable partition maximum is used. The default memory request is 4096 MB per job on all partitions.

The debug partition has higher scheduling priority than the ordinary CPU partitions, but it is limited to one hour. It should be used for genuine short tests, not production workloads.

Submitting batch jobs

Create a shell script such as submit.sh. A typical CPU job is:

#!/bin/bash -l
#SBATCH --cluster=bioslurm
#SBATCH --nodes=1
#SBATCH --ntasks=8
#SBATCH --mem=8G
#SBATCH --gres=scratch:200
#SBATCH --time=04:00:00
#SBATCH --partition=short
#SBATCH --chdir=/home/NETID/slurm
#SBATCH --job-name=myjob
#SBATCH --output=myjob.%j.out
#SBATCH --mail-user=NETID@cornell.edu
#SBATCH --mail-type=END,FAIL

set -euo pipefail

cd "$SLURM_TMPDIR"

# Copy input to $SLURM_TMPDIR, run the program here,
# and copy final output back to permanent storage before exit.

Submit it with:

sbatch submit.sh

Options may instead be placed on the command line:

sbatch --cluster=bioslurm -N 1 -n 8 --mem=8G --gres=scratch:200 -t 04:00:00 -p short submit.sh

A command-line option overrides the same option in the script header. The option --cluster=bioslurm may be omitted if the environment variable SLURM_CLUSTERS is defined in a shell from which a job is being submitted:

export SLURM_CLUSTERS=bioslurm

The above command can be inserted into the user's .bashrc file.

Choosing a partition

# CPU job no longer than 24 hours
#SBATCH --partition=short

# CPU job longer than 24 hours, up to 7 days
#SBATCH --partition=long

# Short test, no longer than 1 hour
#SBATCH --partition=debug

# GPU job
#SBATCH --partition=gpu

CPU, memory, and node requests

One-node threaded programs

Most BioHPC applications are either serial or multithreaded but not MPI-enabled. Keep such jobs on one node:

#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=16
#SBATCH --mem=64G
#SBATCH --gres=scratch:200

export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK

For software whose command-line option specifies a thread count, pass $SLURM_CPUS_PER_TASK or the same numeric value.

Independent processes

For eight independent single-threaded tasks, request:

#SBATCH --cluster=bioslurm
#SBATCH --nodes=1
#SBATCH --ntasks=8
#SBATCH --cpus-per-task=1
#SBATCH --gres=scratch:100

MPI jobs

Only MPI-aware software can use multiple nodes as one parallel job. Request the required number of tasks and nodes, then launch through srun or the application-specific MPI procedure.

Specific nodes and features

# Run specifically on cbsulm14
#SBATCH --nodelist=cbsulm14

# Request a node carrying the avx2 feature
#SBATCH --constraint=avx2

# Avoid selected nodes
#SBATCH --exclude=cbsulm09,cbsulm10

Scratch syntax

# Request 200 GB of local scratch
#SBATCH --gres=scratch:200

# GPU plus 200 GB scratch
#SBATCH --gres=gpu:tP100:1,scratch:200

If the scratch option is omitted, 50 GB is allocated by default. Scratch is a consumable node resource: a job may wait for a node that has enough free scratch capacity even if CPU and memory are otherwise available.

Memory syntax

# Total memory for the entire job
#SBATCH --mem=48G

# Memory per allocated CPU
#SBATCH --mem-per-cpu=2G

For example, --ntasks=8 --cpus-per-task=3 --mem-per-cpu=2G requests 24 CPUs and 48 GB total.

Checking actual memory use

After a job finishes, inspect its requested and peak resident memory (relpace JOBID with the actual ID of your job):

sacct --cluster=bioslurm -j JOBID --format=JobID,JobName,Partition,ReqCPUS,ReqMem,MaxRSS,Elapsed,State,ExitCode

Use the .batch line for the batch script's peak memory. Request somewhat more than observed MaxRSS for future jobs with similar data and parameters.

GPU jobs

The gpu partition provides three Tesla P100 GPUs: one on cbsugpu02 and two on cbsugpu03. Scratch and GPU resources can be requested together in one comma-separated --gres option. A one-GPU job can use either node:

#!/bin/bash -l
#SBATCH --cluster=bioslurm
#SBATCH --partition=gpu
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=8
#SBATCH --mem=64G
#SBATCH --gres=gpu:tP100:1,scratch:200
#SBATCH --time=12:00:00
#SBATCH --output=gpu.%j.out

nvidia-smi
python my_gpu_program.py

A job requiring two GPUs must run on cbsugpu03:

#SBATCH --partition=gpu
#SBATCH --nodelist=cbsugpu03
#SBATCH --gres=gpu:tP100:2,scratch:200

Within a correctly launched job, SLURM sets CUDA_VISIBLE_DEVICES to the GPU or GPUs assigned to that job. Do not override this variable unless you understand the consequences.

Monitoring and controlling jobs

If the SLURM_CLUSTERS variable is not set, add option --cluster=bioslurm to each of the sinfo, scontrol, squeue, scancel, and sacct commands below:

# Partition and node summary
sinfo

# Detailed partition information
scontrol show partitions

# Detailed node information
scontrol show nodes

# Your jobs
squeue -u "$USER"

# Explain why a pending job is waiting
squeue -j JOBID -o "%.18i %.9P %.8j %.8u %.2t %.10M %.6D %R"
scontrol show job JOBID

# Cancel a job
scancel JOBID

# Accounting information after or during a job
sacct -j JOBID --format=JobID,Partition,AllocCPUS,ReqMem,MaxRSS,Elapsed,State,ExitCode

Common reasons for a job being in PENDING state include Resources, Priority, QOSGrpCpuLimit, and QOSGrpMemLimit. The last two indicate that the aggregate long-partition QOS limit is currently reached. A job requesting a large scratch allocation can also remain pending under a general resource-related reason until a node has sufficient free scratch capacity.

Interactive jobs

Request an interactive shell with srun (add option --cluster=bioslurm in the SLURM_CLUSTERS variable is not set):

# Ordinary CPU session, up to 4 hours
srun -N 1 -n 1 --cpus-per-task=4 --mem=16G --gres=scratch:100 -t 04:00:00 -p short --pty bash -l

# Short debugging session
srun -N 1 -n 1 --cpus-per-task=2 --mem=8G --gres=scratch:50 -t 01:00:00 -p debug --pty bash -l

# One-GPU interactive session
srun -N 1 -n 1 --cpus-per-task=4 --mem=32G -t 04:00:00 -p gpu \
     --gres=gpu:tP100:1,scratch:100 --pty bash -l

The command waits until resources are available. Add --immediate=60 to give up if the allocation cannot be obtained within 60 seconds.

Array jobs

sbatch --cluster=bioslurm --array=0-30 --gres=scratch:100 myscript.sh

This submits 31 array elements. Each array element receives its own scratch allocation and its own $SLURM_TMPDIR. Inside the script, use $SLURM_ARRAY_TASK_ID to select the input, output, or parameters for each element.

Limit simultaneous elements with %N:

sbatch --cluster=bioslurm --array=0-30%4 --gres=scratch:100 myscript.sh

The cluster permits arrays of up to 100000 elements; valid indices therefore normally range from 0 through 99999. Avoid submitting huge numbers of very short tasks. Bundle tasks lasting only seconds into fewer, longer jobs.

Containers

When a container runtime is used inside a SLURM allocation, ensure that the container itself does not escape the CPU and memory allocation. For a job allocated four CPUs and 42 GB, a Docker-style invocation should use somewhat less than the full memory allocation:

docker1 run --cpus=4 --memory=40g IMAGE COMMAND

The exact invocation may depend on the container tool and image. GPU containers must also be launched in a GPU allocation and configured to expose only the GPUs assigned through CUDA_VISIBLE_DEVICES.

Other considerations

Environment variables

Variable	Meaning
SLURM_JOB_ID	Unique job ID.
SLURM_TMPDIR	Preferred variable for the automatically created, quota-controlled local scratch directory for this job.
JOB_SCRATCH	Alias of SLURM_TMPDIR.
TMPDIR, TMP, TEMP	Additional aliases pointing to the same per-job local scratch directory.
SLURM_JOB_NODELIST	Node or nodes allocated to the job.
SLURM_NTASKS	Number of tasks requested.
SLURM_CPUS_PER_TASK	CPUs allocated per task when requested.
SLURM_MEM_PER_NODE	Memory requested per node with --mem, expressed in MB.
SLURM_ARRAY_JOB_ID	Master ID of an array job.
SLURM_ARRAY_TASK_ID	Index of the current array element.
CUDA_VISIBLE_DEVICES	GPU IDs made visible to a GPU job.

Backups and scratch cleanup

Local scratch is temporary and is deleted automatically at job end. It is not backed up. Keep irreplaceable data in an approved permanent storage location and copy final results out of $SLURM_TMPDIR before the job exits.

Software

Software installed for the BioHPC environment is generally available on cluster nodes. Use the established BioHPC software setup procedures and submit installation requests through the normal support channel when system-wide installation is needed.

Help

For technical problems, include the cluster name (bioslurm), job ID, submission command or script, relevant output, and the approximate time of the problem when contacting BioHPC support at support@biohpc.cornell.edu.

---

This cluster-specific guide was adapted from the BioHPC SLURM-on-demand documentation and the active bioslurm configuration supplied by the administrator.

 

Website credentials:

login

Web Accessibility Help