Quick start

Prerequisites

• You need a BioHPC account.
• You must belong to a BioHPC membership credit account. To check your membership accounts, visit My Credit Accounts page on the BioHPC website.
  • If you do not have an account with Type=membership, you can create one by clicking the New Credit Account link.
  • Once you have a membership account, you can purchase an annual or monthly membership by clicking on the corresponding 'Renew' link on the My Credit Accounts page. Alternatively, if you have a separate hourly credit account with compute units, it is possible to convert compute units into membership credit, using the 'Transfer hours between accounts' link at the bottom of My Credit Accounts.

Run your first Slurm job

A batch job is a command or script that Slurm runs for you on one of the cluster's compute nodes.

To start, log into cbsulogin.biohpc.cornell.edu:

ssh YOUR_USER_ID@cbsulogin.biohpc.cornell.edu

Make a working directory, change into it, and create a batch file called hello.sh:

mkdir bioslurm_tutorial
cd bioslurm_tutorial
nano hello.sh

You are now in the nano text editor. Paste the following into the file, save, and exit nano:

#!/bin/bash -l 
#SBATCH --cluster=bioslurm
#SBATCH --job-name=hello
#SBATCH --output=hello.%j.out
#SBATCH --time=00:05:00
#SBATCH --mem=1G
              
echo "Hello from BioSlurm!"
echo "Job started at $(date)"
echo "SLURM_JOB_ID = $SLURM_JOB_ID"
echo "HOSTNAME = $HOSTNAME"
echo "SLURM_TMPDIR = $SLURM_TMPDIR"

sleep 60
echo "Job finished at $(date)"

Submit it:

sbatch hello.sh 

You should see:

Submitted batch job 12345 

Where 12345 is replaced with your SLURM_JOB_ID. Check job status:

squeue -u $USER --cluster bioslurm

Once the job is finished, it will no longer show up with the squeue command, but you can see past jobs with:

sacct_l --cluster bioslurm

The job output will appear in the file hello.12345.out. View the output:

cat hello.12345.out 

What just happened: when you submitted the job with sbatch hello.sh, Slurm located a server with the required resources. It assigned the server to your job (printed to output as $HOSTNAME), created a working directory for you at $SLURM_TMPDIR (which is located at /workdir/$USER_ID/$SLURM_JOB_ID). This directory exists for you on the assigned node, during the time your job is running. It then ran all the commands in your script. The job had a 5 minute time limit, after which it would be cancelled. But since the only commands are to print some information and sleep for 60 seconds, the job should successfullly finish after 1 minute. The output file hello.12345.out (where 12345 is the $SLURM_JOB_ID) contains any text printed to stdout or stderr while the job runs.

Example analysis job

Here is an example script that shows how you would copy files to your working directory, run some analysis, and copy results back to your home directory.

Note: Never run analysis directly on files stored in /home, you should always first copy the files to $SLURM_TMPDIR. It is acceptable to run analysis directly on files in /home2, but it still may be more efficient to first copy them to $SLURM_TMPDIR first.

#!/bin/bash -l 
#SBATCH --cluster=bioslurm
#SBATCH --job-name=myanalysis
#SBATCH --output=myanalysis.%j.out
#SBATCH --cpus-per-task=4
#SBATCH --mem=16G
#SBATCH --time=04:00:00
#SBATCH --gres=scratch:100

### Change to your working directory which is local to your assigned compute node
cd "$SLURM_TMPDIR"

### Copy input files you need to your working directory
cp ~/myproject/input.bam .

### Run your analysis
samtools index input.bam

### Copy results back to your home directory before the job exits
cp input.bam.bai ~/myproject/

Submit:

sbatch myanalysis.sh 

Try an interactive job

Interactive jobs are useful for debugging. This command starts a short shell session on a compute node:

srun --cluster=bioslurm --time=01:00:00 --pty bash -l

When resources are available, Slurm assigns a server to you and logs you into an interactive shell on that server. Run hostname to see its name and exit to end the session (and cancel the job). Your working directory will still be created, you can go to it with cd $SLURM_TMPDIR.

srun accepts the same options as sbatch, see the next section for a quick overview, and Interactive jobs for more examples.

Using bioslurm with AI agents

Inform the agent you intend to use bioslurm and instruct it to read the file bioslurm.md available form BioHPC in directory /programs/ai_pipelines/slurm. If the agent is running on one of the BioHPC servers, this file is automatically sourced in as long as the BioHPC-provided file /programs/ai_pipelines/AGENTS.md or /programs/ai_pipelines/CLAUDE.md is used. Otherwise, please copy bioslurm.md to where it can be reached by the agent. The file contains agent-friendly description of the cluster and instructions on how to create proper submission scripts. Followiong these instructions, the agent will create a script and present it to you for inspection and approval before submitting.

Useful sbatch parameters

These parameters may be specified at the top of the script with #SBATCH notation, or as arguments to the sbatch command.

Reference guide

Cluster hardware

The cluster bioslurm consists of 5 CPU compute nodes and 3 GPU nodes:

On cbsugpu02, eight CPU threads are intentionally left outside Slurm for controller and system services.

Storage

Home directories

Home directories under /home are on networked storage. Do not run I/O-intensive computations directly in a home directory. Home is suitable for scripts, small configuration files, logs, and final results, but not for repeatedly reading or writing large working datasets.

Local scratch storage

Every bioslurm job receives a private directory on node-local scratch storage. Request the desired capacity in GB with the Slurm GRES option --gres=scratch:N. For example, the following requests 200 GB:

#SBATCH --gres=scratch:200

If --gres=scratch:N is omitted, the job receives 50 GB by default .

At job start, a directory is automatically created as:

/workdir/<userID>/<JobID>

The same directory is exposed inside the job through all of these environment variables:

$SLURM_TMPDIR
$JOB_SCRATCH
$TMPDIR
$TMP
$TEMP

SLURM_TMPDIR is the preferred name to use in scripts. The scratch directory is the only location on the compute node that is writable by jobs. User home directories, including /home2, remain accessible for reading and writing as usual, but they reside on network storage rather than on the local node.

Using local scratch efficiently

For best performance, jobs should perform most input/output operations in the node-local directory identified by $SLURM_TMPDIR. A typical workflow is:

#!/bin/bash -l
#SBATCH --gres=scratch:200

# Move to the node-local scratch directory
cd "$SLURM_TMPDIR"

# Copy input data from permanent storage
cp ~/project/input.fastq.gz .

# Run the analysis
my_program input.fastq.gz

# Copy results back before the job exits
cp results.txt ~/project/results/

This pattern keeps I/O-intensive temporary and intermediate files on local disks while preserving only the files that are needed after the job completes.

When several jobs use the same input: /home2

If multiple jobs need to read the same input data, placing that data in /home2/<userID> can avoid copying an identical large dataset into each job's private scratch directory. The /home2 filesystem is mounted on all BioSlurm compute nodes and is accessible from within jobs.

For example, a shared reference can remain on /home2 while each job writes temporary files and intermediate output to its own local scratch:

# Shared input/reference data
reference=/home2/$USER/genomes/hg38.fa

# Per-job local working directory
cd "$SLURM_TMPDIR"

my_program --reference "$reference" --input input.fastq.gz

As a general rule, perform computation and temporary I/O in $SLURM_TMPDIR. Copy input data into scratch when practical, and copy final results back to permanent storage before exit. Use /home2 when sharing the same input across jobs avoids needless duplication, while keeping in mind the limitations of network I/O.

Partitions and limits

If --partition is omitted, the job goes to short. If --time is omitted, the applicable partition maximum is used. The default memory request is 4096 MB per job on all partitions.

The debug partition has higher scheduling priority than the ordinary CPU partitions, but it is limited to one hour. It should be used for genuine short tests, not production workloads.

Submitting batch jobs

Create a shell script such as submit.sh. A typical CPU job is:

#!/bin/bash -l
#SBATCH --cluster=bioslurm
#SBATCH --ntasks=8
#SBATCH --mem=8G
#SBATCH --gres=scratch:200
#SBATCH --time=04:00:00
#SBATCH --partition=short
#SBATCH --chdir=/home/NETID/slurm
#SBATCH --job-name=myjob
#SBATCH --output=myjob.%j.out
#SBATCH --mail-user=NETID@cornell.edu
#SBATCH --mail-type=END,FAIL

set -euo pipefail

cd "$SLURM_TMPDIR"

# Copy input to $SLURM_TMPDIR, run the program here,
# and copy final output back to permanent storage before exit.

# Create an empty 'marker file' to tell Slurm epilog that the script completed.
touch "$SLURM_TMPDIR/.job_completed_ok"

Submit it with:

sbatch submit.sh

Options may instead be placed on the command line:

sbatch --cluster=bioslurm -N 1 -n 8 --mem=8G --gres=scratch:200 -t 04:00:00 -p short submit.sh

A command-line option overrides the same option in the script header.

If the job finishes prematurely, the file.job_completed_ok will be absent from the job working directory. In such a case, all files *.log, *.out, and *.err will be automatically copied to the directory from which the job was submitted (or to the one specified by the --chdir option), to the subfolder slurm-logs (if absent, this subfolder will be created). The saved log files can then be used for debugging purposes.

The option --cluster=bioslurm may be omitted if the environment variable SLURM_CLUSTERS is defined in a shell from which a job is being submitted:

export SLURM_CLUSTERS=bioslurm

The above command can be inserted into the user's .bashrc file.

Choosing a partition

# CPU job no longer than 24 hours
#SBATCH --partition=short

# CPU job longer than 24 hours, up to 7 days
#SBATCH --partition=long

# Short test, no longer than 1 hour
#SBATCH --partition=debug

# GPU job
#SBATCH --partition=gpu

CPU, memory, and node requests

One-node threaded programs

Most BioHPC applications are either serial or multithreaded but not MPI-enabled. Keep such jobs on one node:

#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=16
#SBATCH --mem=64G
#SBATCH --gres=scratch:200

export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK

For software whose command-line option specifies a thread count, pass $SLURM_CPUS_PER_TASK or the same numeric value.

Independent processes

For eight independent single-threaded tasks, request:

#SBATCH --cluster=bioslurm
#SBATCH --nodes=1
#SBATCH --ntasks=8
#SBATCH --cpus-per-task=1
#SBATCH --gres=scratch:100

MPI jobs

Only MPI-aware software can use multiple nodes as one parallel job. Request the required number of tasks and nodes, then launch through srun or the application-specific MPI procedure.

Specific nodes and features

# Run specifically on cbsulm14
#SBATCH --nodelist=cbsulm14

# Request a node carrying the avx2 feature
#SBATCH --constraint=avx2

# Avoid selected nodes
#SBATCH --exclude=cbsulm09,cbsulm10

Scratch syntax

# Request 200 GB of local scratch
#SBATCH --gres=scratch:200

# GPU plus 200 GB local scratch
#SBATCH --gres=gpu:tP100:1,scratch:200

If the scratch option is omitted, 50 GB is allocated by default. Scratch is a consumable node resource: a job may wait for a node that has enough free scratch capacity even if CPU and memory are otherwise available.

Memory syntax

# Total memory for the entire job
#SBATCH --mem=48G

# Memory per allocated CPU
#SBATCH --mem-per-cpu=2G

For example, --ntasks=8 --cpus-per-task=3 --mem-per-cpu=2G requests 24 CPUs and 48 GB total.

Checking actual memory use

After a job finishes, inspect its requested and peak resident memory (relpace JOBID with the actual ID of your job):

sacct --cluster=bioslurm -j JOBID --format=JobID,JobName,Partition,ReqCPUS,ReqMem,MaxRSS,Elapsed,State,ExitCode

Use the .batch line for the batch script's peak memory. Request somewhat more than observed MaxRSS for future jobs with similar data and parameters.

GPU jobs

The gpu partition provides three Tesla P100 GPUs: one on cbsugpu02 and two on cbsugpu03, as well as two (newer) NVIDIA A40 GPUs on cbsugpu06. Scratch and GPU resources can be requested together in one comma-separated --gres option. A one-GPU job can use either node:

#!/bin/bash -l
#SBATCH --cluster=bioslurm
#SBATCH --partition=gpu
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=8
#SBATCH --mem=64G
#SBATCH --gres=gpu:tP100:1,scratch:200
#SBATCH --time=12:00:00
#SBATCH --output=gpu.%j.out

nvidia-smi
python my_gpu_program.py

A job requiring two GPUs must run on cbsugpu03:

#SBATCH --partition=gpu
#SBATCH --nodelist=cbsugpu03
#SBATCH --gres=gpu:tP100:2,scratch:200

on cbsugpu06:

#SBATCH --partition=gpu
#SBATCH --nodelist=cbsugpu06
#SBATCH --gres=gpu:A40:2,scratch:200

Within a correctly launched job, Slurm sets CUDA_VISIBLE_DEVICES to the GPU or GPUs assigned to that job. Do not override this variable unless you understand the consequences.

Monitoring and controlling jobs

If the SLURM_CLUSTERS variable is not set, add option --cluster=bioslurm to each of the sinfo, scontrol, squeue, scancel, and sacct commands below:

# Partition and node summary
sinfo

# Detailed partition information
scontrol show partitions

# Detailed node information
scontrol show nodes

# Your jobs
squeue -u "$USER"

# Explain why a pending job is waiting
squeue -j JOBID -o "%.18i %.9P %.8j %.8u %.2t %.10M %.6D %R"
scontrol show job JOBID

# Cancel a job
scancel JOBID

# Accounting information after or during a job
sacct -j JOBID --format=JobID,Partition,AllocCPUS,ReqMem,MaxRSS,Elapsed,State,ExitCode

Common reasons for a job being in PENDING state include Resources, Priority, QOSGrpCpuLimit, and QOSGrpMemLimit. The last two indicate that the aggregate long-partition QOS limit is currently reached. A job requesting a large scratch allocation can also remain pending under a general resource-related reason until a node has sufficient free scratch capacity.

Interactive jobs

Request an interactive shell with srun (add option --cluster=bioslurm in the SLURM_CLUSTERS variable is not set):

# Ordinary CPU session, up to 4 hours
srun -N 1 -n 1 --cpus-per-task=4 --mem=16G --gres=scratch:100 -t 04:00:00 -p short --pty bash -l

# Short debugging session
srun -N 1 -n 1 --cpus-per-task=2 --mem=8G --gres=scratch:50 -t 01:00:00 -p debug --pty bash -l

# One-GPU interactive session
srun -N 1 -n 1 --cpus-per-task=4 --mem=32G -t 04:00:00 -p gpu \
     --gres=gpu:tP100:1,scratch:100 --pty bash -l

The command waits until resources are available. Add --immediate=60 to give up if the allocation cannot be obtained within 60 seconds.

Array jobs

sbatch --cluster=bioslurm --array=0-30 --gres=scratch:100 myscript.sh

This submits 31 array elements. Each array element receives its own scratch allocation and its own $SLURM_TMPDIR. Inside the script, use $SLURM_ARRAY_TASK_ID to select the input, output, or parameters for each element.

Limit simultaneous elements with %N:

sbatch --cluster=bioslurm --array=0-30%4 --gres=scratch:100 myscript.sh

The cluster permits arrays of up to 100000 elements; valid indices therefore normally range from 0 through 99999. Avoid submitting huge numbers of very short tasks. Bundle tasks lasting only seconds into fewer, longer jobs.

Containers

When a container runtime is used inside a Slurm allocation, ensure that the container itself does not escape the CPU and memory allocation. For a job allocated four CPUs and 42 GB, a Docker-style invocation should use somewhat less than the full memory allocation:

docker1 run --cpus=4 --memory=40g IMAGE COMMAND

The exact invocation may depend on the container tool and image. GPU containers must also be launched in a GPU allocation and configured to expose only the GPUs assigned through CUDA_VISIBLE_DEVICES.

Other considerations

Environment variables

Backups and scratch cleanup

Local scratch is temporary and is deleted automatically at job end. It is not backed up. Keep irreplaceable data in an approved permanent storage location and copy final results out of $SLURM_TMPDIR before the job exits.

Software

Software installed for the BioHPC environment is generally available on cluster nodes. Use the established BioHPC software setup procedures and submit installation requests through the normal support channel when system-wide installation is needed.

Help

For technical problems, include the cluster name (bioslurm), job ID, submission command or script, relevant output, and the approximate time of the problem when contacting BioHPC support at support@biohpc.cornell.edu.

This cluster-specific guide was adapted from the BioHPC Slurm-on-demand documentation and the active bioslurm configuration supplied by the administrator.