bioslurm_hands

BioSlurm Training Workshop

Hands-on Project

For this workshop, the AI agent will help you create and submit SLURM jobs, check their status, troubleshoot problems, and adjust computing resources.

About the prompts in this workshop

Throughout this workshop, we provide example prompts that you can copy and use with the AI agent. These are suggested prompts, not commands with fixed syntax.

You do not need to memorize or reproduce the exact wording. You can describe what you want in your own words, ask follow-up questions, change your request, or ask the agent to explain something you do not understand.

Think of the prompts as examples of what information to give the agent and what questions to ask, rather than syntax that must be followed literally.

As you become more comfortable working with AI agents, you will likely use shorter prompts and develop the analysis through conversation.

Connect to BioHPC and start an AI Agent

Set up Codex or Claude Code

If this is your first time using Codex or Claude Code on BioHPC, follow the appropriate setup instructions:

Codex: Codex setup instructions

Claude Code: Claude Code setup instructions

Connect to BioHPC


ssh your_user_id@cbsulogin.biohpc.cornell.edu

You can also connect using VS Code Remote SSH.

Create a directory for the workshop


xxxxxxxxxx
mkdir -p /home2/$USER/bioslurm_workshop

Start your AI agent in the project directory


x
cd /home2/$USER/bioslurm_workshop

#ChatGPT user
codex

#Claude user
claude

Load BioSlurm.md
This file provides the information that AI Agent should know about BioSlurm.


xxxxxxxxxx
Read /programs/ai_pipelines/slurm/bioslurm.md

For this workshop, you only have access to computing nodes in the training partition of BioSlurm.


xxxxxxxxxx
Use only the training partition.

Exercise 1. Explore BioSlurm

1.1 Do I have access to BioSlurm?

1.1.1 AI prompts


xxxxxxxxxx
Do I have access to BioSlurm?

1.1.2 Equivalent BASH commands


xxxxxxxxxx
sacctmgr show assoc | grep $USER

Interpretation:

Output containing your username and bioslurm indicates an account association.
No output may mean that you do not have an association.

1.2 What is happening on BioSlurm?

1.2.1 AI prompts


xxxxxxxxxx
What is the current status of BioSlurm?

What jobs are currently in the queue?

Do I have any jobs in the queue?

1.2.2 Equivalent BASH commands


xxxxxxxxxx
squeue -M bioslurm -p training

squeue -M bioslurm -p training -u $USER

Interpretation:

Displays all active BioSlurm jobs, includes both pending (PD) and running (R) jobs.

1.3 Ask the agent about the cluster

1.3.1 AI prompts


xxxxxxxxxx
What computing resources are available on BioSlurm?

What partitions are available, and when should I use each one?

The agent will explain the CPU/GPU nodes and short, long, gpu, and debug partitions from the provided instructions.

1.3.2 Some common BASH commands


xxxxxxxxxx
# Estimated start time for pending jobs
squeue -M bioslurm -p training -u "$USER" --start

# Partitions and available nodes
sinfo -M bioslurm -p training

# Detailed job information
scontrol -M bioslurm show job JOB_ID

Exercise 2. Submit your first BioSlurm job

2.1 Generate a scatter plot with R

2.1.1 AI Prompts


xxxxxxxxxx
Use R to generate 10 pairs of random numbers and make a scatter plot. Run the analysis as a BioSlurm job.

Check the status of my job.

Did the job finish successfully? Show me the output files.

The AI agent should create a SLURM batch script, then use the sbatch command to submit the job to BioSlurm.

After the agent submits the job, look in your project directory. Can you identify the files it created? Ask the agent to explain the SLURM batch script and the resources requested for the job.

You can also ask:


xxxxxxxxxx
Check the status of my job.

Did the job finish successfully? What output files were created?

Exercise 3. Parallelize a Bioinformatics Analysis

3.1 Align multiple samples to a reference genome

In this exercise, we have FASTQ files from three samples. Each sample can be aligned to the reference genome independently, making this analysis suitable for parallel execution.

3.1.1 AI Prompts


xxxxxxxxxx
Create a project directory at /home2/$USER/alignments and copy all files from /shared_data/bioslurm_ws/sequences into it.

Align the FASTQ files from all samples to the reference genome using BWA-MEM. Run the sample alignments in parallel on BioSlurm.

Do not submit the jobs yet. Show me your proposed approach and resource requests first.

Then ask Agent:


xxxxxxxxxx
How are you parallelizing this analysis?

The agent may create multiple independent SLURM jobs or use a SLURM job array. Ask it to explain the approach it selected.

Also examine the requested CPU, memory, scratch space, and runtime. For real analyses, these requirements are not always known precisely in advance. The AI agent can estimate them, but you may need to adjust the requests based on your knowledge of the data and previous jobs.

Then let the agent submit the jobs and monitor their progress.

Exercise 4. Request Specific Computing Resources

Some analyses require specific hardware or computing resources. For example, GPU software may require a particular type of GPU.

BioSlurm currently has P100 and A40 GPUs, and a GPU job must explicitly request a GPU resource.

4.1 Run a GPU job

4.1.1 AI prompts


xxxxxxxxxx
On BioSlurm, create and submit a small GPU test job in /home2/$USER/gpu_test.

Use the gpu partition and request exactly one P100 GPU, one CPU, 1 GB of memory, and five minutes. Run NVIDIA's CUDA vector addition example.

After the agent submits the job, ask:


xxxxxxxxxx
Show me the SLURM resource requests you used. Explain what each one does.

Then perhaps:


xxxxxxxxxx
Which BioSlurm nodes could this job run on?

Exercise 5. Run a Nextflow Workflow on BioSlurm

Nextflow is a workflow management system that can run multi-step analyses and submit individual tasks to a SLURM cluster.

In this exercise, we will use the sequencing data from Exercise 3 for variant calling.

The variant-calling workflow should:

align reads to the reference genome using BWA-MEM;
generate variant likelihoods using bcftools mpileup;
call variants using bcftools call;
use an appropriate nf-core pipeline.

It is OK if you do not remember the technical details of how to perform variant calling. The Bioinformatics Facility is working on curating and developing commonly used data-analysis workflows as AI skills. Once these skills are available, you will be able to load the appropriate skill and ask the AI agent to perform the analysis following the curated workflow.

For this exercise, the variant-calling skill is not provided. Instead, we will give the AI agent the necessary workflow instructions directly in the prompt. In other words, you will provide the "skill" through the prompt..

5.1 Ask the AI agent to set up the workflow

5.1.1 AI prompt


xxxxxxxxxx
- Using the sequencing data in /home2/$USER/alignments, perform variant calling using an appropriate nf-core Nextflow pipeline. - Use BWA-MEM for read alignment, followed by bcftools mpileup and bcftools call for variant calling.
- Run the workflow on BioSlurm. Nextflow itself should run in a small BioSlurm job and submit the individual analysis tasks as BioSlurm jobs.
- Do not start the analysis yet. First identify the appropriate nf-core pipeline and show me your proposed workflow.

Then ask


xxxxxxxxxx
Explain how this nf-core pipeline implements the variant-calling steps I requested.

Review the proposed workflow with the AI agent. Ask questions about any steps, parameters, or resource requests that you do not understand. You can also ask the agent to modify the plan if necessary.

Once you are satisfied with the plan, tell the agent:


xxxxxxxxxx
Go ahead and run the workflow.