BioHPC Cloud:
: User Guide

Part1. Run pipeline with Singularity (the easiest) 

Here is how to run test data set provided by the pipeline developer (run software in "screen" persistent session)

export PYTHONPATH=/programs/caper/lib/python3.6/site-packages:/programs/caper/lib64/python3.6/site-packages
export PATH=/programs/caper/bin:$PATH

mkdir /workdir/$USER
cd /workdir/$USER/
cp -r /programs/atac-seq-pipeline-1.9.3 /workdir/$USER

#download test data set
wget https://storage.googleapis.com/encode-pipeline-test-samples/encode-atac-seq-pipeline/ENCSR356KRQ_subsampled_caper.json

#process test dataset
export ATACROOT=/workdir/$USER/atac-seq-pipeline-1.9.3

caper run $ATACROOT/atac.wdl -i ENCSR356KRQ_subsampled_caper.json --singularity --singularity-cachedir $ATACROOT

If no --output directory specified, the output directory is atac. The result files under atac, in execution directory, the files are documented in https://encode-dcc.github.io/wdl-pipelines/output_atac.html.

# After the work is finished, organize output results with croo

export PYTHONPATH=/programs/caper/lib/python3.6/site-packages:/programs/caper/lib64/python3.6/site-packages
export PATH=/programs/caper/bin:$PATH
cd atac
ls -l 
cd xxxxxxx  #replace xxxxxxx with the run directory you get from "ls -l"
croo metadata.json
qc2tsv qc/qc.json  > qc.tsv

The results should be in the directories "peak" "qc" and "signal" , report files croo*, and QC table qc.tsv 

When you run you real data, you need to 

1)prepare reference genome file.

• in the directory "atac-seq-pipeline-1.9.3", there is a script "build_genome_data.sh". Edit this script, including "YOUR_OWN_GENOME" ,  REGEX_BFILT_PEAK_CHR_NAME=".*"(optional), MITO_CHR_NAME="chrM" (if not known, remove "chrM" include the two quotation marks),  REF_FA="https://some.where.com/your.genome.fa.gz"(change to full file path of the genome fasta file", BLACKLIST= (add path if available, otherwise keep empty). 
• run these commands to create genome database. (after done, you should see a directory /workdir/$USER/genomedb with a .tsv file)

source /home/$USER/miniconda3/bin/activate encode-atac-seq-pipeline

/workdir/$USER/atac-seq-pipeline-1.9.3

./build_genome_data.sh YOUR_OWN_GENOME /workdir/$USER/genomedb

* I modified this script build_genome_data.sh to work with local genome fasta file. If you are working with human/mouse data, see

 https://github.com/ENCODE-DCC/atac-seq-pipeline/blob/master/docs/build_genome_database.md

2) prepare input json file.

 https://github.com/ENCODE-DCC/atac-seq-pipeline  (section under "Input JSON file")

3) Number of cpu per task is specified in the file atac.wdl. In most cases, there is no need to change (for example, default setting for aligner bowtie2 is 6 cores per job). However, you might want to restrict number of simultaneously jobs when running the caper command (e.g. limit to up to 4 jobs, --max-concurrent-tasks 4), otherwise, all available cores will be used.

Part 2. Install/runpipeline with conda (alternative way)

Here are the instructions to install the pipeline in conda. (It could take 2 hours, run these steps in "screen" persistent session). I also provided below how to run pre-installed v1.7.0.

1. Install Conda if you do not have it in your home directory. You can follow this page to install conda https://biohpc.cornell.edu/lab/doc/Software_Installation_exercises1.html (part 1).

2. Install the latest version atac-seq-pipeline.(there is an issue with bowtie2 in Conda right now 2021/05/03. A temporary fix is to downgrade tbb after installation. I included this step in the commands below. I will remove this step once Conda fixed the bowtie2 problem).

source /home/$USER/miniconda3/bin/activate
mkdir /workdir/$USER
cd /workdir/$USER
git clone https://github.com/ENCODE-DCC/atac-seq-pipeline.git
bash scripts/uninstall_conda_env.sh
bash scripts/install_conda_env.sh mamba
conda activate encode-atac-seq-pipeline
conda install tbb=2020.2
pip install caper croo

#keep a copy of the atac.wdl file in your home directory. This file might be useful later
cp -r /workdir/$USER/atac-seq-pipeline/atac.wdl /home/$USER/ 

3. Run software through cromwell on a local computer.

#set environment

source /home/$USER/miniconda3/bin/activate encode-atac-seq-pipeline

# run test data.

The cpu core per task can be set in the atac.wdl file (under "# group: resource_parameter"). Maximum number of concurrent tasks can be restricted by the parameter max-concurrent-tasks.

cd /workdir/$USER
wget https://storage.googleapis.com/encode-pipeline-test-samples/encode-atac-seq-pipeline/ENCSR356KRQ_subsampled_caper.json

caper run atac.wdl -i ENCSR356KRQ_subsampled_caper.json v

*Here are the instructions to run version 1.7.0 (4/2020) installed on BioHPC, and process a test data set. Run the software using "screen". It could take 30 min for the test data. (Instructions to install latest version is below)

## Activate the conda environment

source /programs/miniconda3/bin/activate encode-atac-seq-pipeline

##Test run

cd /workdir/$USER
cp /programs/atac-seq-pipeline-1.7.0/atac.wdl .

wget https://storage.googleapis.com/encode-pipeline-test-samples/encode-atac-seq-pipeline/ENCSR356KRQ_subsampled_caper.json

caper run ./atac.wdl -i ENCSR356KRQ_subsampled_caper.json

##When working with your own data, you need to create your own .json file. Please read the documentation, https://github.com/ENCODE-DCC/atac-seq-pipeline, under the section "Input JSON file"

## qc2tsv (https://github.com/ENCODE-DCC/qc2tsv#installation) and Croo (https://github.com/ENCODE-DCC/croo#installation) under the same conda environment 

To run latest version of atac-seq pipeline, please install the software in your home directory. We are not able to maintain multiple different versions simultaneously.