institute of biotechnology >> brc >> bioinformatics >> internal >> biohpc cloud: user guide
 

BioHPC Cloud:
: User Guide

 


BioHPC Cloud Software

There is 928 software titles installed in BioHPC Cloud. The sofware is available on all machines (unless stated otherwise in notes), complete list of programs is below, please click on a title to see details and instructions. Tabular list of software is available here

Please read details and instructions before running any program, it may contain important information on how to properly use the software in BioHPC Cloud.

3d-dna, 454 gsAssembler or gsMapper, a5, ABRicate, ABruijn, ABySS, AdapterRemoval, adephylo, Admixtools, Admixture, agrep, albacore, Alder, AlleleSeq, ALLMAPS, ALLPATHS-LG, Alphafold, AMOS, AMPHORA, amplicon.py, AMRFinder, analysis, ANGSD, Annovar, ant, antiSMASH, anvio, apollo, arcs, ARGweaver, Arlequin, ART, aspera, assembly-stats, ASTRAL, atac-seq-pipeline, ataqv, athena_meta, ATLAS, Atlas-Link, ATLAS_GapFill, atom, ATSAS, Augustus, AWS command line interface, AWS v2 Command Line Interface, axe, axel, BactSNP, bakta, bam2fastx, bamtools, bamUtil, barcode_splitter, BarNone, Basset, BayeScan, Bayescenv, baypass, bazel, BBMap/BBTools, BCFtools, bcl2fastq, BCP, Beagle, Beast2, bedops, BEDtools, bfc, bgc, bgen, bigQF, bigWig, bioawk, biobakery, biobambam, Bioconductor, biom-format, BioPerl, BioPython, Birdsuite, Bismark, Blackbird, blasr, BLAST, BLAST_to_BED, blast2go, BLAT, BlobToolKit, BLUPF90, BMGE, bmtagger, bonito, Boost, Bowtie, Bowtie2, BPGA, Bracken, BRAKER, BRAT-NextGen, BRBseqTools, BreedingSchemeLanguage, breseq, brocc, BSseeker2, BUSCO, BWA, bwa-mem2, bwa-meth, bwtool, cactus, CAFE, canu, CAP3, caper, CarveMe, catch, cBar, CBSU RNAseq, CCMetagen, CCTpack, cd-hit, cdbfasta, CEGMA, CellRanger, cellranger-arc, cellranger-atac, cellranger-dna, centrifuge, centroFlye, CFM-ID, CFSAN SNP pipeline, CheckM, chimera, chip-seq-pipeline, chromosomer, Circlator, Circos, Circuitscape, CITE-seq-Count, ClermonTyping, clues, CLUMPP, clust, Clustal Omega, CLUSTALW, Cluster, cmake, CMSeq, CNVnator, compat, CONCOCT, Conda, Cooler, copyNumberDiff, cortex_var, CoverM, CRISPRCasFinder, CRISPResso, CrossMap, CRT, cuda, Cufflinks, cutadapt, cuteSV, dadi, dadi-1.6.3_modif, danpos, dDocent, DeconSeq, Deepbinner, DeepTE, deepTools, defusion, delly, DESMAN, destruct, DETONATE, diamond, diploSHIC, discoal, Discovar, Discovar de novo, distruct, DiTASiC, DIYABC, Docker, dREG, dREG.HD, drep, drive, Drop-seq, dropEst, dropSeqPipe, dsk, dssat, Dsuite, dTOX, duphold, dynare, ea-utils, ecopcr, ecoPrimers, ectyper, EDGE, edirect, EDTA, eems, EgaCryptor, EGAD, EIGENSOFT, elai, EMBOSS, EMIRGE, Empress, entropy, epa-ng, ephem, epic2, ermineJ, ete3, EVM, exabayes, exonerate, ExpansionHunterDenovo-v0.8.0, eXpress, FALCON, FALCON_unzip, Fast-GBS, fasta, FastANI, fastcluster, FastME, FastML, fastp, FastQ Screen, fastq_pair, fastq_species_detector, FastQC, fastqsplitter, fastsimcoal2, fastspar, fastStructure, FastTree, FASTX, feems, feh, FFmpeg, fgbio, fineRADstructure, fineSTRUCTURE, FIt-SNE, flash, flash2, flexbar, Flexible Adapter Remover, Flye, FMAP, FragGeneScan, FragGeneScan, freebayes, FSA, funannotate, FunGene Pipeline, G-PhoCS, GADMA, GAEMR, Galaxy in Docker, Galaxy Server, GATK, gatk4, gatk4amplicon.py, gblastn, Gblocks, GBRS, gcc, GCTA, GDAL, gdc-client, GEM library, GEMMA, GENECONV, geneid, GeneMark, GeneMarker, Genome STRiP, GenomeMapper, GenomeStudio (Illumina), GenomeThreader, genometools, GenomicConsensus, genozip, gensim, GEOS, germline, gerp++, GET_PHYLOMARKERS, GffCompare, gffread, giggle, git, glactools, GlimmerHMM, GMAP/GSNAP, GNU Compilers, GNU parallel, go-perl, GO2MSIG, GoShifter, gradle-4.4, graftM, grammy, GraPhlAn, graphtyper, graphviz, GRiD, Grinder, GROMACS, GroopM, GSEA, gsort, GTDB-Tk, GTFtools, Gubbins, GUPPY, hail, HapCompass, HAPCUT, HAPCUT2, hapflk, HaploMerger, Haplomerger2, haplostrips, HapSeq2, HarvestTools, haslr, hdf5, hget, hh-suite, HiC-Pro, HiCExplorer, HISAT2, HMMER, Homer, HOTSPOT, HTSeq, htslib, https://github.com/CVUA-RRW/RRW-PrimerBLAST, humann, HUMAnN2, hyperopt, HyPhy, hyphy-analyses, iAssembler, IBDLD, idba, IDBA-UD, IDP-denovo, idr, idseq, IgBLAST, IGoR, IGV, IMa2, IMa2p, IMAGE, ImageJ, ImageMagick, Immcantation, impute2, IMSA-A, INDELseek, infernal, Infomap, inStrain, InStruct, Intel MKL, InteMAP, InterProScan, ipyrad, IQ-TREE, iRep, jags, Jane, java, jbrowse, JCVI, jellyfish, JoinMap, juicer, julia, jupyter, kallisto, Kent Utilities, keras, khmer, kinfin, king, KmerFinder, KmerGenie, kraken, KrakenTools, KronaTools, kSNP, kWIP, LACHESIS, lammps, LAST, lastz, lcMLkin, LDAK, LeafCutter, leeHom, lep-anchor, Lep-MAP3, lftp, Liftoff, Lighter, LinkedSV, LINKS, LocARNA, LocusZoom, lofreq, longranger, LS-GKM, LTR_retriever, LUCY, LUCY2, LUMPY, lyve-SET, MACE, MACS, MaCS simulator, MACS2, maffilter, MAFFT, mafTools, MAGeCK, MAGeCK-VISPR, Magic-BLAST, magick, MAKER, mapDamage, MAQ, MARS, MASH, mashtree, Mashtree, MaSuRCA, MATLAB, Matlab_runtime, Mauve, MaxBin, MaxQuant, McClintock, mccortex, mcl, MCscan, MCScanX, medusa, megahit, MeGAMerge, MEGAN, MELT, MEME Suite, MERLIN, MetaBAT, MetaCache, MetaCRAST, metaCRISPR, MetAMOS, MetaPathways, MetaPhlAn, metaron, MetaVelvet, MetaVelvet-SL, methpipe, mfeprimer, MGmapper, Migrate-n, mikado, MinCED, Minimac3, Minimac4, minimap2, mira, miRDeep2, MISO (misopy), MITObim, MiXCR, MixMapper, MKTest, mlift, mlst, MMAP, MMSEQ, MMseqs2, MMTK, modeltest, MODIStsp-2.0.5, module, moments, mono, monocle3, mosdepth, mothur, MrBayes, mrsFAST, msld, MSMC, msprime, MSR-CA Genome Assembler, msstats, MSTMap, mugsy, MultiQC, multiz-tba, MUMandCo, MUMmer, mummer2circos, muscle, MUSIC, Mutation-Simulator, muTect, MZmine, nag-compiler, nanofilt, NanoPlot, Nanopolish, nanovar, ncftp, NECAT, Nemo, Netbeans, NEURON, new_fugue, Nextflow, NextGenMap, nf-core/rnaseq, ngmlr, NGS_data_processing, NGSadmix, ngsDist, ngsF, ngsLD, NgsRelate, ngsTools, NGSUtils, NINJA, NLR-Annotator, NLR-Parser, Novoalign, NovoalignCS, nQuire, NRSA, NuDup, numactl, nvidia-docker, nvtop, Oases, OBITools, Octave, OMA, Oneflux, openmpi, openssl, OrthoFinder, orthologr, Orthomcl, pacbio, PacBioTestData, PAGIT, pal2nal, paleomix, PAML, panaroo, pandas, pandaseq, pandoc, PanPhlAn, Panseq, Parsnp, PASA, PASTEC, PAUP*, pb-assembly, pbalign, pbbam, pbh5tools, PBJelly, pblat, pbmm2, PBSuite, pbsv, PCAngsd, pcre, pcre2, PeakRanger, PeakSplitter, PEAR, PEER, PennCNV, peppro, PfamScan, pgap, PGDSpider, ph5tools, Phage_Finder, PHAST, phenopath, Phobius, PHRAPL, PHYLIP, PhyloCSF, phyloFlash, phylophlan, PhyloPhlAn2, phylophlan3, PhyML, Picard, PICRUSt2, pigz, Pilon, Pindel, piPipes, PIQ, PlasFlow, platanus, Platypus, plink, plink2, Plotly, Point Cloud Library, popbam, PopCOGenT, PopLDdecay, Porechop, poretools, portcullis, POUTINE, pplacer, PRANK, preseq, primalscheme, primer3, PrimerBLAST, PrimerPooler, prinseq, prodigal, progenomics, progressiveCactus, PROJ, prokka, Proseq2, ProtExcluder, protolite, PSASS, psmc, psutil, purge_dups, pyani, PyCogent, pycoQC, pyfaidx, pyGenomeTracks, PyMC, pymol-open-source, pyopencl, pypy, pyRAD, Pyro4, pyseer, PySnpTools, python, PyTorch, PyVCF, QIIME, QIIME2, QTCAT, Quake, Qualimap, QuantiSNP2, QUAST, quickmerge, QUMA, R, RACA, racon, rad_haplotyper, RADIS, RadSex, RagTag, rapt, RAPTR-SV, RATT, RAxML, raxml-ng, Ray, rclone, Rcorrector, RDP Classifier, REAGO, REAPR, Red, ReferenceSeeker, regenie, Relate, RelocaTE2, Repbase, RepeatMasker, RepeatModeler, RERconverge, RFMix, RGAAT, rgdal, RGI, Rgtsvm, Ribotaper, ripgrep, rJava, rMATS, RNAMMER, rnaQUAST, Rnightlights, Roary, Rockhopper, rohan, rphast, Rqtl, Rqtl2, RSEM, RSeQC, RStudio, rtfbs_db, ruby, sabre, SaguaroGW, salmon, Sambamba, samblaster, sample, SampleTracker, samplot, samtabix, Samtools, Satsuma, Satsuma2, SCALE, scanorama, scikit-learn, Scoary, scythe, seaborn, SEACR, SecretomeP, selscan, Sentieon, seqkit, SeqPrep, seqtk, Seurat, sf, sgrep, sgrep sorted_grep, SHAPEIT, SHAPEIT4, shasta, Shiny, shore, SHOREmap, shortBRED, SHRiMP, sickle, sift4g, SignalP, SimPhy, simuPOP, singularity, sinto, sistr_cmd, SKESA, skewer, SLiM, SLURM, smcpp, smoove, SMRT Analysis, SMRT LINK, snakemake, snap, SnapATAC, SNAPP, snATAC, SNeP, Sniffles, snippy, snp-sites, SnpEff, SNPgenie, SNPhylo, SNPsplit, SNVPhyl, SOAP2, SOAPdenovo, SOAPdenovo-Trans, SOAPdenovo2, SomaticSniper, sorted_grep, spaceranger, SPAdes, SPALN, SparCC, SPARTA, sqlite, SRA Toolkit, srst2, stacks, Stacks 2, stairway-plot, stampy, STAR, Starcode, statmodels, STITCH, STPGA, StrainPhlAn, strawberry, Strelka, stringMLST, StringTie, STRUCTURE, Structure_threader, Struo2, stylegan2-ada-pytorch, subread, supernova, SURPI, sutta, SV-plaudit, SVDetect, SVseq2, svtools, svtyper, SWAMP, sweed, SweepFinder, SweepFinder2, sweepsims, tabix, Taiji, Tandem Repeats Finder (TRF), tardis, TargetP, TASSEL 3, TASSEL 4, TASSEL 5, tbl2asn, tcoffee, TensorFlow, TEToolkit, TEtranscripts, texlive, TFEA, tfTarget, thermonucleotideBLAST, ThermoRawFileParser, TMHMM, tmux, Tomahawk, TopHat, Torch, traitRate, Trans-Proteomic Pipeline (TPP), TransComb, TransDecoder, TRANSIT, transrate, TRAP, treeCl, treemix, Trim Galore!, trimal, trimmomatic, Trinity, Trinotate, tRNAscan-SE, UCSC Kent utilities, UMAP, UMI-tools, Unicycler, UniRep, unitig-caller, unrar, usearch, vamb, Variant Effect Predictor, VarScan, VCF-kit, vcf2diploid, vcfCooker, vcflib, vcftools, vdjtools, Velvet, vep, VESPA, vg, ViennaRNA, VIP, viral-ngs, virmap, VirSorter, VirusDetect, VirusFinder 2, vispr, VizBin, vmatch, vsearch, vt, WASP, wget, wgs-assembler (Celera), windowmasker, Wise2 (Genewise), wombat, Xander_assembler, xpclr, yaha

Details for atac-seq-pipeline (If the copy-pasted commands do not work, use this tool to remove unwanted characters)

Name:atac-seq-pipeline
Version:2.1.1
OS:Linux
About:This pipeline is designed for automated end-to-end quality control and processing of ATAC-seq or DNase-seq data.
Added:10/27/2018 9:50:44 AM
Updated:2/24/2022 12:17:27 PM
Link:https://github.com/ENCODE-DCC/atac-seq-pipeline
Notes:

Run the pipeline in "screen" persistant session. If you have run previous version caper and atac-seq-pipeline, or you run into problems with this pipeline, delete the .caper directory to reset caper ("rm -fr $HOME/.caper")

Instructions to run v2.1.1 (for v 1.10 see below)

Run the pipeline in "screen" persistant session.

1. Install caper version 2.1.3 in your home directory. Copy atac-seq-pipeline to the server you are working on, and set environment

pip install caper==2.1.3 croo qc2tsv --upgrade

mkdir /workdir/$USER
cd /workdir/$USER
cp -r /programs/atac-seq-pipeline-2.1.1 /workdir/$USER

 

2. Prepare reference genome database

1) If you work with human and mouse data, the Encode project provides pre-prepared genome database. Go to this page , under the section for "Reference genome", you will find URL of the reference genome database. For example, "https://storage.googleapis.com/encode-pipeline-genome-data/genome_tsv/v3/hg38.tsv". You will need this URL later.

2) If you work with other species, follow the instructions below to prepare the reference genome database.

  • Edit the script "/workdir/$USER/atac-seq-pipeline-2.1.1/build_genome_data_mod.sh",  setting the values for GENOME, GENOME DEST_DIR, REF_FA, MITO_CHR_NAME, the rest of the parameters are optional.  Make sure that you need to use the full path for file name and destination directory, for example, /workdir/$USER/atac-seq-pipeline-2.1.1/genomedb, /workdir/$USER/atac-seq-pipeline-2.1.1/mygenome.fasta.  The chromsome names match between the genome files. For plant genome, you might want to create a new genome fasta file with mitochondria and chloroplast merged, and call it a mitochandria genome. 
  • Run these commands to create genome database. After it is done, you should see a directory /workdir/$USER/$DEST_DIR with a .tsv file inside. You will need this .tsv file later. 
cd /workdir/$USER/atac-seq-pipeline-2.1.1
cp /PATH/TO/your.genome.fa.gz ./
singularity exec atac-seq-pipeline.sif ./build_genome_data_mod.sh

 

3. Put your atac-seq data files ( *.fastq.gz) into the directory /workdir/$USER/atac-seq-pipeline-2.1.1

 

4. Prepare a .json text file to specify all input files, and keep it in /workdir/$USER/atac-seq-pipeline-2.1.1

  • You can modify from this example file (for local files, replace URL with file or directory name). Detailed documentation of the json file can be found in this page (The section under "Input JSON file")
     

5. Set number of cpu per task.
Optionally, you can modify the file /workdir/$USER/atac-seq-pipeline-2.1.1/atac.wdl, and change the number of cpu per task (under "group: resource_parameter"). In most cases, there is no need to change (for example, default setting for aligner bowtie2 is 6 cores per job which is good). However, you might want to restrict number of simultaneously jobs when running the caper command (e.g. limit to up to 4 jobs, --max-concurrent-tasks 4), otherwise, all available cores on the server will be used.

 

6. run pipeline

export PATH=~/.local/bin:$PATH
caper run atac.wdl -i my.json --singularity atac-seq-pipeline.sif

 

7. summarize results

cd atac
ls -l
cd xxxxxxx         #replace xxxxxxx with the run directory you get from ls -l
croo metadata.json
qc2tsv qc/qc.json > qc.tsv

The results should be in the directories "peak" "qc" and "signal" , report files croo*, and QC table qc.tsv 

 

Run pipeline with example data files

Here is how to run test data set provided by the pipeline developer (run software in "screen" persistent session)

export PATH=~/.local/bin:$PATH
export ATACROOT=/workdir/$USER/atac-seq-pipeline-2.1.1 

mkdir /workdir/$USER
cd /workdir/$USER/
cp -r /programs/atac-seq-pipeline-2.1.1 /workdir/$USER

#download test data set
wget https://raw.githubusercontent.com/ENCODE-DCC/atac-seq-pipeline/master/example_input_json/ENCSR356KRQ_subsampled.json

#process test dataset
caper run $ATACROOT/atac.wdl -i ENCSR356KRQ_subsampled.json --singularity $ATACROOT/atac-seq-pipeline.sif

If no --output directory specified, the output directory is atac. The result files under atac, in execution directory, the files are documented in https://encode-dcc.github.io/wdl-pipelines/output_atac.html.

# After the work is finished, organize output results with croo

cd atac
ls -l 
cd xxxxxxx  #replace xxxxxxx with the run directory you get from "ls -l"
croo metadata.json
qc2tsv qc/qc.json  > qc.tsv

The results should be in the directories "peak" "qc" and "signal" , report files croo*, and QC table qc.tsv 

 

Instructions to run v1.10.0

1. Copy software directory to the server you are working on, and set environment

export PYTHONPATH=/programs/caper/lib/python3.6/site-packages:/programs/caper/lib64/python3.6/site-packages
export PATH=/programs/caper/bin:$PATH
export version=1.10.0

mkdir /workdir/$USER
cd /workdir/$USER
cp -r /programs/atac-seq-pipeline-${version} /workdir/$USER

 

2. Prepare reference genome database

1) If you work with human and mouse data, the Encode project provides pre-prepared genome database. Go to this page , under the section for "Reference genome", you will find URL of the reference genome database. For example, "https://storage.googleapis.com/encode-pipeline-genome-data/genome_tsv/v3/hg38.tsv". You will need this URL later.

2) If you work with other species, follow the instructions below to prepare the reference genome database.

  • Edit the script "/workdir/$USER/atac-seq-pipeline-1.10.0/build_genome_data_mod.sh",  setting the values for GENOME, GENOME DEST_DIR REF_FA MITO_CHR_NAME, the rest of the parameters are optional.  Make sure that the chromsome names match between the genome files. For plant genome, you might want to create a new genome fasta file with mitochondria and chloroplast merged, and call it a mitochandria genome. 
  • Run these commands to create genome database. After it is done, you should see a directory /workdir/$USER/$DEST_DIR with a .tsv file inside. You will need this .tsv file later. 
cd /workdir/$USER/atac-seq-pipeline-1.10.0
cp /PATH/TO/your.genome.fa.gz ./
singularity exec atac-seq-pipeline.sif ./build_genome_data.sh

 

3. Put your atac-seq data files ( *.fastq.gz) into the directory /workdir/$USER/atac-seq-pipeline-1.10.0

 

4. Prepare a .json text file to specify all input files, and keep it in /workdir/$USER/atac-seq-pipeline-1.10.0.

  • You can modify from this example file (for local files, replace URL with file or directory name). Detailed documentation of the json file can be found in this page (The section under "Input JSON file")
     

5. Set number of cpu per task.
Optionally, you can modify the file /workdir/$USER/atac-seq-pipeline-1.10.0/atac.wdl, and change the number of cpu per task (under "group: resource_parameter"). In most cases, there is no need to change (for example, default setting for aligner bowtie2 is 6 cores per job which is good). However, you might want to restrict number of simultaneously jobs when running the caper command (e.g. limit to up to 4 jobs, --max-concurrent-tasks 4), otherwise, all available cores on the server will be used.

 

6. run pipeline

caper run atac.wdl -i my.json --singularity atac-seq-pipeline.sif

 

7. summarize results

cd atac
ls -l
cd xxxxxxx         #replace xxxxxxx with the run directory you get from ls -l
croo metadata.json
qc2tsv qc/qc.json > qc.tsv

The results should be in the directories "peak" "qc" and "signal" , report files croo*, and QC table qc.tsv 

 

Run pipeline with example data files

Here is how to run test data set provided by the pipeline developer (run software in "screen" persistent session)

export PYTHONPATH=/programs/caper/lib/python3.6/site-packages:/programs/caper/lib64/python3.6/site-packages
export PATH=/programs/caper/bin:$PATH
export version=1.10.0

mkdir /workdir/$USER
cd /workdir/$USER/
cp -r /programs/atac-seq-pipeline-${version} /workdir/$USER

#download test data set
wget https://storage.googleapis.com/encode-pipeline-test-samples/encode-atac-seq-pipeline/ENCSR356KRQ_subsampled_caper.json

#process test dataset
export ATACROOT=/workdir/$USER/atac-seq-pipeline-${version}

caper run $ATACROOT/atac.wdl -i ENCSR356KRQ_subsampled_caper.json --singularity $ATACROOT/atac-seq-pipeline.sif

If no --output directory specified, the output directory is atac. The result files under atac, in execution directory, the files are documented in https://encode-dcc.github.io/wdl-pipelines/output_atac.html.

# After the work is finished, organize output results with croo

export PYTHONPATH=/programs/caper/lib/python3.6/site-packages:/programs/caper/lib64/python3.6/site-packages
export PATH=/programs/caper/bin:$PATH
cd atac
ls -l 
cd xxxxxxx  #replace xxxxxxx with the run directory you get from "ls -l"
croo metadata.json
qc2tsv qc/qc.json  > qc.tsv

The results should be in the directories "peak" "qc" and "signal" , report files croo*, and QC table qc.tsv 

 


Notify me if this software is upgraded or changed [You need to be logged in to use this feature]

 

Website credentials: login  Web Accessibility Help