docker_exercises_202210

Docker/Singularity(Apptainer) - Exercises 2

The commands we provided might contain hidden characters. If a copy-pasted command does not run, use this page to clean up the command line: https://biohpc.cornell.edu/clean.aspx

Part 1. Use Conda in Docker

1.1 Two base images can be used for building Conda in Docker: continuumio/miniconda3 and mambaorg/micromamba. Here you will use continuumio/miniconda3.

docker1 run -dit continuumio/miniconda3

1.2 Start a bash shell inside the container.

docker1 ps

export CID=xxxxxxxx        #replace xxxxxxxx with the "CONTAINER ID" 

docker1 exec -it $CID bash

Do you see "(base) " in the bash prompt? That is because in this container, "miniconda3" would automatically start in the "base" environment.

1.3 Install some Conda packages into the container

add "-y" option if you want to skip the confirmation step. But I found it a good idea not to skip it, so that you can check the installation plan.
"-c" options are required , as there are not pre-defined channels in the container. In this case, both "bioconda" and "conda-forge" channels are added because some dependency libraries for samtools-1.16.1 only exist in conda-forge.
Version is not specified for "bwa" here, Conda would get the "compatible" version. But sometimes Conda could install a really old version, because it is deemed by Conda as the only compatible one. To avoid this, it is a good idea to explicitly specify the version.

conda install -c bioconda -c conda-forge samtools=1.16.1

conda install -y -c bioconda bwa

#verify that samtools and bwa are installed properly
samtools --help

#samtools has an error related to the library libncursesw.so.6. 
#someone suggested that updating ncurses would solve the problem  
conda install -c conda-forge ncurses=6.3

samtools --help

1.4 Python3 and pip3 are pre-installed. You can use "pip" to install Python modules in the same Conda environment.

pip install pandas==1.5.1

1.5 Inside the container, the Conda directory is "/opt/conda", not the usual "~/miniconda3"

The directory is defined in the file "~/.conda/environments.txt"

more ~/.conda/environments.txt

ls -l /opt

1.6 After you install all the Conda packages, make sure to exit the container bash shell, and commit the container to a new Docker image.

Also, it is a good idea to save a tar file of the image.

exit

docker1 commit $CID myapp1

docker1 images

#save a tar file of the image
docker1 save -o /workdir/$USER/myapp1.tar biohpc_$USER/myapp1

1.7 Run software in Conda

docker1 run biohpc_$USER/myapp1 samtools --help

Part 2. Convert Docker image to Singularity image, and run software in Singularity container

2.1 Build a Singularity .sif file from the Docker .tar file

cd /workdir/$USER

singularity build myapp1.sif docker-archive://myapp1.tar

#test running samtools
./myapp1.sif samtools --help

2.2 Run software in Singularity

In this exercise, run samtools to convert a sam file into a bam file.

2.2.1 Try the short form of the command

cd /workdir/$USER

cp /shared_data/qisun/test.sam ./


./myapp1.sif samtools view -b -o test.bam test.sam

This short command should work on Singularity version 3.5 installed on BioHPC CentOS 7 servers. But not in the Rocky 9 servers with Apptainer version 1.1.2. This reason is that the new Apptainer stops binding current directory ($PWD) into the container. You will need to use the full command.

2.2.1 Run full command

singularity run -C --bind $PWD:/data --pwd /data myapp1.sif samtools view -b -o test.bam test.sam

ls -l test.bam

#"-C": fully contained;

#"--bind $PWD:/data": $PWD is a Linux variable for current working directory. This option would bind $PWD as "/data" in container;

#"--pwd /data" : set "/data" as current working directory in container;

2.3 Pull a image from Dockerhub and convert it into a Singularity image .sif file

In this exercise, you will pull the latest rocker/r-rer image, and convert it into a singularity .sif file.

The Rocker container automatically starts the "R" command when the container starts. You would see the latest version of R starts.

2.3.1 Run short command

singularity pull r.sif docker://rocker/r-ver

./r.sif

quit()

2.3.1 Run full command

singularity run -C --bind $PWD --pwd $PWD r.sif 

quit()

#--bind: bind the current directory as the same path in the container.

#--pwd: set default working directory

Part 3. Working with Dockerfile

Dockerfile is a script file that can be used to build an image.

2.1 Create a text file named "Dockerfile".

#make a new directory

mkdir /workdir/$USER/myapp2

cd /workdir/$USER/myapp2

#use your favorite text editor to create a text file with the following content, and name the file "Dockerfile".

#Dockerfile should be the only file in the directory, unless you want to copy other files into this image.

FROM continuumio/miniconda3
RUN conda install -y -c bioconda -c conda-forge \
      samtools=1.16.1 ncurses=6.3

RUN conda clean --all --yes

2.2. Use the "docker1 build" command to create a Docker image named "myapp2".

docker1 build -t myapp2 /workdir/$USER/myapp2

2.3. Run software in Docker

docker1 run biohpc_$USER/myapp2 samtools --help