Who is this for?

This workshop is for a beginner. You don’t need any prior experience with Docker or container technology. In this course participants are expected to launch Docker images on their local machines, mount volumes, and set ports as needed to use RStudio.

Since it’s a beginners course, please note that we will be starting at a basic level. The next section of the course, the advanced section will give you deeper insight into how container technology can be used on the cloud with some demonstrations.

Why Containers and Why Docker

A Container is a completely isolated environment. They can have their own processes or services or network interfaces, volume mounts, just like a virtual machine. They differ from VM’s only in a single aspect, they “share” the OS kernel. Containers are running instances of images.

Docker is simply the most popular container technology being used today. An Image is a static (fixed) template and container is a running version of the image. An image can be used to create a container.

Main reason behind using containers: As a user, you want to delegate problems of software installation to the developer (in this case - Bioconductor). Bioconductor gives you these static docker images which solve this issue of software installation on your infrastructure.

The total disk space used by all of the running containers on disk is some combination of each container’s size and the virtual size values. If multiple containers started from the same exact image, the total size on disk for these containers would be SUM (size of containers) plus one container’s (virtual size). Be careful and monitor the size of your containers as it may grow very quickly. Read more about Container size on disk.

Docker warmup

Get a list of docker images

docker images

Let’s pull (get) an image specifically bioconductor/release_base2:latest (/:). This will take a minute or two.

docker pull bioconductor/release_base2:latest

Run the image to start a container,

docker run -p 8787:8787 -e PASSWORD=bioc bioconductor/release_base2:latest

where, -p if the port, and -e is the environment variable

Open a new terminal and you can check to see your container is running,

docker ps

Kill the container

docker kill <container_id>

Remove the image (you don’t want to do this one right now)

docker rmi bioconductor/release_base2:latest

Interactively running docker images ,

## For bash
docker run -it bioconductor/release_base2 bash

## For R directly
docker run -it bioconductor/release_base2 R

Interactively run Containers,

## Get the container ID
docker ps

You can do this only if the container with that ID is running currently,

docker exec -it <container ID>

Bioconductor on Docker

In this section we will discuss what flavors are available to users, and how to build on these images.

Bioconductor provides Docker images in a few ways:

docker pull bioconductor/bioconductor_full:devel

docker pull bioconductor/bioconductor_full:RELEASE_3_9

docker pull bioconductor/bioconductor_full:RELEASE_3_8

Flavors of Bioconductor

The flavors you “need” for your work should be based on the packages you want for your analysis. There are a list of packages for each flavor you should evaluate before using the container.

For most beginners a good place to start is the bioconductor/release_core2 docker container which includes all the “core” bioconductor packages.

Mount volumes to your Docker container

It is useful sometimes to mount a volume either with data or a list of installed packages.

You can bind as many volumes as needed with the -v/--volume command. The format is given as

-v /host/path:/container/path

If you wanted to share data with your docker container, run the container with the command,

docker run \
    -v /data:/home/rstudio/data
    -e PASSWORD=bioc
    -p 8787:8787
    bioconductor/release_base2

To share libraries with your Docker container, bioconductor provides a way so that the R running within your docker image is able to see the packages in your container,

-v /shared-Bio-3.9:/usr/local/lib/R/host-site-library

This location /usr/local/lib/R/host-site-library has special meaning, and the configuration within the bioconductor container allows the user to share built packages. The path on the docker image that should be mapped to a local R library directory is /usr/local/lib/R/host-site-library.

1. Create a directory, on your host machine, (as given as whichever path suits you the best)

    mkdir /shared-BioC-3.9

2. Run the image by sharing that directory, for the R package libraries and the data.

    docker run \
        -v /shared-BioC-3.9:/usr/local/lib/R/host-site-library \
        -v /data:/home/rstudio/data \
        -it bioconductor/release_base2 \
        R

   Within R,

    BiocManager::install('BiocGenerics')

All the packages in that directory will be directly available to use instantly the next time R is started from the container. Another big advantage is, it prevents your docker image from growing in size due to package installation and data. This allows the users to distribute smaller docker images containing only what is needed.

Conclusion of Beginners section

Re cap of items you have learnt in this section,

1. How to start Bioconductor based docker images.

2. Where these images are available.

3. What flavors of Bioconductor images are present.

4. How to bind/mount volumes on these Bioconductor docker images.

These skills get you started on using Bioconductor on containers.

Who is this section for?

The advanced section is for anyone who has some familiarity with how Docker works, and wants to run things on the “Cloud”. This will be a demonstration section, since we are not providing “credits” to use a certain cloud (Google or AWS).

This section aims to give some direction to users who are looking to use Bioconductor on the cloud. We also discuss some future developments, which look promising.

Containers on the cloud

Running your R session on the cloud with RStudio makes you ‘machine independent’. It also allows you to access data hosted on the cloud cheaper and faster for example 1000 Genomes data which is available on the google cloud: gs://genomics-public-data/1000-genomes.

One of the biggest advantages to cloud machines is the on-demand scalability to run programs.

In this section, there will be a demonstration showing the usability with the Google cloud Infrastructure, but take note that everything you can do on the Google Cloud you can do with AWS.

Mount volumes on the cloud (Optional)

There are many publicly available genomics data sets hosted by cloud services, Google cloud genomics datasets are located on the bucket, gs://genomics-public-data.

You can mount storage buckets on your VM to your container using GCSFUSE. For this you need to choose an image with access to the google cloud and gcsfuse. To install gcsfuse, and

Create a directory, where you want the google bucket

mkdir /path/to/mount

Mount a bucket on your google cloud

gcsfuse example-bucket /path/to/mount

Start working with the mounted bucket

ls /path/to/mount

Localize select packages

For faster installation of packages from Bioconductor on cloud based instances, as provide as a demonstration ONLY Google storage buckets with all the software packages which are pre-compiled (1600+).

Bioconductor provides the entire pre-compiled binary libraries for the following versions of R and Bioconductor. These are publicly accessible buckets.

NOTE: Requester Pays (aka: You) for the access to these buckets. So by accessing the data on these packages to be downloaded on to your google cloud instance your billing project gets charged.

To copy packages over to the instance, using the command line:

## Get the name of your billing project
gcloud config list project

## Copy packages over to your local directory
gsutil \
    -u <name-of-billing-project> \
    -m \
    cp -r gs://bioconductor-full-devel/<package_name> <local_directory>

We also provide a very helpful package (which is part of another project) called AnVIL. With the help of the AnVIL::install() function, users are able to localize packages from buckets and install them on containers at high speed without having to wait for compilation.

## Launch a bioconductor_full:RELEASE_3_9 image
gcloud compute instances create-with-container bioc-release \
        --container-image bioconductor/bioconductor_full:RELEASE_3_9 \
        --container-restart-policy never \
        --container-env PASSWORD=bioc \
        --tags http-server

## Login in to your Rstudio session
## username: Rstudio, password:bioc

## Install the AnVIL package
BiocManager::install('Bioconductor/AnVIL_rapiclient')
BiocManager::install('Bioconductor/AnVIL')

## Use AnVIL install to install packages on your bioconductor-full image
AnVIL::install('GenomicRanges')

HPC-Singularity containers

Singularity is a container engine just like Docker. Read more about on the Singularity webpage.

Advantages of using Singularity are:

• Singularity is compatible on HPC infrastructures and is easily installable.

• It has a BSD license (open source!!)

• Works very well with Docker images which are already available.

Singularity comes as a module which can be installed on a cluster facility. Once the module is available, it is easy for the user to run software in a containerized fashion, without depending on the system admin for new software installations.

module load singularity

Bioconductor provides Singularity containers on the Singularity Hub. There are three containers provided right now which represent versions of Bioconductor,

1. devel (master)

   ## Pull image singularity pull –name bioc-devel.simg shub://Bioconductor/bioconductor_full

   ## Use image’s R singularity shell bioc-devel.simg /usr/local/bin/R

2. RELEASE_3_9

   singularity pull –name bioc-release-3-9.simg shub://Bioconductor/bioconductor_full:release_3_9

3. RELEASE_3_8

   singularity pull –name bioc-release-3-8.simg shub://Bioconductor/bioconductor_full:release_3_8

You can set an alias to your Singularity image on your .bashrc file as,

alias singulaR="singularity shell \
  -B /mnt/lustre/users/nturaga/my-lib:/usr/local/lib/R/host-site-library \
  /mnt/lustre/users/nturaga/bioc-devel.simg \
  /usr/local/bin/R"

where,

-B is how you mount a volume on the singularity image <local path>:<container path>. This command shows my R library being mounted from my head node on the singularity container.

/mnt/lustre/users/nturaga/bioc-devel.simg refers to the location of my singularity image.

/usr/local/bin/R is the location of R on the container.

From here on out, the user can build packages or use packages on their local cluster using,

singulaR CMD build <package name>

or run scripts using

singulaR -f <my_R_script.R>

Launch and run parallel jobs

Use BiocParallel to launch and run jobs on Singularity containers. The following example has been run on SLURM (This example has been provided thanks to Martin Morgan, Levi Waldron and Ludwig Geistlinger).

To use the containerized singularity image in conjunction with BiocParallel, a recent R(3.6.0) installation on the cluster is needed that, as a minimum requirement, has also the packages BiocParallel and batchtools installed (minimum requirement would be similar to the image biocparallel-example)

NOTE: Submission of jobs directly from within the singularity container on the head node is not possible. The container doesn’t recognize the cluster environment and its specific configuration. It might also cause issues to mount volumes.

For a SLURM cluster, we need to first define a template file for the jobs. slurm-singularity.tmpl is given below as an example. Loading the module ‘singularity’ and using the full path to the singularity image <image.simg> is important.

#!/bin/bash

<%
# relative paths are not handled well by Slurm
log.file = fs::path_expand(log.file)
-%>

#SBATCH -p <queue>
#SBATCH --job-name=<%= job.name %>
#SBATCH --output=<%= log.file %>
#SBATCH --error=<%= log.file %>

module load singularity

singularity shell /mnt/fs/nturaga/bioconductor-image-w-BiocParallel-latest.simg R -e 'batchtools::doJobCollection("<%= uri %>")'

Using BiocParallel and batchtools, it is then possible to run the following code on your head node in R using BatchtoolsParam. This batch tools functionality was introduced last year as an improvement on previously available batch processing methods in BiocParallel.

## Load the library
library(BiocParallel)
library(batchtools)

## define your BPPARAM,
param <- BatchtoolsParam(
    ## define number of workers
    workers = 5,
    ## define the cluster you are using, in this case SLURM
    cluster = "slurm",
    ## point to the custome template
    template="slurm-singularity.tmpl"
)

## Supply the param to bplapply for parallel execution
## of a simple function that returns the hostname of the
## compute nodes for testing purpose. Run bplapply.
test_cluster <- function(n) system2('hostname')

bplapply(1:5, test_cluster, BPPARAM = param)

Future work

There is quite a bit of work in progress to develop solutions for Bioconductor to meet scalable data analysis requirements. A few of them are highlighted below,

1. BiocParallel::RedisParam

   It is based on the redis data-structure (Redis).

   Link: RedisParam

   enabling an alternative, modern mechanism for distributed computation.

   This is yet to be integrated into BiocParallel. The RedisParam back end allows computation to be distributed across instances or containers running on the cloud. To explain the way the RedisParam functions, think of 5 worker instances waiting to to hear from a single manager instance which sends a job or multiple jobs.

2. Kubernetes

   As the next iteration of our efforts on the cloud, we needed RedisParam to be working on a container orchestration technology like Kubernetes.

   We use Kubernetes to deploy the “cluster” on the cloud, scalability and to automate management of the deployment to free the user from mundane tasks such as worrying if a worker node has failed.

   Link: Bioconductor-k8sredis-k8s

3. Helm charts

   helm.sh is a package manager for Kubernetes. Once we created the Kubernetes deployment for RedisParam we wanted to make sure it’s easily distributed and used in the community.

   Helm basically works on templating the K8s deployment and making a easy to distribute tarball (or package).

   Link: Bioconductor-k8sredis-helm-chart

sessionInfo

sessionInfo()

## R version 3.6.0 (2019-04-26)
## Platform: x86_64-pc-linux-gnu (64-bit)
## Running under: Debian GNU/Linux 9 (stretch)
## 
## Matrix products: default
## BLAS/LAPACK: /usr/lib/libopenblasp-r0.2.19.so
## 
## locale:
##  [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
##  [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
##  [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=C             
##  [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
##  [9] LC_ADDRESS=C               LC_TELEPHONE=C            
## [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       
## 
## attached base packages:
## [1] stats     graphics  grDevices utils     datasets  methods   base     
## 
## other attached packages:
## [1] tibble_2.1.3       dplyr_0.8.1        BiocManager_1.30.4
## 
## loaded via a namespace (and not attached):
##  [1] Rcpp_1.0.1       knitr_1.23       magrittr_1.5     tidyselect_0.2.5
##  [5] R6_2.4.0         rlang_0.3.4      stringr_1.4.0    tools_3.6.0     
##  [9] xfun_0.7         png_0.1-7        htmltools_0.3.6  yaml_2.2.0      
## [13] digest_0.6.19    assertthat_0.2.1 crayon_1.3.4     bookdown_0.11.1 
## [17] purrr_0.3.2      glue_1.3.1       evaluate_0.14    rmarkdown_1.13  
## [21] blogdown_0.13.1  stringi_1.4.3    compiler_3.6.0   pillar_1.4.1    
## [25] pkgconfig_2.0.2