How to branch the Annotation package repository

Bioconductor Maintainer

May 09, 2025

Branching Annotation

The document describes how to branch the annotation repository for a new release. This can be done the day of the release or several days after.

Before branching, Check and make sure there is enough room on master for the annotations. If there isn’t enough room follow the guidelines for moving old Bioconductor releases into S3 As of release 3.21 Annotations are around 89G. See also inst/script/tempS3 for commands to migrate packages to S3. See also playbook for moving release to OSN: https://github.com/bioconductor/archive-sync. The live backup for legacy releases is OSN but we continue to backup both as we switch to S3 if OSN goes down.

These instructions use BioC 3.22 as the new devel version and nebbiolo2 as the devel master builder.

As of 3.14, the linux builders have moved to DFCI in Boston. Personal accounts and the jump account should be set up for access.

Set up nebbiolo2

Create new annotation repo on devel master builder (nebbiolo2) 

ssh to nebbiolo2 as biocpush user
cd ~/PACKAGES/3.22/data
mkdir annotation

rsync annotations from current release

In this case the current release will be on nebbiolo1

Notes

  • Run with --dry-run first
  • Note the trailing slash / and dot . 
# From nebbiolo1 to nebbiolo2 (logged into nebbiolo1):
cd ~/PACKAGES/3.21/data/annotation
rsync --dry-run -ave ssh . biocpush@nebbiolo2:PACKAGES/3.22/data/annotation/

Live Run if all looks correct

rsync -ave ssh . biocpush@nebbiolo2:PACKAGES/3.22/data/annotation/

There may be symlinks in the windows and macosx folders. These will only be present when the new version of BioC uses a new version of R which happens every other version of BioC. If they are present, they will look something like this:

~/PACKAGES/3.7/data/annotation/bin/windows/contrib/3.5 -> 3.4
~/PACKAGES/3.7/data/annotation/bin/macosx/contrib/3.5 -> 3.4

Remove the symlinks and rename the folder with content to the version of R being used by devel e.g.,

rm ~/PACKAGES/3.7/data/annotation/bin/windows/contrib/3.5 -> 3.4
mv 3.4 3.5

The PACKAGES and PACKAGES.gz files are empty. They are needed for install.packages() even though we don’t have binaries for annotations.

A rerun of the rsync command should only show changes in the files we just edited:

rsync --dry-run -ave ssh biocpush@nebbiolo1:PACKAGES/3.21/data/annotation/ .

Set up <master.bioconductor.org>

Log on master as webadmin

Navigate to the new devel repo

ssh ubuntu@master.biocondcutor.org
sudo su - webadmin
cd /extra/www/bioc/packages/3.22/data/annotation

Products are sent to master from both nebbiolo2 and staging.bioconductor.org. Because of this, the rsync from nebbiolo2 to master is not from the top level; we want to leave the products deposited by staging.bioconductor.org untouched.

There will likely be a simlink at

/extra/www/bioc/packages/3.22/data/annotation -> ../../3.21/data/annotation

pointing to the current release annotation repo. In recent years, the symlinks have appeared underneath an already created annotation repostiory. The point is to start with a non-linked empty annotation folder. Remove annotation and recreate:

rm annotation

Create a new annotation folder 

mkdir annotation

Back to nebbiolo2 to run the propagation scripts

  1. Remove any deprecated packages from the 3.21 annotation repo that should not be included in the devel 3.22 repo.

  2. Update and run cronjob by hand (should be commented out).

Symlink must be gone on master before running this!!

Run the crontab entry by hand. Monitor ~/cron.log/3.22/propagate-data-annotation-* log file.

  1. Test BiocManager::install() to see if it finds the new repo. Try installing a package that should be found and try installing a removed package that should fail.

Update bioconductor.org/config.yaml to build landing pages for 3.22

Uncomment ‘data/annotation’:

devel_repos:
- "bioc"
- "data/experiment"
- "data/annotation"