Version-bump-and-branch-creation-HOWTO.RmdOn the day prior to the release, we will need to apply and push the following changes to all the Bioconductor packages that go in the new release, in this order:
devel branch
devel branch
For example, for the BioC 3.22 release, we will need to do this for
all the packages listed in the software.txt,
data-annotation.txt, data-experiment.txt,
workflows.txt, and books.txt files of the
RELEASE_3_22 branch of the manifest repo.
However, note that for the packages listed in the
data-annotation.txt, we won’t perform any version bump
i.e. we will only create the new release branch.
Important exceptions:
The BiocVersion package (software package): This package will only need the new branch and a simple y -> y + 1 version bump.
The data-annotation packages: For these packages we will create the new branch but we won’t bump versions.
We’ll use Python script
bump_version_and_create_branch.py to apply and push these
changes. This will need to be done on the day prior to the release, and
BEFORE THE BioC 3.22 BUILDS START FOR SOFTARE, DATA-ANNOTATION,
DATA-EXPERIMENT, WORKFLOW, AND BOOK PACKAGES.
Look for the prerun jobs in the crontab for the
biocbuild user on the central BioC 3.22 builder to get the
times the software and data-experiment builds get kicked off. Make sure
to check the crontab again a couple of days before the release as we
sometimes make small adjustments to the crontabs on the build machines.
Also be sure to translate to your local time if you are not on the East
Coast.
These steps should be performed typically a couple of days before the steps in sections C. and D..
Update this document to reflect the BioC version to be released
i.e. replace all occurrences of 3.22 and
RELEASE_3_22 with appropriate version. This will avoid
potentially disastrous mistakes when copying/pasting/executing commands
from this document.
Choose a Linux machine with enough disk space to clone all the
software, data-experiment, workflow, and book packages (as of October
2021, the total size of all the package clones is about 122G). The
machine needs to have the git client and Python. The
procedure described here doesn’t require sudo privileges.
Make sure to pick up a machine that has fast and reliable internet
access. The Linux build machines are a good choice. If you want to use
one of them, use your personal account or the biocpush
account. Do NOT use the biocbuild account to not interfere
with the builds. Using a Mac server might work but was not
tested.
Make sure to use the -A flag to enable forwarding of
the authentication agent connection when you ssh to the
machine e.g.:
Clone (or update) the BBS git repo:
Create the git.bioconductor.org folder:
Populate git.bioconductor.org with git clones of the
manifest repo and all the package repos (software,
data-annotation, data-experiment, workflows, and books). This takes more
than 3h so is worth doing in advance e.g. a couple of days before the
release. It will save time when performing the steps described in
sections C., D., and
E. below on the day prior to the release.
export BBS_HOME="$HOME/BBS"
# clone `manifest` repo
$BBS_HOME/utils/update_bioc_git_repos.py manifest RELEASE_3_22
# clone software package repos (takes approx. 2.5h)
time $BBS_HOME/utils/update_bioc_git_repos.py software devel RELEASE_3_22
# clone data-annotation package repos (takes a couple of minutes)
time $BBS_HOME/utils/update_bioc_git_repos.py data-annotation devel RELEASE_3_22
# clone data-experiment package repos (takes approx. 2.5h)
time $BBS_HOME/utils/update_bioc_git_repos.py data-experiment devel RELEASE_3_22
# clone workflow package repos (takes approx. 9 min)
time $BBS_HOME/utils/update_bioc_git_repos.py workflows devel RELEASE_3_22
# clone book repos (takes < 1 min)
time $BBS_HOME/utils/update_bioc_git_repos.py books devel RELEASE_3_22Make sure you can push changes to the BioC git server (at git.bioconductor.org):
# check config file
cat ~/.gitconfig
# make any necessary adjustment with
git config --global user.email "you@example.com"
git config --global user.name "Your Name"
# also make sure to set push.default to matching
git config --global push.default matching
# check config file again
cat ~/.gitconfig
# try to push
cd ~/git.bioconductor.org/software/affy
git push # should display 'Everything up-to-date'Clone (or update) the bioc_git_transition git
repo:
Find packages with duplicate commits
Historically the GIT-SVN mirror was the primary source of duplicate commits. Since the transition to git, we should not be seeing many new duplicates. Additionally, we’ve implemented a gitolite hook to prevent any new duplicate commits from getting through. For these reasons, this check for duplicates is becoming obsolete and can probably be removed at the next release in Spring 2019.
# Local copy of bioc_git_transition
export BIOC_GIT_TRANSITION="$HOME/bioc_git_transition"
# software packages
export WORKING_DIR="$HOME/git.bioconductor.org/software"
export MANIFEST_FILE="$HOME/git.bioconductor.org/manifest/software.txt"
cd $WORKING_DIR
pkgs_in_manifest=`grep 'Package: ' $MANIFEST_FILE | sed 's/Package: //g'`
# Check last 30 commits in each package (like the duplicate commits hook
# does):
for pkg in $pkgs_in_manifest; do
echo ""
echo ">>> check $pkg package for duplicate commits"
python3 $BIOC_GIT_TRANSITION/misc/detect_duplicate_commits.py $pkg 30
done > duplicatecommits.out 2>&1Anyone involved in the “bump and branch” procedure should
temporarily be added to the admin group in
gitolite-admin/conf/gitolite.conf. Membership in this group
enables the creation of a new branch and pushing to any branch
regardless of inactivated (commented out) lines in
gitolite-admin/conf/packages.conf.
Perform these steps on the day prior to the release. They must be completed before the software builds get kicked off (see A. Introduction). The full procedure should take about 3 hours. Make sure to reserve enough time.
IMPORTANT: No 3.22 build should start until:
the bump and branch process is complete,
and BBS_BIOC_GIT_BRANCH is switched
from devel to RELEASE_3_22 on the central BioC
3.22 builder (this switch is documented in section F3 below).
Maybe no build is scheduled to start during the next 3 hours, in
which case everything is fine. Otherwise, comment out all the builds
that are at the risk of starting before step F3 below is completed.
Commenting out the prerun and run entries for
each build should be enough.
Announce or ask a core team member to announce on the bioc-devel mailing list that people must stop committing/pushing changes to the BioC git server (git.bioconductor.org) for the next 3 hours.
The RELEASE_3_21 lines in
gitolite-admin/conf/packages.conf were commented out when
the release builds were frozen. At this point, only the
devel lines are still active.
Deactivate all push access by commenting out the devel
lines in gitolite-admin/conf/packages.conf.
NOTE: Do not change the branch from RELEASE_3_21 to RELEASE_3_22, it is not a good solution. Maintainers now will be able to push their own RELEASE_3_22 branch before we are able to create it at release time. This issue reflects the issue https://stat.ethz.ch/pipermail/bioc-devel/2019-May/015048.html.
Using vim, it is possible with a one liner:
:g/RW devel/s/^/#Or Lori created sed commands see
scripts/sedCommandsForVariousTasks.txt
Once packages.conf is updated, push to
gitolite-admin on the git server to make the changes
effective.
Log on to git.bioconductor.org as the git user.
Comment out the hook lines in
packages.conf.
Remove the pre-receive.h00-pre-receive-hook-software
file from the hook/ directory in each package, e.g.,
/home/git/repositories/packages/<PACKAGE>.git/hooks
Or (trying as of release 3.22) try using hooks exclusion file. See scripts/sedCommandsForVariousTasks.txt. This changes is made in the hook_maintainer repo.
Make sure to use the -A flag to enable forwarding of the
authentication agent connection e.g.:
See B. Preliminary steps above for the details.
BBS_HOME, WORKING_DIR and
MANIFEST_FILE environment variables
BBS_HOME:
IMPORTANT NOTE: The settings for WORKING_DIR and
MANIFEST_FILE are specific to the type of packages. The
following settings are for the software packages. Make sure to
adapt for data-experiment packages, workflows, and books.
WORKING_DIR: Point WORKING_DIR to the
folder containing the software packages:
All the remaining steps in section C. must be performed from within this folder:
MANIFEST_FILE: Point MANIFEST_FILE to
the manifest file for software packages. This must be the file from the
RELEASE_3_22 branch of the manifest repo:
bump_version_and_create_branch.py
We’re going to run the script on all the software packages. It will take care of applying and pushing the changes described in A. Introduction.
cd $WORKING_DIR
pkgs_in_manifest=`grep 'Package: ' $MANIFEST_FILE | sed 's/Package: //g'`
$BBS_HOME/utils/bump_version_and_create_branch.py --push RELEASE_3_22 $pkgs_in_manifest >bump_version_and_create_branch.log 2>&1 &Alternatively, the bump_version_and_create_branch.py run
above can be replaced with a 2-pass run:
# First pass (apply all the changes but do NOT push them):
$BBS_HOME/utils/bump_version_and_create_branch.py RELEASE_3_22 $pkgs_in_manifest >bump_version_and_create_branch.log1 2>&1 &
# Second pass (push all the changes):
$BBS_HOME/utils/bump_version_and_create_branch.py --push RELEASE_3_22 $pkgs_in_manifest >bump_version_and_create_branch.log2 2>&1 &The 2-pass run can be useful if one wants to inspect the changes before pushing them.
Notes:
You can follow progress with
tail -f bump_version_and_create_branch.log.
The BiocVersion package will automatically receive special treatment.
In the 2-pass run, the second pass checks the packages and
applies the changes only if needed (i.e. if a package does not already
have the RELEASE_3_22 branch) before pushing the
changes.
If for some reason the
bump_version_and_create_branch.py script stops prematurly,
it can be safely re-run with the same arguments. This is because the
script only does things to packages that need treatment. Packages that
were already successfully treated will no longer be touched.
A typical error is Error: duplicate commits
(happened for affyPLM and Rdisop first time I tested this). Report these
errors to gitolite experts Lori or Jen. Once the problem is
fixed, re-run the script.
Repeat steps C6 to C9 above but for C7 define the environment variables as follows:
Note that for data-annotation packages we create the new branch WITHOUT BUMPING VERSIONS.
Repeat steps C6 to C9 above but for C7 define the environment variables as follows:
export WORKING_DIR="$HOME/git.bioconductor.org/data-annotation"
export MANIFEST_FILE="$HOME/git.bioconductor.org/manifest/data-annotation.txt"Then for C8, make sure to call the
bump_version_and_create_branch.py script with the
--no-bump option:
RELEASE_3_22 branch
This is done by editing the conf/packages.conf file in
the gitolite-admin repo
(git clone git@git.bioconductor.org:gitolite-admin).
If not done already, replace all instances of
RELEASE_3_21 with RELEASE_3_22.
Uncomment all RELEASE_3_22 and devel
lines.
See scripts/sedCommandsForVariousTasks.txt for help on re-enabling access
If using hooks exclusion file for allowing version bump. Restore original hooks file.
Run gitolite setup from /home/git/repositories to
re-enable the hooks.
Test that a non-super user can push access is enabled on the dummy package BiocGenerics_test.
Check:
Announce or ask a core team member to announce on the bioc-devel mailing list that committing/pushing changes to the BioC git server (git.bioconductor.org) can resume.
BBS_BIOC_GIT_BRANCH from devel
to RELEASE_3_22 on central BioC 3.22 builder
DON’T FORGET THIS STEP! Its purpose is to make the BioC 3.22 builds
grab the RELEASE_3_22 branch of all packages instead of
their devel branch.
Log on the central BioC 3.22 builder as biocbuild and
replace
with
in ~/BBS/3.22/config.sh
Also replace
with
in ~/BBS/3.22/config.bat
Then remove the manifest and MEAT0 folders
from ~/bbs-3.22-bioc/,
~/bbs-3.22-data-annotation/,
~/bbs-3.22-data-experiment/,
~/bbs-3.22-workflows/, and ~/bbs-3.22-books/.
They’ll get automatically re-created and re-populated when the builds
start.
Once you’re done, DON’T FORGET TO UNCOMMENT ALL THE BUILDS THAT WERE TEMPORARILY COMMENTED OUT IN THE CRONTAB (see section C1 above).
The Bioconductor organization on GitHub hosts repositories that may
also be packages in Bioconductor. To identify and update these packages
with the latest version bump from a Bioconductor release, use the
ReleaseLaunch R package at https://github.com/Bioconductor/ReleaseLaunch.
First, create a fine-grained Personal Access Token (PAT) under User > Settings > Developer Settings > PATs > Fine-grained tokens.
When generating a new token, select Bioconductor (or
organization) as the resource owner, then ‘All repositories’ under
‘Repository access’. Finally, click on the Generate token
green button. Use gitcreds::gitcreds_set() to store the
fine-grained PAT.
NOTES
If you are renaming branches, select ‘Read and Write’ under ‘Contents’ in the ‘Repository permissions’ section.
Depending on the organization’s settings, R / Bioconductor packages hosted on GitHub may also be updated with a classic PAT.
These commands only work if the user has admin access to the organization’s GitHub account.
To update all packages, run the following command:
add_gh_release_branches(release = "RELEASE_3_22", org = "Bioconductor")Be sure to edit the release argument in the
function.
To update an individual package, run the following function:
add_gh_release_branch(
package_name = "updateObject", release = "RELEASE_3_22", gh_branch = "devel"
)Note that the gh_branch corresponds to the default
branch on GitHub. On some repositories, the gh_branch can
be devel.
Packages will be cloned into the current working directory.