-
Notifications
You must be signed in to change notification settings - Fork 27
Making New Releases
When a new LArSoft version is released, it is customary to create a new release of the dune software stack and install these on cvmfs. This consists of 1) updating dunecore's dependence on LArSoft 2) making new tags of the software stack repositories 3) building the new release on the Jenkins 4) distributing on cmvfs.
Tip on testing the build of the DUNE software stack on Jenkins with a pull request: Add a comment in the PR conversation
trigger build with DUNE/[repo]#PRNUMBER
and you can do multiple PRs with that i.e.
trigger build with DUNE/dunesw#73 DUNE/dunereco#58 (numbers are made up)
One should have a local release with all packages in the dune software suite. This can be done by typing mrb g dune_suite in the srcs directory. Before starting with the releases, it is best practice to update each repository from its remote git repository. Commit any of your own local changes in the repositories and merge into your local develop branch if needed, then use the following command to git pull from each (starting from within your srcs area):
for i in */; do echo $i; cd $i; git pull; cd ../; done
You'll also want to update the master branch, as this is updated during the release flow. If others have made new releases, you'll need this history in your local repos.
for i in */; do echo $i; cd $i; git fetch origin master:master; cd ../; done
In your local LArSoft directory, remove your local products directory and make a new local products directory for the latest version using mrb newDev -p -v <version> -q <quals>. I recommend just using e26:prof for the quals.
Then, update the larsoft version in dunecore/ups/product_deps.
Additionally, some other dependencies might need to be updated. duneutil/scripts/list_dependencies.py can be used (after sourcing local products) to list the dependencies and their current versions. Look through this list and find the latest/newly required versions of each dependency (using ups list -aK+ <product>) and update if needed.
Run a local test a build, and commit the changes before moving on:
for i in */; do echo $i; cd $i; git commit -a -m 'Update dependencies.'; cd ../; done
Now that the larsoft version has been updated and major patches/pull requests/commits for this release have been built locally, it is time to push all these changes to the remote repository before testing the rest of the builds using the CI test framework. Go ahead and push these changes (from within the srcs dir)
for i in */; do echo $i; cd $i; git push; cd ../; done
A crucial part of making releases is checking that the CI tests succeed. The DUNE CI tests can be found here.
To start a CI test, run the following setup commands
source /cvmfs/fermilab.opensciencegrid.org/products/common/etc/setups.sh
setup lar_ci
export CI_CERT=/tmp/ci_cert.pem; kx509 -o ${CI_CERT}
Then run this command to trigger the tests
for i in e26 c14; do for j in prof debug; do trigger --cert ${CI_CERT} --build-delay 0 --version develop --force-platform slf7 --quals "${i}:${j}" --workflow duneswstandalone_wf --jobname dune_ci; done; done
Before beginning to make new tags, it's important to check that the software builds correctly and the CI tests succeed. If the build fails, click on the red 'X' under the "Build" header, then click on the log. Check the issue and contact any relevant developers.
The important CI test to check is the e26:prof version. If all e26:prof tests pass, no investigation is needed. Some can issue a warning, however, if the CPU or memory usage is even slightly outside the stated limits. If they're close enough, there's no real issue. If a suspiciously low CPU usage is shown, this is evidence of a failure.
Before building on Jenkins, a tag for each repository is needed, followed updating the versions/dependencies within the suite. The first can be started using the following command from within srcs:
for i in */; do echo $i; cd $i; git flow release start <version>; cd ../; done
Note that the version should be of the form: vXX_YY_ZZdDD where XX_YY_ZZ corresponds to the version of LArSoft depended on within dunecore, and DD denotes the suite release for that version (starting at 00).
If you get this error Fatal: Not a gitflow-enabled repo yet. Please run 'git flow init' first. Do this:
for i in */; do echo $i; cd $i; git flow init -d; cd ../; done
This only needs to be done once.
A script is available in duneutil that can be used to update the version numbers defined within CMake and ups/product_deps files. This is invoked with the following command:
./duneutil/scripts/fix_deps_cmake.py <version> where <version> matches above.
At this point, that's all the changes that are needed. It is a good idea to build locally again before finishing the tagging. Clear the MRB_BUILDDIR, remove the local products directory and get a new one, (these two operations can be done with the command mrb zd) then build again and watch out for any issues.
If the local test build succeeds, it's time to finish the tags. Use the following command from within srcs to commit the version number updates:
for i in */; do echo $i; cd $i; git add -u; git commit -m "Updating version to <version>"; cd ../; done
Then you can finish the release tags similarly:
for i in */; do echo $i; cd $i; git flow release finish -m "<version>"; cd ../; done
Note that vi windows will pop up containing automatically generated merge messages. Just :q past these.
Then, you must push the tag, master, and develop branches. Use the following command:
for i in */; do echo $i; cd $i; git push; git push origin master; git push origin <version>; cd ../; done
Now, it's time to build on Jenkins. First, you must build the new release of the suite and then distribute that on cvmfs.
Log on to Jenkins and select dunesw-release-build and, within that, "Build with Parameters". Enter in the version and start the build. This should take about 20 minutes.
When the suite build is finished, fetch it on a gpvm using this script (copy contents into your a script in your home area):
source /path/to/duneutil/scripts/copyFromJenkins -m AlmaLinux-9.4 -q e26 -q c14 dunesw-release-build
(this needs a lightly modified copyFromJenkins script to understand ALMA9.3)
This will fetch 48 tars (8 for each of 12 repositories) and 12 text files containing manifests. Log back on to [email protected] and unpack all of these similarly to above:
for i in /path/to/tarballs/*bz2; do tar -xf $i; done
Once all are unpacked, it is time to sync with cmvfs. From [email protected], you must log on to another machine as follows:
ssh -l cvmfsdune oasiscfs.fnal.gov
As of June 25, 2024, the node oasiscfs05.fnal.gov is running under Alma Linux 9.
Use this handy script to sync all of the directories as such:
./sync-suite.sh
The new versions of the suite will soon be available on cvmfs (around half an hour should be good enough).
Also upload the tarfiles and manifests to the SciSoft web server. In the directory containing the 48 tars and nothing else:
copyToSciSoft *.bz2
and
copyToSciSoft *.txt
If you have other things in that directory, use more specific wildcards, such as those containing the version tag.