Improve test-release CI job runtime #954
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
As we add more jobs and have existing ones perform more work, CI slows down. One of the main offenders is the 'test-release' job, which is found to have a both cold start runtime but also doesn't play particularly nice with our caching infrastructure. This has potential trickle down effects on other jobs, which may only get scheduled on runners once others have finished (because there exist overall limits on the number of runners in use), potentially delaying overall CI time doubly.
The slowness of this job seems to come down to two things: 1) our release builds use LTO etc., making linking a very lengthy
process
2) it turns out that release builds are not incremental by default [0],
causing a lot of recompilation to happen, despite our efforts of
caching intermediate artifacts
To fix these issues, we disable the link time optimization step in CI and enable incremental compilation. Doing so reduces runtime from >=3min down to ~1min.
[0] https://doc.rust-lang.org/cargo/reference/profiles.html#release