Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve test-release CI job runtime #954

Merged
merged 1 commit into from
Jan 2, 2025

Conversation

danielocfb
Copy link
Collaborator

As we add more jobs and have existing ones perform more work, CI slows down. One of the main offenders is the 'test-release' job, which is found to have a both cold start runtime but also doesn't play particularly nice with our caching infrastructure. This has potential trickle down effects on other jobs, which may only get scheduled on runners once others have finished (because there exist overall limits on the number of runners in use), potentially delaying overall CI time doubly.
The slowness of this job seems to come down to two things: 1) our release builds use LTO etc., making linking a very lengthy
process
2) it turns out that release builds are not incremental by default [0],
causing a lot of recompilation to happen, despite our efforts of
caching intermediate artifacts

To fix these issues, we disable the link time optimization step in CI and enable incremental compilation. Doing so reduces runtime from >=3min down to ~1min.

[0] https://doc.rust-lang.org/cargo/reference/profiles.html#release

As we add more jobs and have existing ones perform more work, CI slows
down. One of the main offenders is the 'test-release' job, which is
found to have a both cold start runtime but also doesn't play
particularly nice with our caching infrastructure. This has potential
trickle down effects on other jobs, which may only get scheduled on
runners once others have finished (because there exist overall limits on
the number of runners in use), potentially delaying overall CI time
doubly.
The slowness of this job seems to come down to two things:
1) our release builds use LTO etc., making linking a very lengthy
   process
2) it turns out that release builds are not incremental by default [0],
   causing a lot of recompilation to happen, despite our efforts of
   caching intermediate artifacts

To fix these issues, we disable the link time optimization step in CI
and enable incremental compilation. Doing so reduces runtime from >=3min
down to ~1min.

[0] https://doc.rust-lang.org/cargo/reference/profiles.html#release

Signed-off-by: Daniel Müller <[email protected]>
Copy link

codecov bot commented Jan 2, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 94.47%. Comparing base (99fc30a) to head (856651c).
Report is 1 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #954   +/-   ##
=======================================
  Coverage   94.47%   94.47%           
=======================================
  Files          57       57           
  Lines       10650    10650           
=======================================
  Hits        10062    10062           
  Misses        588      588           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@danielocfb danielocfb merged commit c4e3f41 into libbpf:main Jan 2, 2025
41 checks passed
@danielocfb danielocfb deleted the topic/test-release-ci branch January 2, 2025 20:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants