Skip to content

Commit

Permalink
Fail codecov CI check based on patch coverage goal (pulumi#17998)
Browse files Browse the repository at this point in the history
This PR sets the target coverage for net-new code in PR to 80%. The
check will fail if patch coverage is below that. I've set threshold to
5% to allow some minimal drop in coverage if the overall goal is still
achieved.

We'll have to learn what the exact numbers should be over time, but this
is a start.

When the check fails, the GH experience will look like this:

<img width="1058" alt="image"
src="https://github.com/user-attachments/assets/c3a8acfa-f663-4976-9985-b883c251b2da">

Note that:

1. "Merge when ready" button is not green but it's also not disabled, so
we can merge without the check succeeding. We shouldn't normally do
this, but we can for situations when we explicitly deem this check
unhelpful.
2. Cove coverage checks will only appear if all tests succeed and
coverage data is uploaded to codecov.
3. There is a small delay between ci-ok check succeeding and code
coverage appearing.
4. If a PR is approved before the checks complete and we click the Merge
button, we can miss the failing coverage gate.

I think it's a reasonable step forward that we can learn from.

Resolves pulumi#17996
  • Loading branch information
mikhailshilkov authored Dec 11, 2024
1 parent fd2b278 commit ca7d6a3
Showing 1 changed file with 6 additions and 1 deletion.
7 changes: 6 additions & 1 deletion codecov.yml
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,12 @@ coverage:
# Patch tracks the coverage of the changes in a single patch.
patch:
default:
informational: true
# coverage for new/modified code
target: 80%
# only allow coverage to drop by 5% in new/modified code
threshold: 5%
# make codecov block PR if coverage goals aren't met
informational: false

ignore:
# Integration tests, lifecycle tests and codegen test data
Expand Down

0 comments on commit ca7d6a3

Please sign in to comment.