Skip to content

Generate baseline hashes during tests to reduce flakiness #234

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

ConorMacBride
Copy link
Member

@ConorMacBride ConorMacBride commented Nov 6, 2024

  • The pytest-mpl test suite relies on lots of hardcoded hashes remaining the same, which leads to lots of test failures on different platforms.
  • Instead of hardcoding the hashes we can quickly generate them during each test run and then modify specific hashes to simulate an expected test failure.

Closes #232
Closes #225
Closes #179

@ConorMacBride ConorMacBride added the infrastructure PRs related to the package and tests etc label Nov 6, 2024
@ConorMacBride ConorMacBride force-pushed the remove-hashes branch 3 times, most recently from 668b09c to ce61e89 Compare November 6, 2024 21:34
@ConorMacBride ConorMacBride marked this pull request as ready for review November 6, 2024 22:18
@ConorMacBride
Copy link
Member Author

Hi @astrofrog @Cadair! How would you feel about merging this? It would make the test suite much more stable over time, as we'd be less sensitive to minor visual changes in matplotlib. As I mentioned in #240:

I was thinking earlier that we could remove most of the hardcoded hashes in the tests by generating them during the test run: 1) run code to generate hashes 2) modify/break some hashes 3) test that pytest-mpl passes and fails correctly when rerun. It removes the flakiness, but does mean that we only test that matplotlib can produce identical hashes over repeated runs and not necessarily over different matplotlib or OS versions.

@astrofrog
Copy link
Collaborator

I'd be happy to see this merged!

@Cadair
Copy link
Contributor

Cadair commented Apr 29, 2025

Seems sensible to me. :shipit:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
infrastructure PRs related to the package and tests etc
Projects
None yet
3 participants