Investigate workflow job failing on main: e2ePerformanceTests / Find the baseline and delta refs, and check for an existing build artifact for that commit

[github-actions]

🚨 Failure Summary 🚨:

• 📋 Job Name: e2ePerformanceTests / Find the baseline and delta refs, and check for an existing build artifact for that commit
• 🔧 Failure in Workflow: Process new code merged to main
• 🔗 Triggered by PR: PR Link
• 👤 PR Author: @hannojg
• 🤝 Merged by: @marcaaron
• 🐛 Error Message:
  warning: ubuntu-latest pipelines will use ubuntu-24.04 soon. For more details, see Ubuntu-latest workflows will use Ubuntu-24.04 image actions/runner-images#10636
  failure: HttpError: Not Found

⚠️ Action Required ⚠️:

🛠️ A recent merge appears to have caused a failure in the job named e2ePerformanceTests / Find the baseline and delta refs, and check for an existing build artifact for that commit.
This issue has been automatically created and labeled with Workflow Failure for investigation.

👀 Please look into the following:

1. Why the PR caused the job to fail?
2. Address any underlying issues.

🐛 We appreciate your help in squashing this bug!

[marcaaron]

Looks like some kind of Http error on GitHubs end possibly

[hannojg]

hm, for some reason when that PR got merged the job tried to look for a PR with your user instead of using the PR number. Maybe there is something broken about that logic (I think the PR number should've been used)

[hannojg]

While the above error might have been a GitHub hiccup, it seems that the e2e tests are still broken though.
All runs are currently failing after ~ 1h45m on AWS.
Checking whats wrong with that …

[hannojg]

@perunt it seems that the tests are failing due to this error in the linking test:

@kirillzyusko I know that we have a mechanism for marking a test as failed in a way that it won't fail the whole pipeline, but just the faulty test. Can you please let us know what the API for that is? Then @perunt can open a PR to fix that not the whole pipeline fails.
After that we should probably investigate why that error is happening though.

[hannojg]

Issue is being investigated over here:

• https://margelo.slack.com/archives/C06JLFDLLUX/p1735216017374219

[kirillzyusko]

Yes, we have a fault tolerance mechanism, but it's used to safely handle flaky tests I believe.

But this test explicitly fails with error:

E2EClient.submitTestResults({
    branch: Config.E2E_BRANCH,
    error: 'Linked message not found',
    name: `${name} test can't find linked message`,
}).then(() => E2EClient.submitTestDone());

And for such cases we don't have safe failure handling, because they are clearly indicate that it's something wrong with the test itself (i. e. it's not flakiness) 🤷‍♂️

And it forces us to investigate a reason instead of silently ignoring them 👀

[melvin-bot]

This issue has not been updated in over 15 days. @hannojg eroding to Monthly issue.

P.S. Is everyone reading this sure this is really a near-term priority? Be brave: if you disagree, go ahead and close it out. If someone disagrees, they'll reopen it, and if they don't: one less thing to do!

[hannojg]

This is fixed by now, can we please close this @mountiny (I don't have the permission for some reason)