[CI] HealthNodeUpgradeIT testHealthNode {upgradedNodes=1} failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-periodic #5303 / 8.6.2_bwc
• elasticsearch-periodic #5303 / 8.5.3_bwc
• elasticsearch-pull-request #45409 / 8.6.2_bwc
• elasticsearch-pull-request #45409 / 8.5.3_bwc
• elasticsearch-periodic #5297 / 8.5.3_bwc
• elasticsearch-periodic #5297 / 8.6.2_bwc
• elasticsearch-periodic #5291 / 8.6.2_bwc
• elasticsearch-periodic #5291 / 8.5.3_bwc
• elasticsearch-periodic #5285 / 8.6.2_bwc
• elasticsearch-periodic #5285 / 8.5.3_bwc

Reproduction Line:

./gradlew ":qa:rolling-upgrade:v8.6.2#bwcTest" -Dtests.class="org.elasticsearch.upgrades.HealthNodeUpgradeIT" -Dtests.method="testHealthNode {upgradedNodes=1}" -Dtests.seed=C8A826B8E990C4F9 -Dtests.bwc=true -Dtests.locale=hu-Latn-HU -Dtests.timezone=EST -Druntime.java=23

Applicable branches:
8.x

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

org.elasticsearch.client.ResponseException: method [GET], host [http://[::1]:44001], URI [_internal/_health], status line [HTTP/1.1 405 Method Not Allowed]
{"error":"Incorrect HTTP method for uri [_internal/_health] and method [GET], allowed: [POST]","status":405}

Issue Reasons:

• [8.x] 12 consecutive failures in step 8.6.2_bwc
• [8.x] 13 consecutive failures in step 8.5.3_bwc
• [8.x] 25 failures in test testHealthNode {upgradedNodes=1} (2.6% fail rate in 961 executions)
• [8.x] 12 failures in step 8.6.2_bwc (100.0% fail rate in 12 executions)
• [8.x] 13 failures in step 8.5.3_bwc (100.0% fail rate in 13 executions)
• [8.x] 12 failures in pipeline elasticsearch-periodic (100.0% fail rate in 12 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

Pinging @elastic/es-storage-engine (Team:StorageEngine)

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[elasticsearchmachine]

This has been muted on branch 8.x

Mute Reasons:

• [8.x] 13 consecutive failures in step 8.5.3_bwc
• [8.x] 11 consecutive failures in step 8.6.2_bwc
• [8.x] 24 failures in test testHealthNode {upgradedNodes=1} (2.6% fail rate in 934 executions)
• [8.x] 13 failures in step 8.5.3_bwc (100.0% fail rate in 13 executions)
• [8.x] 11 failures in step 8.6.2_bwc (100.0% fail rate in 11 executions)
• [8.x] 12 failures in pipeline elasticsearch-periodic (100.0% fail rate in 12 executions)

Build Scans:

• elasticsearch-periodic #5303 / 8.5.3_bwc
• elasticsearch-pull-request #45409 / 8.6.2_bwc
• elasticsearch-pull-request #45409 / 8.5.3_bwc
• elasticsearch-periodic #5297 / 8.5.3_bwc
• elasticsearch-periodic #5297 / 8.6.2_bwc
• elasticsearch-periodic #5291 / 8.6.2_bwc
• elasticsearch-periodic #5291 / 8.5.3_bwc
• elasticsearch-periodic #5285 / 8.6.2_bwc
• elasticsearch-periodic #5285 / 8.5.3_bwc
• elasticsearch-periodic #5279 / 8.5.3_bwc

[PeteGillinElastic]

This (and #118158) complains that the GET /_internal/_health endpoint doesn't exist when running the BWC tests for 8.5.3 and 8.6.2. This is because we renamed that endpoint in 8.7.0. (It was very briefly at GET /_health but settled on GET /_health_report.)

[PeteGillinElastic]

For the record, this entire test was removed from main, but obviously still runs in these BWC cases.

[PeteGillinElastic]

This looks related to #106933 although that is a different exception. At any rate, my understanding is that no change made to the test code in main will affect the BWC test (which uses the test code from the historic tag).

My feeling is that we can just suppress these tests permanently (or for as long as we run the 8.x BWC tests). The _internal endpoint was never meant to be backwards compatible so this seems fine.

[PeteGillinElastic]

For the record again: These failures are all on 8.x. We don't do the 8.5.3 or 8.6.2 BWC tests on main.

[PeteGillinElastic]

Wait, I don't think this test is doing what I had assumed it was doing. When I run the test from 8.x, it seems to be taking the test code from the head of that branch. (The test class doesn't even exist at 8.6.2 or 8.5.3.) But if I get the test to do GET / then it reports that the node is version 8.18.0-SNAPSHOT.

[PeteGillinElastic]

Ah, okay. When I make the test do GET /_cluster/state?filter_path=nodes_features,nodes.*.version, I get this:

{nodes_features=[{features=[], node_id=-wPGWBI-TeOcB9mwggk1sw}, {features=[], node_id=05OafmF5QyCHuTqfkrChpw}, {features=[], node_id=44gnsY7HTsK4sOaHDGEMwg}], nodes={44gnsY7HTsK4sOaHDGEMwg={version=8.18.0}, -wPGWBI-TeOcB9mwggk1sw={version=8.6.2}, 05OafmF5QyCHuTqfkrChpw={version=8.6.2}}}

So this test is running a mixture of 8.6.2 and 8.18.0. I expect that this would not report having the health.supports_health_report_api feature (because that was enabled from 8.7.0 and the cluster feature is only considered present if it is present on all nodes) and so the test will try to GET _internal/_health. That would succeed if the request got served by an 8.6.2 node but not if it got served by an 8.18.0 node. So I think whether the test passes or fails depends which node that request goes to. I'm not even sure whether that's deterministic.

[PeteGillinElastic]

I'm coming back to the view that we should suppress the test for the 8.5.3 and 8.6.2 upgrades. I don't think we can reliably do that test when some of the nodes in the cluster have _internal/_health and some of them have _health_report.

[PeteGillinElastic]

Fixed by c124f1b .