[CI] QueryBuilderBWCIT testQueryBuilderBWC {p0=UPGRADED} failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-periodic #5210 / 8.6.2_bwc
• elasticsearch-periodic #5210 / 8.0.1_bwc
• elasticsearch-periodic #5210 / 8.3.3_bwc
• elasticsearch-periodic #5210 / 8.7.1_bwc
• elasticsearch-periodic #5210 / 8.4.3_bwc
• elasticsearch-periodic #5210 / 8.5.3_bwc
• elasticsearch-periodic #5210 / 8.2.3_bwc
• elasticsearch-periodic #5210 / 8.1.3_bwc
• elasticsearch-pull-request #44674 / 8.4.3_bwc
• elasticsearch-pull-request #44674 / 8.6.2_bwc

Reproduction Line:

./gradlew ":x-pack:qa:full-cluster-restart:v8.6.2#bwcTest" -Dtests.class="org.elasticsearch.xpack.restart.QueryBuilderBWCIT" -Dtests.method="testQueryBuilderBWC {p0=UPGRADED}" -Dtests.seed=B33306C723CFDAB2 -Dtests.bwc=true -Dtests.locale=zgh-MA -Dtests.timezone=Europe/Busingen -Druntime.java=23

Applicable branches:
main

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.io.EOFException: null

Issue Reasons:

• [main] 3 consecutive failures in step 8.6.2_bwc
• [main] 3 consecutive failures in step 8.0.1_bwc
• [main] 3 consecutive failures in step 8.3.3_bwc
• [main] 3 consecutive failures in step 8.7.1_bwc
• [main] 3 consecutive failures in step 8.4.3_bwc
• [main] 3 consecutive failures in step 8.5.3_bwc
• [main] 3 consecutive failures in step 8.2.3_bwc
• [main] 3 consecutive failures in step 8.1.3_bwc
• [main] 24 failures in test testQueryBuilderBWC {p0=UPGRADED} (4.7% fail rate in 509 executions)
• [main] 3 failures in step 8.6.2_bwc (100.0% fail rate in 3 executions)
• [main] 3 failures in step 8.0.1_bwc (100.0% fail rate in 3 executions)
• [main] 3 failures in step 8.3.3_bwc (100.0% fail rate in 3 executions)
• [main] 3 failures in step 8.7.1_bwc (100.0% fail rate in 3 executions)
• [main] 3 failures in step 8.4.3_bwc (100.0% fail rate in 3 executions)
• [main] 3 failures in step 8.5.3_bwc (100.0% fail rate in 3 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

This has been muted on branch main

Mute Reasons:

• [main] 6 failures in test testQueryBuilderBWC {p0=UPGRADED} (1.1% fail rate in 539 executions)
• [main] 2 failures in step 8.17.0_bwc-snapshots (1.5% fail rate in 135 executions)
• [main] 2 failures in step 8.16.1_bwc-snapshots (1.4% fail rate in 147 executions)
• [main] 3 failures in pipeline elasticsearch-pull-request (2.3% fail rate in 129 executions)

Build Scans:

• elasticsearch-periodic #4929 / 8.5.3_bwc
• elasticsearch-pull-request #41713 / 9.0.0_bwc-snapshots
• elasticsearch-pull-request #41640 / 8.17.0_bwc-snapshots
• elasticsearch-pull-request #41640 / 8.16.1_bwc-snapshots
• elasticsearch-pull-request #41624 / 8.17.0_bwc-snapshots
• elasticsearch-pull-request #41624 / 8.16.1_bwc-snapshots

[elasticsearchmachine]

Pinging @elastic/es-delivery (Team:Delivery)

[mark-vieira]

Looks to be the same as #116990

[elasticsearchmachine]

Pinging @elastic/es-search-foundations (Team:Search Foundations)

[drempapis]

Investigated in the pr #117831

[elasticsearchmachine]

This has been muted on branch main

Mute Reasons:

• [main] 3 consecutive failures in test testQueryBuilderBWC {p0=UPGRADED}
• [main] 3 consecutive failures in step 8.5.3_bwc
• [main] 3 consecutive failures in step 8.2.3_bwc
• [main] 3 consecutive failures in step 8.1.3_bwc
• [main] 2 consecutive failures in step 8.4.3_bwc
• [main] 2 consecutive failures in step 8.6.2_bwc
• [main] 2 consecutive failures in step 8.0.1_bwc
• [main] 2 consecutive failures in step 8.7.1_bwc
• [main] 2 consecutive failures in step 8.3.3_bwc
• [main] 19 failures in test testQueryBuilderBWC {p0=UPGRADED} (3.9% fail rate in 486 executions)
• [main] 3 failures in step 8.5.3_bwc (100.0% fail rate in 3 executions)
• [main] 3 failures in step 8.2.3_bwc (100.0% fail rate in 3 executions)
• [main] 3 failures in step 8.1.3_bwc (100.0% fail rate in 3 executions)
• [main] 2 failures in step 8.4.3_bwc (100.0% fail rate in 2 executions)
• [main] 2 failures in step 8.6.2_bwc (100.0% fail rate in 2 executions)

Build Scans:

• elasticsearch-periodic #5210 / 8.5.3_bwc
• elasticsearch-periodic #5210 / 8.2.3_bwc
• elasticsearch-periodic #5210 / 8.1.3_bwc
• elasticsearch-pull-request #44674 / 8.4.3_bwc
• elasticsearch-pull-request #44674 / 8.6.2_bwc
• elasticsearch-pull-request #44674 / 8.5.3_bwc
• elasticsearch-pull-request #44674 / 8.0.1_bwc
• elasticsearch-pull-request #44674 / 8.7.1_bwc
• elasticsearch-pull-request #44674 / 8.3.3_bwc
• elasticsearch-pull-request #44674 / 8.1.3_bwc

[drempapis]

Hey @thecoop, happy new year!

Can you please provide some feedback on this issue? Is it related to 116941

[thecoop]

@drempapis I'm unclear what's going on here, or what I need to do. When did this start failing? Do you know what the problem is?

[drempapis]

That started on November 18, and I supposed the cause was the boolean bwc_tests_enabled in elasticsearch/build.gradle did not flip the value before merging for 116941.

Debugging it locally, e.g., for v.8.5.3#bwcTest, I get an EOF when reading the input.

 QueryBuilder queryBuilder = input.readNamedWriteable(QueryBuilder.class);

Reverting the code when reading the transportVersion run successfully.

 transportVersion = TransportVersion.fromId(
                            parseLegacyVersion(getOldClusterVersion()).map(Version::id).orElse(TransportVersions.MINIMUM_COMPATIBLE.id())
                        );

[thecoop]

QueryBuilderBWCIT needs to deserialize data from pre-8.18 nodes, even though v9 cannot communicate with <8.18 nodes?

[drempapis]

The response seems truncated or has a deserialization mismatch when reading from input for < 8.18 nodes

[thecoop]

So this test needs to deserialize data written by 8.0 nodes, even though the minimum version a 9.0 node can communicate with is 8.18?

[mark-vieira]

So this test needs to deserialize data written by 8.0 nodes, even though the minimum version a 9.0 node can communicate with is 8.18?

Yes, because we support upgrades from 8.0 nodes if doing a full-cluster-restart, which is what this test is doing.

[thecoop]

Reinstated the check in #119669

[drempapis]

Closing after the #119669 was merged