ESQL: Compute infrastruture for LEFT JOIN #118889

nik9000 · 2024-12-17T19:59:37Z

This adds some infrastructure that we can use to run LOOKUP JOIN using real LEFT JOIN semantics.

Right now if LOOKUP JOIN matches many rows in the lookup index we merge all of the values into a multivalued field. So the number of rows emitted from LOOKUP JOIN is the same as the number of rows that comes into LOOKUP JOIN.

This change builds the infrastructure to emit one row per match, mostly reusing the infrastructure from ENRICH.

This adds some infrastructure that we can use to run LOOKUP JOIN using real LEFT JOIN semantics. Right now if LOOKUP JOIN matches many rows in the `lookup` index we merge all of the values into a multivalued field. So the number of rows emitted from LOOKUP JOIN is the same as the number of rows that comes into LOOKUP JOIN. This change builds the infrastructure to emit one row per match, mostly reusing the infrastructure from ENRICH.

elasticsearchmachine · 2024-12-17T20:00:02Z

Pinging @elastic/es-analytical-engine (Team:Analytics)

nik9000 · 2024-12-17T20:03:32Z

I'd like to add another randomized test to this and then get it in. Then I'd like to plug it in in a follow up change. That change will:

Change the LookupResponse to return a List
Remove the MergePositionsOperator from lookup and return all of the enriched pages.
Plug this into the LookupFromIndexOperator and have it consume all the pages from the response, returning the results of this thing.
Fix all the tests it breaks.
Party

x-pack/plugin/esql/compute/src/main/java/org/elasticsearch/compute/data/Block.java

costin · 2024-12-17T20:45:30Z

...ql/compute/src/main/java/org/elasticsearch/compute/operator/lookup/RightChunkedLeftJoin.java

+ * |     l99 | null | null |
+ * }</pre>
+ */
+class RightChunkedLeftJoin implements Releasable {


👍 for the asciiart

...ql/compute/src/main/java/org/elasticsearch/compute/operator/lookup/RightChunkedLeftJoin.java

costin · 2024-12-17T21:02:54Z

...ql/compute/src/main/java/org/elasticsearch/compute/operator/lookup/RightChunkedLeftJoin.java

+        try {
+            int b = 0;
+            while (b < leftHand.getBlockCount()) {
+                blocks[b] = leftHand.getBlock(b).filter(filter);


Nit:
To avoid allocating the array, a new method could be added to block for consecutive integers such as Block#filter(int start, int stop) or depending how well it gets JITed a Range class Block#filter(range)

Yeah. There's a lot we could do here to be honest. The range version of filter might be quite nice.

I think those might be for a follow up.

costin

LGTM

nik9000 · 2024-12-17T21:21:45Z

I've pushed a randomize test that seems to fail .7% of the time. Fun. I'll have a debug when I'm back.

ivancea

LGTM!

...esql/src/internalClusterTest/java/org/elasticsearch/xpack/esql/action/LookupFromIndexIT.java

ivancea · 2024-12-26T16:35:14Z

...esql/src/internalClusterTest/java/org/elasticsearch/xpack/esql/action/LookupFromIndexIT.java

+        void populate(int docCount, List<String> expected) throws IOException;
+    }
+
+    private void runLookup(PopulateIndices populateIndices) throws IOException {
        // TODO this should *fail* if the target index isn't a lookup type index - it doesn't now.


Unrelated: If this comment talks about the ESQL query, it does fail now

Yeah. That was about the query. Great news!

ivancea · 2024-12-26T16:41:17Z

...mpute/src/test/java/org/elasticsearch/compute/operator/lookup/RightChunkedLeftJoinTests.java

We build multiple vectors/blocks here. Should we add a cranky CB test?

nik9000 · 2025-01-15T19:23:20Z

Looks like I never backported this... bleh. Incoming

This adds some infrastructure that we can use to run LOOKUP JOIN using real LEFT JOIN semantics. Right now if LOOKUP JOIN matches many rows in the `lookup` index we merge all of the values into a multivalued field. So the number of rows emitted from LOOKUP JOIN is the same as the number of rows that comes into LOOKUP JOIN. This change builds the infrastructure to emit one row per match, mostly reusing the infrastructure from ENRICH.

nik9000 · 2025-01-15T19:51:51Z

backport is #120232

* ESQL: Compute infrastruture for LEFT JOIN (#118889) This adds some infrastructure that we can use to run LOOKUP JOIN using real LEFT JOIN semantics. Right now if LOOKUP JOIN matches many rows in the `lookup` index we merge all of the values into a multivalued field. So the number of rows emitted from LOOKUP JOIN is the same as the number of rows that comes into LOOKUP JOIN. This change builds the infrastructure to emit one row per match, mostly reusing the infrastructure from ENRICH. * ESQL: Make LOOKUP more left-joiny (#119475) This makes `LOOKUP` return multiple rows if there are multiple matches. This is the way SQL works so it's *probably* what folks will expect. Even if it isn't, it allows for more optimizations. Like, this change doesn't optimize anything - it just changes the behavior. But there are optimizations you can do *later* that are transparent when we have *this* behavior, but not with the old behavior. Example: ``` - 2 | [German, German, German] | [Austria, Germany, Switzerland] + 2 | German | [Austria, Germany] + 2 | German | Switzerland + 2 | German | null ``` Relates: #118781

nik9000 added >non-issue :Analytics/ES|QL AKA ESQL v9.0.0 v8.18.0 labels Dec 17, 2024

nik9000 requested review from ivancea and alex-spies December 17, 2024 19:59

elasticsearchmachine added the Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) label Dec 17, 2024

costin reviewed Dec 17, 2024

View reviewed changes

x-pack/plugin/esql/compute/src/main/java/org/elasticsearch/compute/data/Block.java Outdated Show resolved Hide resolved

costin reviewed Dec 17, 2024

View reviewed changes

...ql/compute/src/main/java/org/elasticsearch/compute/operator/lookup/RightChunkedLeftJoin.java Show resolved Hide resolved

costin reviewed Dec 17, 2024

View reviewed changes

costin approved these changes Dec 17, 2024

View reviewed changes

nik9000 added 2 commits December 17, 2024 16:18

Random tests

04d8fcb

Can't be small

0dfb43c

Bad

ab03369

ivancea approved these changes Dec 26, 2024

View reviewed changes

nik9000 added 3 commits December 26, 2024 15:58

Merge branch 'main' into left_join_for_lookup_1

ee1bbbe

update after feedback

5243ebf

More tests

4b60086

nik9000 enabled auto-merge (squash) December 27, 2024 18:07

nik9000 merged commit 8afbb52 into elastic:main Dec 27, 2024
16 checks passed

nik9000 mentioned this pull request Jan 3, 2025

ESQL: LOOKUP JOIN should have SQL LEFT JOIN semantics #118781

Closed

alex-spies mentioned this pull request Jan 16, 2025

ESQL: Make LOOKUP more left-joiny (#118889) (#119475) #120232

Merged

ESQL: Compute infrastruture for LEFT JOIN #118889

ESQL: Compute infrastruture for LEFT JOIN #118889

Uh oh!

Conversation

nik9000 commented Dec 17, 2024

Uh oh!

elasticsearchmachine commented Dec 17, 2024

Uh oh!

nik9000 commented Dec 17, 2024

Uh oh!

Uh oh!

costin Dec 17, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

costin Dec 17, 2024

Choose a reason for hiding this comment

Uh oh!

nik9000 Dec 17, 2024

Choose a reason for hiding this comment

Uh oh!

costin left a comment

Choose a reason for hiding this comment

Uh oh!

nik9000 commented Dec 17, 2024

Uh oh!

ivancea left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ivancea Dec 26, 2024

Choose a reason for hiding this comment

Uh oh!

nik9000 Dec 26, 2024

Choose a reason for hiding this comment

Uh oh!

ivancea Dec 26, 2024

Choose a reason for hiding this comment

Uh oh!

nik9000 Dec 26, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

nik9000 commented Jan 15, 2025

Uh oh!

nik9000 commented Jan 15, 2025

Uh oh!

Uh oh!