ESQL: Compute infrastruture for LEFT JOIN #118889
Conversation
This adds some infrastructure that we can use to run LOOKUP JOIN using real LEFT JOIN semantics. Right now if LOOKUP JOIN matches many rows in the `lookup` index we merge all of the values into a multivalued field. So the number of rows emitted from LOOKUP JOIN is the same as the number of rows that comes into LOOKUP JOIN. This change builds the infrastructure to emit one row per match, mostly reusing the infrastructure from ENRICH.
elasticsearchmachine
added
the
Team:Analytics
|
Pinging @elastic/es-analytical-engine (Team:Analytics) |
|
I'd like to add another randomized test to this and then get it in. Then I'd like to plug it in in a follow up change. That change will:
|
| */ | ||
| default Block insertNulls(IntVector before) { | ||
| // TODO remove default and scatter to implementation where it can be a lot more efficient | ||
| try (Builder builder = elementType().newBlockBuilder(getPositionCount() + before.getPositionCount(), blockFactory())) { |
There was a problem hiding this comment.
Nit:
Since there are multiple calls to getPositionCount() and before.getPositionCount() how about extracting them into separate vars:
try (long pc = getPositionCount(), beforePC = before.getPositionCount(), Builder builder = ...) { }
| * | l99 | null | null | | ||
| * }</pre> | ||
| */ | ||
| class RightChunkedLeftJoin implements Releasable { |
| } | ||
| BlockFactory factory = leftHand.getBlock(0).blockFactory(); | ||
| Block[] blocks = new Block[leftHand.getBlockCount() + mergedElementCount]; | ||
| int[] filter = IntStream.range(next, leftHand.getPositionCount()).toArray(); |
There was a problem hiding this comment.
int[] filter = new int[leftHand.getPositionCount()];
Arrays.setAll(filter, i -> next + i);
| try { | ||
| int b = 0; | ||
| while (b < leftHand.getBlockCount()) { | ||
| blocks[b] = leftHand.getBlock(b).filter(filter); |
There was a problem hiding this comment.
Nit:
To avoid allocating the array, a new method could be added to block for consecutive integers such as Block#filter(int start, int stop) or depending how well it gets JITed a Range class Block#filter(range)
There was a problem hiding this comment.
Yeah. There's a lot we could do here to be honest. The range version of filter might be quite nice.
I think those might be for a follow up.
|
I've pushed a randomize test that seems to fail .7% of the time. Fun. I'll have a debug when I'm back. |
This adds some infrastructure that we can use to run LOOKUP JOIN using real LEFT JOIN semantics.
Right now if LOOKUP JOIN matches many rows in the
lookupindex we merge all of the values into a multivalued field. So the number of rows emitted from LOOKUP JOIN is the same as the number of rows that comes into LOOKUP JOIN.This change builds the infrastructure to emit one row per match, mostly reusing the infrastructure from ENRICH.