Skip to content

Commit

Permalink
[SPARK-17685][SQL] Make SortMergeJoinExec's currentVars is null when …
Browse files Browse the repository at this point in the history
…calling createJoinKey

## What changes were proposed in this pull request?

The following SQL query cause `IndexOutOfBoundsException` issue when `LIMIT > 1310720`:
```sql
CREATE TABLE tab1(int int, int2 int, str string);
CREATE TABLE tab2(int int, int2 int, str string);
INSERT INTO tab1 values(1,1,'str');
INSERT INTO tab1 values(2,2,'str');
INSERT INTO tab2 values(1,1,'str');
INSERT INTO tab2 values(2,3,'str');

SELECT
  count(*)
FROM
  (
    SELECT t1.int, t2.int2
    FROM (SELECT * FROM tab1 LIMIT 1310721) t1
    INNER JOIN (SELECT * FROM tab2 LIMIT 1310721) t2
    ON (t1.int = t2.int AND t1.int2 = t2.int2)
  ) t;
```

This pull request fix this issue.

## How was this patch tested?

unit tests

Author: Yuming Wang <[email protected]>

Closes apache#17920 from wangyum/SPARK-17685.
  • Loading branch information
wangyum authored and hvanhovell committed May 10, 2017
1 parent c0189ab commit 771abeb
Show file tree
Hide file tree
Showing 2 changed files with 11 additions and 0 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -371,6 +371,7 @@ case class SortMergeJoinExec(
keys: Seq[Expression],
input: Seq[Attribute]): Seq[ExprCode] = {
ctx.INPUT_ROW = row
ctx.currentVars = null
keys.map(BindReferences.bindReference(_, input).genCode(ctx))
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -264,4 +264,14 @@ class DataFrameJoinSuite extends QueryTest with SharedSQLContext {
val ab = a.join(b, Seq("a"), "fullouter")
checkAnswer(ab.join(c, "a"), Row(3, null, 4, 1) :: Nil)
}

test("SPARK-17685: WholeStageCodegenExec throws IndexOutOfBoundsException") {
val df = Seq((1, 1, "1"), (2, 2, "3")).toDF("int", "int2", "str")
val df2 = Seq((1, 1, "1"), (2, 3, "5")).toDF("int", "int2", "str")
val limit = 1310721
val innerJoin = df.limit(limit).join(df2.limit(limit), Seq("int", "int2"), "inner")
.agg(count($"int"))
checkAnswer(innerJoin, Row(1) :: Nil)
}

}

0 comments on commit 771abeb

Please sign in to comment.