Skip to content

Commit

Permalink
[SPARK-21941] Stop storing unused attemptId in SQLTaskMetrics
Browse files Browse the repository at this point in the history
## What changes were proposed in this pull request?

In a driver heap dump containing 390,105 instances of SQLTaskMetrics this
would have saved me approximately 3.2MB of memory.

Since we're not getting any benefit from storing this unused value, let's
eliminate it until a future PR makes use of it.

## How was this patch tested?

Existing unit tests

Author: Andrew Ash <[email protected]>

Closes apache#19153 from ash211/aash/trim-sql-listener.
  • Loading branch information
ash211 authored and gatorsmile committed Sep 9, 2017
1 parent 31c74fe commit 8a5eb50
Showing 1 changed file with 3 additions and 4 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -255,10 +255,8 @@ class SQLListener(conf: SparkConf) extends SparkListener with Logging {
// heartbeat reports
}
case None =>
// TODO Now just set attemptId to 0. Should fix here when we can get the attempt
// id from SparkListenerExecutorMetricsUpdate
stageMetrics.taskIdToMetricUpdates(taskId) = new SQLTaskMetrics(
attemptId = 0, finished = finishTask, accumulatorUpdates)
finished = finishTask, accumulatorUpdates)
}
}
case None =>
Expand Down Expand Up @@ -478,10 +476,11 @@ private[ui] class SQLStageMetrics(
val stageAttemptId: Long,
val taskIdToMetricUpdates: mutable.HashMap[Long, SQLTaskMetrics] = mutable.HashMap.empty)


// TODO Should add attemptId here when we can get it from SparkListenerExecutorMetricsUpdate
/**
* Store all accumulatorUpdates for a Spark task.
*/
private[ui] class SQLTaskMetrics(
val attemptId: Long, // TODO not used yet
var finished: Boolean,
var accumulatorUpdates: Seq[(Long, Any)])

0 comments on commit 8a5eb50

Please sign in to comment.