Skip to content

Commit

Permalink
[SPARK-2950] Add gc time and shuffle write time to JobLogger
Browse files Browse the repository at this point in the history
The JobLogger is very useful for performing offline performance profiling of Spark jobs. GC Time and Shuffle Write time are available in TaskMetrics but are currently missed from the JobLogger output. This patch adds these two fields.

~~Since this is a small change, I didn't create a JIRA. Let me know if I should do that.~~

cc kayousterhout

Author: Shivaram Venkataraman <[email protected]>

Closes apache#1869 from shivaram/job-logger and squashes the following commits:

1b709fc [Shivaram Venkataraman] Add a space before GC_TIME
c418105 [Shivaram Venkataraman] Add gc time and shuffle write time to JobLogger
  • Loading branch information
shivaram committed Aug 10, 2014
1 parent 3570119 commit 1d03a26
Showing 1 changed file with 6 additions and 3 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -162,6 +162,7 @@ class JobLogger(val user: String, val logDirName: String) extends SparkListener
" START_TIME=" + taskInfo.launchTime + " FINISH_TIME=" + taskInfo.finishTime +
" EXECUTOR_ID=" + taskInfo.executorId + " HOST=" + taskMetrics.hostname
val executorRunTime = " EXECUTOR_RUN_TIME=" + taskMetrics.executorRunTime
val gcTime = " GC_TIME=" + taskMetrics.jvmGCTime
val inputMetrics = taskMetrics.inputMetrics match {
case Some(metrics) =>
" READ_METHOD=" + metrics.readMethod.toString +
Expand All @@ -179,11 +180,13 @@ class JobLogger(val user: String, val logDirName: String) extends SparkListener
case None => ""
}
val writeMetrics = taskMetrics.shuffleWriteMetrics match {
case Some(metrics) => " SHUFFLE_BYTES_WRITTEN=" + metrics.shuffleBytesWritten
case Some(metrics) =>
" SHUFFLE_BYTES_WRITTEN=" + metrics.shuffleBytesWritten +
" SHUFFLE_WRITE_TIME=" + metrics.shuffleWriteTime
case None => ""
}
stageLogInfo(stageId, status + info + executorRunTime + inputMetrics + shuffleReadMetrics +
writeMetrics)
stageLogInfo(stageId, status + info + executorRunTime + gcTime + inputMetrics +
shuffleReadMetrics + writeMetrics)
}

/**
Expand Down

0 comments on commit 1d03a26

Please sign in to comment.