Skip to content

Commit 29355c0

Browse files
committed
[SPARK-46907][CORE] Show driver log location in Spark History Server
### What changes were proposed in this pull request? This PR aims to show `Driver Log Location` in Spark History Server UI if `spark.driver.log.dfsDir` is configured. ### Why are the changes needed? **BEFORE (or `spark.driver.log.dfsDir` is absent)** ![Screenshot 2024-01-29 at 10 11 06 AM](https://github.com/apache/spark/assets/9700541/6d709b4b-d002-422b-a1df-bb5e1b50b539) **AFTER** ![Screenshot 2024-01-29 at 10 10 25 AM](https://github.com/apache/spark/assets/9700541/83b35a7d-5fc9-443a-a6e5-7b6bd98dbdc6) ### Does this PR introduce _any_ user-facing change? No, this is a new additional UI information only for the users who uses `spark.driver.log.dfsDir` configurations. ### How was this patch tested? Manual. ``` $ mkdir /tmp/history $ mkdir /tmp/driver-logs $ SPARK_HISTORY_OPTS="-Dspark.history.fs.logDirectory=/tmp/history -Dspark.driver.log.dfsDir=/tmp/driver-logs" sbin/start-history-server.sh ``` ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#44936 from dongjoon-hyun/SPARK-46907. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>
1 parent 528ac8b commit 29355c0

File tree

1 file changed

+6
-1
lines changed

1 file changed

+6
-1
lines changed

core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -381,7 +381,12 @@ private[history] class FsHistoryProvider(conf: SparkConf, clock: Clock)
381381
} else {
382382
Map()
383383
}
384-
Map("Event log directory" -> logDir) ++ safeMode
384+
val driverLog = if (conf.contains(DRIVER_LOG_DFS_DIR)) {
385+
Map("Driver log directory" -> conf.get(DRIVER_LOG_DFS_DIR).get)
386+
} else {
387+
Map()
388+
}
389+
Map("Event log directory" -> logDir) ++ safeMode ++ driverLog
385390
}
386391

387392
override def start(): Unit = {

0 commit comments

Comments
 (0)