Skip to content

Commit

Permalink
[SPARK-46907][CORE] Show driver log location in Spark History Server
Browse files Browse the repository at this point in the history
### What changes were proposed in this pull request?

This PR aims to show `Driver Log Location` in Spark History Server UI if `spark.driver.log.dfsDir` is configured.

### Why are the changes needed?

**BEFORE (or `spark.driver.log.dfsDir` is absent)**
![Screenshot 2024-01-29 at 10 11 06 AM](https://github.com/apache/spark/assets/9700541/6d709b4b-d002-422b-a1df-bb5e1b50b539)

**AFTER**
![Screenshot 2024-01-29 at 10 10 25 AM](https://github.com/apache/spark/assets/9700541/83b35a7d-5fc9-443a-a6e5-7b6bd98dbdc6)

### Does this PR introduce _any_ user-facing change?

No, this is a new additional UI information only for the users who uses `spark.driver.log.dfsDir` configurations.

### How was this patch tested?

Manual.

```
$ mkdir /tmp/history
$ mkdir /tmp/driver-logs
$ SPARK_HISTORY_OPTS="-Dspark.history.fs.logDirectory=/tmp/history -Dspark.driver.log.dfsDir=/tmp/driver-logs"  sbin/start-history-server.sh
```

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes apache#44936 from dongjoon-hyun/SPARK-46907.

Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
  • Loading branch information
dongjoon-hyun committed Jan 29, 2024
1 parent 528ac8b commit 29355c0
Showing 1 changed file with 6 additions and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -381,7 +381,12 @@ private[history] class FsHistoryProvider(conf: SparkConf, clock: Clock)
} else {
Map()
}
Map("Event log directory" -> logDir) ++ safeMode
val driverLog = if (conf.contains(DRIVER_LOG_DFS_DIR)) {
Map("Driver log directory" -> conf.get(DRIVER_LOG_DFS_DIR).get)
} else {
Map()
}
Map("Event log directory" -> logDir) ++ safeMode ++ driverLog
}

override def start(): Unit = {
Expand Down

0 comments on commit 29355c0

Please sign in to comment.