diff --git a/docs/configuration.md b/docs/configuration.md index 782ccff667076..55773937d4d71 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -793,6 +793,13 @@ Apart from these, the following properties are also available, and may be useful Buffer size to use when writing to output streams, in KiB unless otherwise specified. + + spark.ui.dagGraph.retainedRootRDDs + Int.MaxValue + + How many DAG graph nodes the Spark UI and status APIs remember before garbage collecting. + + spark.ui.enabled true @@ -807,6 +814,15 @@ Apart from these, the following properties are also available, and may be useful Allows jobs and stages to be killed from the web UI. + + spark.ui.liveUpdate.period + 100ms + + How often to update live entities. -1 means "never update" when replaying applications, + meaning only the last write will happen. For live applications, this avoids a few + operations that we can live without when rapidly processing incoming task events. + + spark.ui.port 4040 diff --git a/docs/monitoring.md b/docs/monitoring.md index f6d52ef4597e9..69bf3082f0f27 100644 --- a/docs/monitoring.md +++ b/docs/monitoring.md @@ -185,6 +185,23 @@ Security options for the Spark History Server are covered more detail in the Job history files older than this will be deleted when the filesystem history cleaner runs. + + spark.history.fs.endEventReparseChunkSize + 1m + + How many bytes to parse at the end of log files looking for the end event. + This is used to speed up generation of application listings by skipping unnecessary + parts of event log files. It can be disabled by setting this config to 0. + + + + spark.history.fs.inProgressOptimization.enabled + true + + Enable optimized handling of in-progress logs. This option may leave finished + applications that fail to rename their event logs listed as in-progress. + + spark.history.fs.numReplayThreads 25% of available cores @@ -192,6 +209,14 @@ Security options for the Spark History Server are covered more detail in the Number of threads that will be used by history server to process event logs. + + spark.history.store.maxDiskUsage + 10g + + Maximum disk usage for the local directory where the cache application history information + are stored. + + spark.history.store.path (none)