Skip to content

Commit

Permalink
[SPARK-42880][DOCS] Update running-on-yarn.md to log4j2 syntax
Browse files Browse the repository at this point in the history
### What changes were proposed in this pull request?

Update log4j1 syntax to log4j2, and use ${sys:spark.yarn.app.container.log.dir} to relocate log path.

see https://issues.apache.org/jira/browse/SPARK-42880

### Why are the changes needed?

Since Spark3.3 has changed log4j1 to log4j2, some documents should also be updated.

### Does this PR introduce _any_ user-facing change?
No.

### How was this patch tested?
Just doc.

Closes apache#40504 from frankliee/log4j2.

Authored-by: frankzfli <[email protected]>
Signed-off-by: Sean Owen <[email protected]>
  • Loading branch information
frankliee authored and srowen committed Mar 22, 2023
1 parent 5825db8 commit 7ad1c80
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion docs/running-on-yarn.md
Original file line number Diff line number Diff line change
Expand Up @@ -137,7 +137,7 @@ Note that for the first option, both executors and the application master will s
log4j configuration, which may cause issues when they run on the same node (e.g. trying to write
to the same log file).

If you need a reference to the proper location to put log files in the YARN so that YARN can properly display and aggregate them, use `spark.yarn.app.container.log.dir` in your `log4j.properties`. For example, `log4j.appender.file_appender.File=${spark.yarn.app.container.log.dir}/spark.log`. For streaming applications, configuring `RollingFileAppender` and setting file location to YARN's log directory will avoid disk overflow caused by large log files, and logs can be accessed using YARN's log utility.
If you need a reference to the proper location to put log files in the YARN so that YARN can properly display and aggregate them, use `spark.yarn.app.container.log.dir` in your `log4j2.properties`. For example, `appender.file_appender.fileName=${sys:spark.yarn.app.container.log.dir}/spark.log`. For streaming applications, configuring `RollingFileAppender` and setting file location to YARN's log directory will avoid disk overflow caused by large log files, and logs can be accessed using YARN's log utility.

To use a custom metrics.properties for the application master and executors, update the `$SPARK_CONF_DIR/metrics.properties` file. It will automatically be uploaded with other configurations, so you don't need to specify it manually with `--files`.

Expand Down

0 comments on commit 7ad1c80

Please sign in to comment.