Skip to content

Commit

Permalink
[SPARK-49176][K8S][DOCS] Fix spark.ui.custom.executor.log.url docs …
Browse files Browse the repository at this point in the history
…by adding K8s

### What changes were proposed in this pull request?

This PR aims to fix `spark.ui.custom.executor.log.url` configuration's documentation by adding K8s on top of YARN.
```
-    For now, only YARN mode supports this configuration
+    For now, only YARN and K8s cluster manager supports this configuration
```

In addition, K8s example is added to `running-on-kubernetes.md` docs to help users refer the existing environment variables, `SPARK_APPLICATION_ID` and `SPARK_EXECUTOR_ID`, more easily to enable this feature.

- https://kubernetes.io/docs/tasks/inject-data-application/define-environment-variable-container/#using-environment-variables-inside-of-your-config

### Why are the changes needed?

To fix a wrong description.

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Manual review. This is a doc-only PR.

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes apache#47681 from dongjoon-hyun/SPARK-49176.

Authored-by: Dongjoon Hyun <dhyun@apple.com>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
  • Loading branch information
dongjoon-hyun committed Aug 9, 2024
1 parent 129db9a commit 614bfe6
Show file tree
Hide file tree
Showing 2 changed files with 9 additions and 1 deletion.
2 changes: 1 addition & 1 deletion docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -1526,7 +1526,7 @@ Apart from these, the following properties are also available, and may be useful
which will be also effective when accessing the application on history server. The new log urls must be
permanent, otherwise you might have dead link for executor log urls.
<p/>
For now, only YARN mode supports this configuration
For now, only YARN and K8s cluster manager supports this configuration
</td>
<td>3.0.0</td>
</tr>
Expand Down
8 changes: 8 additions & 0 deletions docs/running-on-kubernetes.md
Original file line number Diff line number Diff line change
Expand Up @@ -430,6 +430,14 @@ The same logs can also be accessed through the
[Kubernetes dashboard](https://kubernetes.io/docs/tasks/access-application-cluster/web-ui-dashboard/) if installed on
the cluster.

When there exists a log collection system, you can expose it at Spark Driver `Executors` tab UI. For example,

```
spark.executorEnv.SPARK_EXECUTOR_ATTRIBUTE_APP_ID='$(SPARK_APPLICATION_ID)'
spark.executorEnv.SPARK_EXECUTOR_ATTRIBUTE_EXECUTOR_ID='$(SPARK_EXECUTOR_ID)'
spark.ui.custom.executor.log.url='https://log-server/log?appId={{APP_ID}}&execId={{EXECUTOR_ID}}'
```

### Accessing Driver UI

The UI associated with any application can be accessed locally using
Expand Down

0 comments on commit 614bfe6

Please sign in to comment.