diff options
| author | Gurvinder Singh <gurvinder.singh@uninett.no> | 2016-09-08 17:20:20 -0700 |
|---|---|---|
| committer | Shixiong Zhu <shixiong@databricks.com> | 2016-09-08 17:20:20 -0700 |
| commit | 92ce8d4849a0341c4636e70821b7be57ad3055b1 (patch) | |
| tree | 6b02cf84575ae2a4aecd8bb9c8646efe3a41b3b9 /docs/configuration.md | |
| parent | 722afbb2b33037a30d385a15725f2db5365bd375 (diff) | |
| download | spark-92ce8d4849a0341c4636e70821b7be57ad3055b1.tar.gz spark-92ce8d4849a0341c4636e70821b7be57ad3055b1.tar.bz2 spark-92ce8d4849a0341c4636e70821b7be57ad3055b1.zip | |
[SPARK-15487][WEB UI] Spark Master UI to reverse proxy Application and Workers UI
## What changes were proposed in this pull request?
This pull request adds the functionality to enable accessing worker and application UI through master UI itself. Thus helps in accessing SparkUI when running spark cluster in closed networks e.g. Kubernetes. Cluster admin needs to expose only spark master UI and rest of the UIs can be in the private network, master UI will reverse proxy the connection request to corresponding resource. It adds the path for workers/application UIs as
WorkerUI: <http/https>://master-publicIP:<port>/target/workerID/
ApplicationUI: <http/https>://master-publicIP:<port>/target/appID/
This makes it easy for users to easily protect the Spark master cluster access by putting some reverse proxy e.g. https://github.com/bitly/oauth2_proxy
## How was this patch tested?
The functionality has been tested manually and there is a unit test too for testing access to worker UI with reverse proxy address.
pwendell bomeng BryanCutler can you please review it, thanks.
Author: Gurvinder Singh <gurvinder.singh@uninett.no>
Closes #13950 from gurvindersingh/rproxy.
Diffstat (limited to 'docs/configuration.md')
| -rw-r--r-- | docs/configuration.md | 14 |
1 files changed, 14 insertions, 0 deletions
diff --git a/docs/configuration.md b/docs/configuration.md index 6e98f67b73..ebd0aa796d 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -658,6 +658,20 @@ Apart from these, the following properties are also available, and may be useful </td> </tr> <tr> + <td><code>spark.ui.reverseProxy</code></td> + <td>false</td> + <td> + Enable running Spark Master as reverse proxy for worker and application UIs. In this mode, Spark master will reverse proxy the worker and application UIs to enable access without requiring direct access to their hosts. Use it with caution, as worker and application UI will not be accessible directly, you will only be able to access them through spark master/proxy public URL. This setting affects all the workers and application UIs running in the cluster and must be set on all the workers, drivers and masters. + </td> +</tr> +<tr> + <td><code>spark.ui.reverseProxyUrl</code></td> + <td></td> + <td> + This is the URL where your proxy is running. This URL is for proxy which is running in front of Spark Master. This is useful when running proxy for authentication e.g. OAuth proxy. Make sure this is a complete URL including scheme (http/https) and port to reach your proxy. + </td> +</tr> +<tr> <td><code>spark.worker.ui.retainedExecutors</code></td> <td>1000</td> <td> |