diff options
Diffstat (limited to 'docs')
-rw-r--r-- | docs/configuration.md | 51 | ||||
-rw-r--r-- | docs/index.md | 1 | ||||
-rw-r--r-- | docs/security.md | 18 |
3 files changed, 70 insertions, 0 deletions
diff --git a/docs/configuration.md b/docs/configuration.md index 017d509854..913c653b0d 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -147,6 +147,34 @@ Apart from these, the following properties are also available, and may be useful How many stages the Spark UI remembers before garbage collecting. </td> </tr> +</tr> + <td>spark.ui.filters</td> + <td>None</td> + <td> + Comma separated list of filter class names to apply to the Spark web ui. The filter should be a + standard javax servlet Filter. Parameters to each filter can also be specified by setting a + java system property of spark.<class name of filter>.params='param1=value1,param2=value2' + (e.g.-Dspark.ui.filters=com.test.filter1 -Dspark.com.test.filter1.params='param1=foo,param2=testing') + </td> +</tr> +<tr> + <td>spark.ui.acls.enable</td> + <td>false</td> + <td> + Whether spark web ui acls should are enabled. If enabled, this checks to see if the user has + access permissions to view the web ui. See <code>spark.ui.view.acls</code> for more details. + Also note this requires the user to be known, if the user comes across as null no checks + are done. Filters can be used to authenticate and set the user. + </td> +</tr> +<tr> + <td>spark.ui.view.acls</td> + <td>Empty</td> + <td> + Comma separated list of users that have view access to the spark web ui. By default only the + user that started the Spark job has view access. + </td> +</tr> <tr> <td>spark.shuffle.compress</td> <td>true</td> @@ -495,6 +523,29 @@ Apart from these, the following properties are also available, and may be useful <td> Whether to overwrite files added through SparkContext.addFile() when the target file exists and its contents do not match those of the source. </td> +<tr> + <td>spark.authenticate</td> + <td>false</td> + <td> + Whether spark authenticates its internal connections. See <code>spark.authenticate.secret</code> if not + running on Yarn. + </td> +</tr> +<tr> + <td>spark.authenticate.secret</td> + <td>None</td> + <td> + Set the secret key used for Spark to authenticate between components. This needs to be set if + not running on Yarn and authentication is enabled. + </td> +</tr> +<tr> + <td>spark.core.connection.auth.wait.timeout</td> + <td>30</td> + <td> + Number of seconds for the connection to wait for authentication to occur before timing + out and giving up. + </td> </tr> </table> diff --git a/docs/index.md b/docs/index.md index 4eb297df39..c4f4d79edb 100644 --- a/docs/index.md +++ b/docs/index.md @@ -103,6 +103,7 @@ For this version of Spark (0.8.1) Hadoop 2.2.x (or newer) users will have to bui * [Configuration](configuration.html): customize Spark via its configuration system * [Tuning Guide](tuning.html): best practices to optimize performance and memory use +* [Security](security.html): Spark security support * [Hardware Provisioning](hardware-provisioning.html): recommendations for cluster hardware * [Job Scheduling](job-scheduling.html): scheduling resources across and within Spark applications * [Building Spark with Maven](building-with-maven.html): build Spark using the Maven system diff --git a/docs/security.md b/docs/security.md new file mode 100644 index 0000000000..9e4218fbcf --- /dev/null +++ b/docs/security.md @@ -0,0 +1,18 @@ +--- +layout: global +title: Spark Security +--- + +Spark currently supports authentication via a shared secret. Authentication can be configured to be on via the `spark.authenticate` configuration parameter. This parameter controls whether the Spark communication protocols do authentication using the shared secret. This authentication is a basic handshake to make sure both sides have the same shared secret and are allowed to communicate. If the shared secret is not identical they will not be allowed to communicate. + +The Spark UI can also be secured by using javax servlet filters. A user may want to secure the UI if it has data that other users should not be allowed to see. The javax servlet filter specified by the user can authenticate the user and then once the user is logged in, Spark can compare that user versus the view acls to make sure they are authorized to view the UI. The configs 'spark.ui.acls.enable' and 'spark.ui.view.acls' control the behavior of the acls. Note that the person who started the application always has view access to the UI. + +For Spark on Yarn deployments, configuring `spark.authenticate` to true will automatically handle generating and distributing the shared secret. Each application will use a unique shared secret. The Spark UI uses the standard YARN web application proxy mechanism and will authenticate via any installed Hadoop filters. If an authentication filter is enabled, the acls controls can be used by control which users can via the Spark UI. + +For other types of Spark deployments, the spark config `spark.authenticate.secret` should be configured on each of the nodes. This secret will be used by all the Master/Workers and applications. The UI can be secured using a javax servlet filter installed via `spark.ui.filters`. If an authentication filter is enabled, the acls controls can be used by control which users can via the Spark UI. + +IMPORTANT NOTE: The NettyBlockFetcherIterator is not secured so do not use netty for the shuffle is running with authentication on. + +See [Spark Configuration](configuration.html) for more details on the security configs. + +See <a href="api/core/index.html#org.apache.spark.SecurityManager"><code>org.apache.spark.SecurityManager</code></a> for implementation details about security. |