<!DOCTYPE html>
<!--[if lt IE 7]> <html class="no-js lt-ie9 lt-ie8 lt-ie7"> <![endif]-->
<!--[if IE 7]> <html class="no-js lt-ie9 lt-ie8"> <![endif]-->
<!--[if IE 8]> <html class="no-js lt-ie9"> <![endif]-->
<!--[if gt IE 8]><!--> <html class="no-js"> <!--<![endif]-->
<head>
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1">
<title>Security - Spark 1.5.1 Documentation</title>
<link rel="stylesheet" href="css/bootstrap.min.css">
<style>
body {
padding-top: 60px;
padding-bottom: 40px;
}
</style>
<meta name="viewport" content="width=device-width">
<link rel="stylesheet" href="css/bootstrap-responsive.min.css">
<link rel="stylesheet" href="css/main.css">
<script src="js/vendor/modernizr-2.6.1-respond-1.1.0.min.js"></script>
<link rel="stylesheet" href="css/pygments-default.css">
<!-- Google analytics script -->
<script type="text/javascript">
var _gaq = _gaq || [];
_gaq.push(['_setAccount', 'UA-32518208-2']);
_gaq.push(['_trackPageview']);
(function() {
var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;
ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
})();
</script>
</head>
<body>
<!--[if lt IE 7]>
<p class="chromeframe">You are using an outdated browser. <a href="http://browsehappy.com/">Upgrade your browser today</a> or <a href="http://www.google.com/chromeframe/?redirect=true">install Google Chrome Frame</a> to better experience this site.</p>
<![endif]-->
<!-- This code is taken from http://twitter.github.com/bootstrap/examples/hero.html -->
<div class="navbar navbar-fixed-top" id="topbar">
<div class="navbar-inner">
<div class="container">
<div class="brand"><a href="index.html">
<img src="img/spark-logo-hd.png" style="height:50px;"/></a><span class="version">1.5.1</span>
</div>
<ul class="nav">
<!--TODO(andyk): Add class="active" attribute to li some how.-->
<li><a href="index.html">Overview</a></li>
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Programming Guides<b class="caret"></b></a>
<ul class="dropdown-menu">
<li><a href="quick-start.html">Quick Start</a></li>
<li><a href="programming-guide.html">Spark Programming Guide</a></li>
<li class="divider"></li>
<li><a href="streaming-programming-guide.html">Spark Streaming</a></li>
<li><a href="sql-programming-guide.html">DataFrames and SQL</a></li>
<li><a href="mllib-guide.html">MLlib (Machine Learning)</a></li>
<li><a href="graphx-programming-guide.html">GraphX (Graph Processing)</a></li>
<li><a href="bagel-programming-guide.html">Bagel (Pregel on Spark)</a></li>
<li><a href="sparkr.html">SparkR (R on Spark)</a></li>
</ul>
</li>
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">API Docs<b class="caret"></b></a>
<ul class="dropdown-menu">
<li><a href="api/scala/index.html#org.apache.spark.package">Scala</a></li>
<li><a href="api/java/index.html">Java</a></li>
<li><a href="api/python/index.html">Python</a></li>
<li><a href="api/R/index.html">R</a></li>
</ul>
</li>
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Deploying<b class="caret"></b></a>
<ul class="dropdown-menu">
<li><a href="cluster-overview.html">Overview</a></li>
<li><a href="submitting-applications.html">Submitting Applications</a></li>
<li class="divider"></li>
<li><a href="spark-standalone.html">Spark Standalone</a></li>
<li><a href="running-on-mesos.html">Mesos</a></li>
<li><a href="running-on-yarn.html">YARN</a></li>
<li class="divider"></li>
<li><a href="ec2-scripts.html">Amazon EC2</a></li>
</ul>
</li>
<li class="dropdown">
<a href="api.html" class="dropdown-toggle" data-toggle="dropdown">More<b class="caret"></b></a>
<ul class="dropdown-menu">
<li><a href="configuration.html">Configuration</a></li>
<li><a href="monitoring.html">Monitoring</a></li>
<li><a href="tuning.html">Tuning Guide</a></li>
<li><a href="job-scheduling.html">Job Scheduling</a></li>
<li><a href="security.html">Security</a></li>
<li><a href="hardware-provisioning.html">Hardware Provisioning</a></li>
<li><a href="hadoop-third-party-distributions.html">3<sup>rd</sup>-Party Hadoop Distros</a></li>
<li class="divider"></li>
<li><a href="building-spark.html">Building Spark</a></li>
<li><a href="https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark">Contributing to Spark</a></li>
<li><a href="https://cwiki.apache.org/confluence/display/SPARK/Supplemental+Spark+Projects">Supplemental Projects</a></li>
</ul>
</li>
</ul>
<!--<p class="navbar-text pull-right"><span class="version-text">v1.5.1</span></p>-->
</div>
</div>
</div>
<div class="container" id="content">
<h1 class="title">Spark Security</h1>
<p>Spark currently supports authentication via a shared secret. Authentication can be configured to be on via the <code>spark.authenticate</code> configuration parameter. This parameter controls whether the Spark communication protocols do authentication using the shared secret. This authentication is a basic handshake to make sure both sides have the same shared secret and are allowed to communicate. If the shared secret is not identical they will not be allowed to communicate. The shared secret is created as follows:</p>
<ul>
<li>For Spark on <a href="running-on-yarn.html">YARN</a> deployments, configuring <code>spark.authenticate</code> to <code>true</code> will automatically handle generating and distributing the shared secret. Each application will use a unique shared secret.</li>
<li>For other types of Spark deployments, the Spark parameter <code>spark.authenticate.secret</code> should be configured on each of the nodes. This secret will be used by all the Master/Workers and applications.</li>
</ul>
<h2 id="web-ui">Web UI</h2>
<p>The Spark UI can also be secured by using <a href="http://docs.oracle.com/javaee/6/api/javax/servlet/Filter.html">javax servlet filters</a> via the <code>spark.ui.filters</code> setting. A user may want to secure the UI if it has data that other users should not be allowed to see. The javax servlet filter specified by the user can authenticate the user and then once the user is logged in, Spark can compare that user versus the view ACLs to make sure they are authorized to view the UI. The configs <code>spark.acls.enable</code> and <code>spark.ui.view.acls</code> control the behavior of the ACLs. Note that the user who started the application always has view access to the UI. On YARN, the Spark UI uses the standard YARN web application proxy mechanism and will authenticate via any installed Hadoop filters.</p>
<p>Spark also supports modify ACLs to control who has access to modify a running Spark application. This includes things like killing the application or a task. This is controlled by the configs <code>spark.acls.enable</code> and <code>spark.modify.acls</code>. Note that if you are authenticating the web UI, in order to use the kill button on the web UI it might be necessary to add the users in the modify acls to the view acls also. On YARN, the modify acls are passed in and control who has modify access via YARN interfaces.</p>
<p>Spark allows for a set of administrators to be specified in the acls who always have view and modify permissions to all the applications. is controlled by the config <code>spark.admin.acls</code>. This is useful on a shared cluster where you might have administrators or support staff who help users debug applications.</p>
<h2 id="event-logging">Event Logging</h2>
<p>If your applications are using event logging, the directory where the event logs go (<code>spark.eventLog.dir</code>) should be manually created and have the proper permissions set on it. If you want those log files secured, the permissions should be set to <code>drwxrwxrwxt</code> for that directory. The owner of the directory should be the super user who is running the history server and the group permissions should be restricted to super user group. This will allow all users to write to the directory but will prevent unprivileged users from removing or renaming a file unless they own the file or directory. The event log files will be created by Spark with permissions such that only the user and group have read and write access.</p>
<h2 id="encryption">Encryption</h2>
<p>Spark supports SSL for Akka and HTTP (for broadcast and file server) protocols. SASL encryption is
supported for the block transfer service. Encryption is not yet supported for the WebUI.</p>
<p>Encryption is not yet supported for data stored by Spark in temporary local storage, such as shuffle
files, cached data, and other application files. If encrypting this data is desired, a workaround is
to configure your cluster manager to store application data on encrypted disks.</p>
<h3 id="ssl-configuration">SSL Configuration</h3>
<p>Configuration for SSL is organized hierarchically. The user can configure the default SSL settings which will be used for all the supported communication protocols unless they are overwritten by protocol-specific settings. This way the user can easily provide the common settings for all the protocols without disabling the ability to configure each one individually. The common SSL settings are at <code>spark.ssl</code> namespace in Spark configuration, while Akka SSL configuration is at <code>spark.ssl.akka</code> and HTTP for broadcast and file server SSL configuration is at <code>spark.ssl.fs</code>. The full breakdown can be found on the <a href="configuration.html">configuration page</a>.</p>
<p>SSL must be configured on each node and configured for each component involved in communication using the particular protocol.</p>
<h3 id="yarn-mode">YARN mode</h3>
<p>The key-store can be prepared on the client side and then distributed and used by the executors as the part of the application. It is possible because the user is able to deploy files before the application is started in YARN by using <code>spark.yarn.dist.files</code> or <code>spark.yarn.dist.archives</code> configuration settings. The responsibility for encryption of transferring these files is on YARN side and has nothing to do with Spark.</p>
<p>For long-running apps like Spark Streaming apps to be able to write to HDFS, it is possible to pass a principal and keytab to <code>spark-submit</code> via the <code>--principal</code> and <code>--keytab</code> parameters respectively. The keytab passed in will be copied over to the machine running the Application Master via the Hadoop Distributed Cache (securely - if YARN is configured with SSL and HDFS encryption is enabled). The Kerberos login will be periodically renewed using this principal and keytab and the delegation tokens required for HDFS will be generated periodically so the application can continue writing to HDFS.</p>
<h3 id="standalone-mode">Standalone mode</h3>
<p>The user needs to provide key-stores and configuration options for master and workers. They have to be set by attaching appropriate Java system properties in <code>SPARK_MASTER_OPTS</code> and in <code>SPARK_WORKER_OPTS</code> environment variables, or just in <code>SPARK_DAEMON_JAVA_OPTS</code>. In this mode, the user may allow the executors to use the SSL settings inherited from the worker which spawned that executor. It can be accomplished by setting <code>spark.ssl.useNodeLocalConf</code> to <code>true</code>. If that parameter is set, the settings provided by user on the client side, are not used by the executors.</p>
<h3 id="preparing-the-key-stores">Preparing the key-stores</h3>
<p>Key-stores can be generated by <code>keytool</code> program. The reference documentation for this tool is
<a href="https://docs.oracle.com/javase/7/docs/technotes/tools/solaris/keytool.html">here</a>. The most basic
steps to configure the key-stores and the trust-store for the standalone deployment mode is as
follows:
* Generate a keys pair for each node
* Export the public key of the key pair to a file on each node
* Import all exported public keys into a single trust-store
* Distribute the trust-store over the nodes</p>
<h3 id="configuring-sasl-encryption">Configuring SASL Encryption</h3>
<p>SASL encryption is currently supported for the block transfer service when authentication
(<code>spark.authenticate</code>) is enabled. To enable SASL encryption for an application, set
<code>spark.authenticate.enableSaslEncryption</code> to <code>true</code> in the application’s configuration.</p>
<p>When using an external shuffle service, it’s possible to disable unencrypted connections by setting
<code>spark.network.sasl.serverAlwaysEncrypt</code> to <code>true</code> in the shuffle service’s configuration. If that
option is enabled, applications that are not set up to use SASL encryption will fail to connect to
the shuffle service.</p>
<h2 id="configuring-ports-for-network-security">Configuring Ports for Network Security</h2>
<p>Spark makes heavy use of the network, and some environments have strict requirements for using tight
firewall settings. Below are the primary ports that Spark uses for its communication and how to
configure those ports.</p>
<h3 id="standalone-mode-only">Standalone mode only</h3>
<table class="table">
<tr>
<th>From</th><th>To</th><th>Default Port</th><th>Purpose</th><th>Configuration
Setting</th><th>Notes</th>
</tr>
<tr>
<td>Browser</td>
<td>Standalone Master</td>
<td>8080</td>
<td>Web UI</td>
<td><code>spark.master.ui.port /<br /> SPARK_MASTER_WEBUI_PORT</code></td>
<td>Jetty-based. Standalone mode only.</td>
</tr>
<tr>
<td>Browser</td>
<td>Standalone Worker</td>
<td>8081</td>
<td>Web UI</td>
<td><code>spark.worker.ui.port /<br /> SPARK_WORKER_WEBUI_PORT</code></td>
<td>Jetty-based. Standalone mode only.</td>
</tr>
<tr>
<td>Driver /<br /> Standalone Worker</td>
<td>Standalone Master</td>
<td>7077</td>
<td>Submit job to cluster /<br /> Join cluster</td>
<td><code>SPARK_MASTER_PORT</code></td>
<td>Akka-based. Set to "0" to choose a port randomly. Standalone mode only.</td>
</tr>
<tr>
<td>Standalone Master</td>
<td>Standalone Worker</td>
<td>(random)</td>
<td>Schedule executors</td>
<td><code>SPARK_WORKER_PORT</code></td>
<td>Akka-based. Set to "0" to choose a port randomly. Standalone mode only.</td>
</tr>
</table>
<h3 id="all-cluster-managers">All cluster managers</h3>
<table class="table">
<tr>
<th>From</th><th>To</th><th>Default Port</th><th>Purpose</th><th>Configuration
Setting</th><th>Notes</th>
</tr>
<tr>
<td>Browser</td>
<td>Application</td>
<td>4040</td>
<td>Web UI</td>
<td><code>spark.ui.port</code></td>
<td>Jetty-based</td>
</tr>
<tr>
<td>Browser</td>
<td>History Server</td>
<td>18080</td>
<td>Web UI</td>
<td><code>spark.history.ui.port</code></td>
<td>Jetty-based</td>
</tr>
<tr>
<td>Executor /<br /> Standalone Master</td>
<td>Driver</td>
<td>(random)</td>
<td>Connect to application /<br /> Notify executor state changes</td>
<td><code>spark.driver.port</code></td>
<td>Akka-based. Set to "0" to choose a port randomly.</td>
</tr>
<tr>
<td>Driver</td>
<td>Executor</td>
<td>(random)</td>
<td>Schedule tasks</td>
<td><code>spark.executor.port</code></td>
<td>Akka-based. Set to "0" to choose a port randomly.</td>
</tr>
<tr>
<td>Executor</td>
<td>Driver</td>
<td>(random)</td>
<td>File server for files and jars</td>
<td><code>spark.fileserver.port</code></td>
<td>Jetty-based</td>
</tr>
<tr>
<td>Executor</td>
<td>Driver</td>
<td>(random)</td>
<td>HTTP Broadcast</td>
<td><code>spark.broadcast.port</code></td>
<td>Jetty-based. Not used by TorrentBroadcast, which sends data through the block manager
instead.</td>
</tr>
<tr>
<td>Executor</td>
<td>Driver</td>
<td>(random)</td>
<td>Class file server</td>
<td><code>spark.replClassServer.port</code></td>
<td>Jetty-based. Only used in Spark shells.</td>
</tr>
<tr>
<td>Executor / Driver</td>
<td>Executor / Driver</td>
<td>(random)</td>
<td>Block Manager port</td>
<td><code>spark.blockManager.port</code></td>
<td>Raw socket via ServerSocketChannel</td>
</tr>
</table>
<p>See the <a href="configuration.html">configuration page</a> for more details on the security configuration
parameters, and <a href="https://github.com/apache/spark/tree/master/core/src/main/scala/org/apache/spark/SecurityManager.scala">
<code>org.apache.spark.SecurityManager</code></a> for implementation details about security.</p>
</div> <!-- /container -->
<script src="js/vendor/jquery-1.8.0.min.js"></script>
<script src="js/vendor/bootstrap.min.js"></script>
<script src="js/vendor/anchor.min.js"></script>
<script src="js/main.js"></script>
<!-- MathJax Section -->
<script type="text/x-mathjax-config">
MathJax.Hub.Config({
TeX: { equationNumbers: { autoNumber: "AMS" } }
});
</script>
<script>
// Note that we load MathJax this way to work with local file (file://), HTTP and HTTPS.
// We could use "//cdn.mathjax...", but that won't support "file://".
(function(d, script) {
script = d.createElement('script');
script.type = 'text/javascript';
script.async = true;
script.onload = function(){
MathJax.Hub.Config({
tex2jax: {
inlineMath: [ ["$", "$"], ["\\\\(","\\\\)"] ],
displayMath: [ ["$$","$$"], ["\\[", "\\]"] ],
processEscapes: true,
skipTags: ['script', 'noscript', 'style', 'textarea', 'pre']
}
});
};
script.src = ('https:' == document.location.protocol ? 'https://' : 'http://') +
'cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML';
d.getElementsByTagName('head')[0].appendChild(script);
}(document));
</script>
</body>
</html>