diff options
author | Charles Allen <charles@allen-net.com> | 2016-11-01 13:14:17 +0000 |
---|---|---|
committer | Sean Owen <sowen@cloudera.com> | 2016-11-01 13:14:17 +0000 |
commit | e34b4e12673fb76c92f661d7c03527410857a0f8 (patch) | |
tree | b9accb74292f7f7f2e5b9dbc1c5201a9b103a9df /docs | |
parent | cb80edc26349e2e358d27fe2ae8e5d6959b77fab (diff) | |
download | spark-e34b4e12673fb76c92f661d7c03527410857a0f8.tar.gz spark-e34b4e12673fb76c92f661d7c03527410857a0f8.tar.bz2 spark-e34b4e12673fb76c92f661d7c03527410857a0f8.zip |
[SPARK-15994][MESOS] Allow enabling Mesos fetch cache in coarse executor backend
Mesos 0.23.0 introduces a Fetch Cache feature http://mesos.apache.org/documentation/latest/fetcher/ which allows caching of resources specified in command URIs.
This patch:
- Updates the Mesos shaded protobuf dependency to 0.23.0
- Allows setting `spark.mesos.fetcherCache.enable` to enable the fetch cache for all specified URIs. (URIs must be specified for the setting to have any affect)
- Updates documentation for Mesos configuration with the new setting.
This patch does NOT:
- Allow for per-URI caching configuration. The cache setting is global to ALL URIs for the command.
Author: Charles Allen <charles@allen-net.com>
Closes #13713 from drcrallen/SPARK15994.
Diffstat (limited to 'docs')
-rw-r--r-- | docs/running-on-mesos.md | 9 |
1 files changed, 7 insertions, 2 deletions
diff --git a/docs/running-on-mesos.md b/docs/running-on-mesos.md index 77b06fcf33..923d8dbebf 100644 --- a/docs/running-on-mesos.md +++ b/docs/running-on-mesos.md @@ -506,8 +506,13 @@ See the [configuration page](configuration.html) for information on Spark config since this configuration is just a upper limit and not a guaranteed amount. </td> </tr> - - +<tr> + <td><code>spark.mesos.fetcherCache.enable</code></td> + <td><code>false</code></td> + <td> + If set to `true`, all URIs (example: `spark.executor.uri`, `spark.mesos.uris`) will be cached by the [Mesos fetcher cache](http://mesos.apache.org/documentation/latest/fetcher/) + </td> +</tr> </table> # Troubleshooting and Debugging |