diff options
| author | Timothy Chen <tnachen@gmail.com> | 2016-08-10 10:11:03 +0100 |
|---|---|---|
| committer | Sean Owen <sowen@cloudera.com> | 2016-08-10 10:11:03 +0100 |
| commit | eca58755fbbc11937b335ad953a3caff89b818e6 (patch) | |
| tree | 13274396606247644ee9de7f68cc791cac6051f8 /docs | |
| parent | bfda53f63a31bf2e8b72ab9e85896a4bec1644e8 (diff) | |
| download | spark-eca58755fbbc11937b335ad953a3caff89b818e6.tar.gz spark-eca58755fbbc11937b335ad953a3caff89b818e6.tar.bz2 spark-eca58755fbbc11937b335ad953a3caff89b818e6.zip | |
[SPARK-16927][SPARK-16923] Override task properties at dispatcher.
## What changes were proposed in this pull request?
- enable setting default properties for all jobs submitted through the dispatcher [SPARK-16927]
- remove duplication of conf vars on cluster submitted jobs [SPARK-16923] (this is a small fix, so I'm including in the same PR)
## How was this patch tested?
mesos/spark integration test suite
manual testing
Author: Timothy Chen <tnachen@gmail.com>
Closes #14511 from mgummelt/override-props.
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/running-on-mesos.md | 11 |
1 files changed, 11 insertions, 0 deletions
diff --git a/docs/running-on-mesos.md b/docs/running-on-mesos.md index 613da68531..a6ce34c761 100644 --- a/docs/running-on-mesos.md +++ b/docs/running-on-mesos.md @@ -467,6 +467,17 @@ See the [configuration page](configuration.html) for information on Spark config Set the Spark Mesos dispatcher webui_url for interacting with the framework. If unset it will point to Spark's internal web UI. </td> + </tr> +<tr> + <td><code>spark.mesos.dispatcher.driverDefault.[PropertyName]</code></td> + <td><code>(none)</code></td> + <td> + Set default properties for drivers submitted through the + dispatcher. For example, + spark.mesos.dispatcher.driverProperty.spark.executor.memory=32g + results in the executors for all drivers submitted in cluster mode + to run in 32g containers. +</td> </tr> <tr> <td><code>spark.mesos.dispatcher.historyServer.url</code></td> |