diff options
author | Matei Zaharia <matei@eecs.berkeley.edu> | 2013-08-15 17:22:49 -0700 |
---|---|---|
committer | Matei Zaharia <matei@eecs.berkeley.edu> | 2013-08-18 19:51:07 -0700 |
commit | 2a4ed10210f9ee32f472e2465094d88561c0ff18 (patch) | |
tree | 5202b6408c8e37d15dc4dadaac5a852d0c22f20a /docs | |
parent | 222c8971285190761354456c2fe07f5c31edf330 (diff) | |
download | spark-2a4ed10210f9ee32f472e2465094d88561c0ff18.tar.gz spark-2a4ed10210f9ee32f472e2465094d88561c0ff18.tar.bz2 spark-2a4ed10210f9ee32f472e2465094d88561c0ff18.zip |
Address some review comments:
- When a resourceOffers() call has multiple offers, force the TaskSets
to consider them in increasing order of locality levels so that they
get a chance to launch stuff locally across all offers
- Simplify ClusterScheduler.prioritizeContainers
- Add docs on the new configuration options
Diffstat (limited to 'docs')
-rw-r--r-- | docs/configuration.md | 30 |
1 files changed, 28 insertions, 2 deletions
diff --git a/docs/configuration.md b/docs/configuration.md index 99624a44aa..dff08a06f5 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -243,8 +243,34 @@ Apart from these, the following properties are also available, and may be useful <td>3000</td> <td> Number of milliseconds to wait to launch a data-local task before giving up and launching it - in a non-data-local location. You should increase this if your tasks are long and you are seeing - poor data locality, but the default generally works well. + on a less-local node. The same wait will be used to step through multiple locality levels + (process-local, node-local, rack-local and then any). It is also possible to customize the + waiting time for each level by setting <code>spark.locality.wait.node</code>, etc. + You should increase this setting if your tasks are long and see poor locality, but the + default usually works well. + </td> +</tr> +<tr> + <td>spark.locality.wait.process</td> + <td>spark.locality.wait</td> + <td> + Customize the locality wait for process locality. This affects tasks that attempt to access + cached data in a particular executor process. + </td> +</tr> +<tr> + <td>spark.locality.wait.node</td> + <td>spark.locality.wait</td> + <td> + Customize the locality wait for node locality. For example, you can set this to 0 to skip + node locality and search immediately for rack locality (if your cluster has rack information). + </td> +</tr> +<tr> + <td>spark.locality.wait.rack</td> + <td>spark.locality.wait</td> + <td> + Customize the locality wait for rack locality. </td> </tr> <tr> |