diff options
| author | Josh Rosen <joshrosen@databricks.com> | 2015-12-10 15:29:04 -0800 |
|---|---|---|
| committer | Andrew Or <andrew@databricks.com> | 2015-12-10 15:29:04 -0800 |
| commit | 23a9e62bad9669e9ff5dc4bd714f58d12f9be0b5 (patch) | |
| tree | 67ab5872819bf8a25ec2de0c0d3af5dceba8d877 /docs | |
| parent | 6a6c1fc5c807ba4e8aba3e260537aa527ff5d46a (diff) | |
| download | spark-23a9e62bad9669e9ff5dc4bd714f58d12f9be0b5.tar.gz spark-23a9e62bad9669e9ff5dc4bd714f58d12f9be0b5.tar.bz2 spark-23a9e62bad9669e9ff5dc4bd714f58d12f9be0b5.zip | |
[SPARK-12251] Document and improve off-heap memory configurations
This patch adds documentation for Spark configurations that affect off-heap memory and makes some naming and validation improvements for those configs.
- Change `spark.memory.offHeapSize` to `spark.memory.offHeap.size`. This is fine because this configuration has not shipped in any Spark release yet (it's new in Spark 1.6).
- Deprecated `spark.unsafe.offHeap` in favor of a new `spark.memory.offHeap.enabled` configuration. The motivation behind this change is to gather all memory-related configurations under the same prefix.
- Add a check which prevents users from setting `spark.memory.offHeap.enabled=true` when `spark.memory.offHeap.size == 0`. After SPARK-11389 (#9344), which was committed in Spark 1.6, Spark enforces a hard limit on the amount of off-heap memory that it will allocate to tasks. As a result, enabling off-heap execution memory without setting `spark.memory.offHeap.size` will lead to immediate OOMs. The new configuration validation makes this scenario easier to diagnose, helping to avoid user confusion.
- Document these configurations on the configuration page.
Author: Josh Rosen <joshrosen@databricks.com>
Closes #10237 from JoshRosen/SPARK-12251.
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/configuration.md | 16 |
1 files changed, 16 insertions, 0 deletions
diff --git a/docs/configuration.md b/docs/configuration.md index 873a2d0b30..55cf4b2dac 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -739,6 +739,22 @@ Apart from these, the following properties are also available, and may be useful </td> </tr> <tr> + <td><code>spark.memory.offHeap.enabled</code></td> + <td>true</td> + <td> + If true, Spark will attempt to use off-heap memory for certain operations. If off-heap memory use is enabled, then <code>spark.memory.offHeap.size</code> must be positive. + </td> +</tr> +<tr> + <td><code>spark.memory.offHeap.size</code></td> + <td>0</td> + <td> + The absolute amount of memory which can be used for off-heap allocation. + This setting has no impact on heap memory usage, so if your executors' total memory consumption must fit within some hard limit then be sure to shrink your JVM heap size accordingly. + This must be set to a positive value when <code>spark.memory.offHeap.enabled=true</code>. + </td> +</tr> +<tr> <td><code>spark.memory.useLegacyMode</code></td> <td>false</td> <td> |