diff options
| author | Josh Rosen <joshrosen@databricks.com> | 2015-09-19 21:40:21 -0700 |
|---|---|---|
| committer | Reynold Xin <rxin@databricks.com> | 2015-09-19 21:40:21 -0700 |
| commit | 2117eea71ece825fbc3797c8b38184ae221f5223 (patch) | |
| tree | 06481ef1968367118e89779335e24245f57f2017 /docs/sql-programming-guide.md | |
| parent | e789000b88a6bd840f821c53f42c08b97dc02496 (diff) | |
| download | spark-2117eea71ece825fbc3797c8b38184ae221f5223.tar.gz spark-2117eea71ece825fbc3797c8b38184ae221f5223.tar.bz2 spark-2117eea71ece825fbc3797c8b38184ae221f5223.zip | |
[SPARK-10710] Remove ability to disable spilling in core and SQL
It does not make much sense to set `spark.shuffle.spill` or `spark.sql.planner.externalSort` to false: I believe that these configurations were initially added as "escape hatches" to guard against bugs in the external operators, but these operators are now mature and well-tested. In addition, these configurations are not handled in a consistent way anymore: SQL's Tungsten codepath ignores these configurations and will continue to use spilling operators. Similarly, Spark Core's `tungsten-sort` shuffle manager does not respect `spark.shuffle.spill=false`.
This pull request removes these configurations, adds warnings at the appropriate places, and deletes a large amount of code which was only used in code paths that did not support spilling.
Author: Josh Rosen <joshrosen@databricks.com>
Closes #8831 from JoshRosen/remove-ability-to-disable-spilling.
Diffstat (limited to 'docs/sql-programming-guide.md')
| -rw-r--r-- | docs/sql-programming-guide.md | 7 |
1 files changed, 0 insertions, 7 deletions
diff --git a/docs/sql-programming-guide.md b/docs/sql-programming-guide.md index 82d4243cc6..7ae9244c27 100644 --- a/docs/sql-programming-guide.md +++ b/docs/sql-programming-guide.md @@ -1936,13 +1936,6 @@ that these options will be deprecated in future release as more optimizations ar Configures the number of partitions to use when shuffling data for joins or aggregations. </td> </tr> - <tr> - <td><code>spark.sql.planner.externalSort</code></td> - <td>true</td> - <td> - When true, performs sorts spilling to disk as needed otherwise sort each partition in memory. - </td> - </tr> </table> # Distributed SQL Engine |