diff options
author | Josh Rosen <joshrosen@databricks.com> | 2015-10-22 09:46:30 -0700 |
---|---|---|
committer | Josh Rosen <joshrosen@databricks.com> | 2015-10-22 09:46:30 -0700 |
commit | f6d06adf05afa9c5386dc2396c94e7a98730289f (patch) | |
tree | 9e3d8e4350e0a465124840eea91f6aa39c00b156 /docs | |
parent | 94e2064fa1b04c05c805d9175c7c78bf583db5c6 (diff) | |
download | spark-f6d06adf05afa9c5386dc2396c94e7a98730289f.tar.gz spark-f6d06adf05afa9c5386dc2396c94e7a98730289f.tar.bz2 spark-f6d06adf05afa9c5386dc2396c94e7a98730289f.zip |
[SPARK-10708] Consolidate sort shuffle implementations
There's a lot of duplication between SortShuffleManager and UnsafeShuffleManager. Given that these now provide the same set of functionality, now that UnsafeShuffleManager supports large records, I think that we should replace SortShuffleManager's serialized shuffle implementation with UnsafeShuffleManager's and should merge the two managers together.
Author: Josh Rosen <joshrosen@databricks.com>
Closes #8829 from JoshRosen/consolidate-sort-shuffle-implementations.
Diffstat (limited to 'docs')
-rw-r--r-- | docs/configuration.md | 7 |
1 files changed, 2 insertions, 5 deletions
diff --git a/docs/configuration.md b/docs/configuration.md index 46d92ceb76..be9c36bdfe 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -437,12 +437,9 @@ Apart from these, the following properties are also available, and may be useful <td><code>spark.shuffle.manager</code></td> <td>sort</td> <td> - Implementation to use for shuffling data. There are three implementations available: - <code>sort</code>, <code>hash</code> and the new (1.5+) <code>tungsten-sort</code>. + Implementation to use for shuffling data. There are two implementations available: + <code>sort</code> and <code>hash</code>. Sort-based shuffle is more memory-efficient and is the default option starting in 1.2. - Tungsten-sort is similar to the sort based shuffle, with a direct binary cache-friendly - implementation with a fall back to regular sort based shuffle if its requirements are not - met. </td> </tr> <tr> |