diff options
| author | Patrick Wendell <pwendell@gmail.com> | 2014-01-12 21:31:43 -0800 |
|---|---|---|
| committer | Patrick Wendell <pwendell@gmail.com> | 2014-01-12 21:31:43 -0800 |
| commit | 0b96d85c2063bd2864b5753496551c6cf2f9047a (patch) | |
| tree | b084622810d44975d723c4387b8250ebfa5f82d5 /core | |
| parent | 0ab505a29e21b5a03928e0bbd3950f6f8e08ae32 (diff) | |
| parent | 2802cc80bcf267fd19a4cb43da505b82af675e08 (diff) | |
| download | spark-0b96d85c2063bd2864b5753496551c6cf2f9047a.tar.gz spark-0b96d85c2063bd2864b5753496551c6cf2f9047a.tar.bz2 spark-0b96d85c2063bd2864b5753496551c6cf2f9047a.zip | |
Merge pull request #399 from pwendell/consolidate-off
Disable shuffle file consolidation by default
After running various performance tests for the 0.9 release, this still seems to have performance issues even on XFS. So let's keep this off-by-default for 0.9 and users can experiment with it depending on their disk configurations.
Diffstat (limited to 'core')
| -rw-r--r-- | core/src/main/scala/org/apache/spark/storage/ShuffleBlockManager.scala | 2 |
1 files changed, 1 insertions, 1 deletions
diff --git a/core/src/main/scala/org/apache/spark/storage/ShuffleBlockManager.scala b/core/src/main/scala/org/apache/spark/storage/ShuffleBlockManager.scala index 6e0ff143b7..e2b24298a5 100644 --- a/core/src/main/scala/org/apache/spark/storage/ShuffleBlockManager.scala +++ b/core/src/main/scala/org/apache/spark/storage/ShuffleBlockManager.scala @@ -64,7 +64,7 @@ class ShuffleBlockManager(blockManager: BlockManager) { // Turning off shuffle file consolidation causes all shuffle Blocks to get their own file. // TODO: Remove this once the shuffle file consolidation feature is stable. val consolidateShuffleFiles = - conf.getBoolean("spark.shuffle.consolidateFiles", true) + conf.getBoolean("spark.shuffle.consolidateFiles", false) private val bufferSize = conf.getInt("spark.shuffle.file.buffer.kb", 100) * 1024 |