diff options
| author | Patrick Wendell <pwendell@gmail.com> | 2014-01-12 21:31:43 -0800 |
|---|---|---|
| committer | Patrick Wendell <pwendell@gmail.com> | 2014-01-12 21:31:43 -0800 |
| commit | 0b96d85c2063bd2864b5753496551c6cf2f9047a (patch) | |
| tree | b084622810d44975d723c4387b8250ebfa5f82d5 /docs | |
| parent | 0ab505a29e21b5a03928e0bbd3950f6f8e08ae32 (diff) | |
| parent | 2802cc80bcf267fd19a4cb43da505b82af675e08 (diff) | |
| download | spark-0b96d85c2063bd2864b5753496551c6cf2f9047a.tar.gz spark-0b96d85c2063bd2864b5753496551c6cf2f9047a.tar.bz2 spark-0b96d85c2063bd2864b5753496551c6cf2f9047a.zip | |
Merge pull request #399 from pwendell/consolidate-off
Disable shuffle file consolidation by default
After running various performance tests for the 0.9 release, this still seems to have performance issues even on XFS. So let's keep this off-by-default for 0.9 and users can experiment with it depending on their disk configurations.
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/configuration.md | 2 |
1 files changed, 1 insertions, 1 deletions
diff --git a/docs/configuration.md b/docs/configuration.md index ad75e06fc7..40a57c4bc6 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -382,7 +382,7 @@ Apart from these, the following properties are also available, and may be useful <tr> <td>spark.shuffle.consolidateFiles</td> - <td>true</td> + <td>false</td> <td> If set to "true", consolidates intermediate files created during a shuffle. Creating fewer files can improve filesystem performance for shuffles with large numbers of reduce tasks. It is recommended to set this to "true" when using ext4 or xfs filesystems. On ext3, this option might degrade performance on machines with many (>8) cores due to filesystem limitations. </td> |