From 460f655cc60b1a1759b6f85ae8860595fc7586b4 Mon Sep 17 00:00:00 2001 From: Patrick Wendell Date: Thu, 9 Jan 2014 22:42:50 -0800 Subject: Enable shuffle consolidation by default. Bump this to being enabled for 0.9.0. --- docs/configuration.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) (limited to 'docs') diff --git a/docs/configuration.md b/docs/configuration.md index 6717757781..b1a0e19167 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -371,7 +371,7 @@ Apart from these, the following properties are also available, and may be useful spark.shuffle.consolidateFiles - false + true If set to "true", consolidates intermediate files created during a shuffle. Creating fewer files can improve filesystem performance for shuffles with large numbers of reduce tasks. It is recommended to set this to "true" when using ext4 or xfs filesystems. On ext3, this option might degrade performance on machines with many (>8) cores due to filesystem limitations. -- cgit v1.2.3