diff options
| author | Davies Liu <davies@databricks.com> | 2015-08-13 21:12:59 -0700 |
|---|---|---|
| committer | Davies Liu <davies.liu@gmail.com> | 2015-08-13 21:12:59 -0700 |
| commit | bd35385d53a6b039e0241e3e73092b8b0a8e455a (patch) | |
| tree | db32f1433e3559368ae57089f818f73df88414a6 /core | |
| parent | 8187b3ae477e2b2987ae9acc5368d57b1d5653b2 (diff) | |
| download | spark-bd35385d53a6b039e0241e3e73092b8b0a8e455a.tar.gz spark-bd35385d53a6b039e0241e3e73092b8b0a8e455a.tar.bz2 spark-bd35385d53a6b039e0241e3e73092b8b0a8e455a.zip | |
[SPARK-9945] [SQL] pageSize should be calculated from executor.memory
Currently, pageSize of TungstenSort is calculated from driver.memory, it should use executor.memory instead.
Also, in the worst case, the safeFactor could be 4 (because of rounding), increase it to 16.
cc rxin
Author: Davies Liu <davies@databricks.com>
Closes #8175 from davies/page_size.
Diffstat (limited to 'core')
| -rw-r--r-- | core/src/main/scala/org/apache/spark/shuffle/ShuffleMemoryManager.scala | 4 |
1 files changed, 3 insertions, 1 deletions
diff --git a/core/src/main/scala/org/apache/spark/shuffle/ShuffleMemoryManager.scala b/core/src/main/scala/org/apache/spark/shuffle/ShuffleMemoryManager.scala index 8c3a72644c..a0d8abc2ee 100644 --- a/core/src/main/scala/org/apache/spark/shuffle/ShuffleMemoryManager.scala +++ b/core/src/main/scala/org/apache/spark/shuffle/ShuffleMemoryManager.scala @@ -175,7 +175,9 @@ private[spark] object ShuffleMemoryManager { val minPageSize = 1L * 1024 * 1024 // 1MB val maxPageSize = 64L * minPageSize // 64MB val cores = if (numCores > 0) numCores else Runtime.getRuntime.availableProcessors() - val safetyFactor = 8 + // Because of rounding to next power of 2, we may have safetyFactor as 8 in worst case + val safetyFactor = 16 + // TODO(davies): don't round to next power of 2 val size = ByteArrayMethods.nextPowerOf2(maxMemory / cores / safetyFactor) val default = math.min(maxPageSize, math.max(minPageSize, size)) conf.getSizeAsBytes("spark.buffer.pageSize", default) |