diff options
| author | Reynold Xin <rxin@databricks.com> | 2015-08-06 23:18:29 -0700 |
|---|---|---|
| committer | Reynold Xin <rxin@databricks.com> | 2015-08-06 23:18:29 -0700 |
| commit | 4309262ec9146d7158ee9957a128bb152289d557 (patch) | |
| tree | 47f6dabaaea5c5fcff48dcbdb114aede873ae320 /R | |
| parent | 7aaed1b114751a24835204b8c588533d5c5ffaf0 (diff) | |
| download | spark-4309262ec9146d7158ee9957a128bb152289d557.tar.gz spark-4309262ec9146d7158ee9957a128bb152289d557.tar.bz2 spark-4309262ec9146d7158ee9957a128bb152289d557.zip | |
[SPARK-9700] Pick default page size more intelligently.
Previously, we use 64MB as the default page size, which was way too big for a lot of Spark applications (especially for single node).
This patch changes it so that the default page size, if unset by the user, is determined by the number of cores available and the total execution memory available.
Author: Reynold Xin <rxin@databricks.com>
Closes #8012 from rxin/pagesize and squashes the following commits:
16f4756 [Reynold Xin] Fixed failing test.
5afd570 [Reynold Xin] private...
0d5fb98 [Reynold Xin] Update default value.
674a6cd [Reynold Xin] Address review feedback.
dc00e05 [Reynold Xin] Merge with master.
73ebdb6 [Reynold Xin] [SPARK-9700] Pick default page size more intelligently.
Diffstat (limited to 'R')
| -rwxr-xr-x | R/run-tests.sh | 2 |
1 files changed, 1 insertions, 1 deletions
diff --git a/R/run-tests.sh b/R/run-tests.sh index 18a1e13bdc..e82ad0ba2c 100755 --- a/R/run-tests.sh +++ b/R/run-tests.sh @@ -23,7 +23,7 @@ FAILED=0 LOGFILE=$FWDIR/unit-tests.out rm -f $LOGFILE -SPARK_TESTING=1 $FWDIR/../bin/sparkR --conf spark.buffer.pageSize=4m --driver-java-options "-Dlog4j.configuration=file:$FWDIR/log4j.properties" $FWDIR/pkg/tests/run-all.R 2>&1 | tee -a $LOGFILE +SPARK_TESTING=1 $FWDIR/../bin/sparkR --driver-java-options "-Dlog4j.configuration=file:$FWDIR/log4j.properties" $FWDIR/pkg/tests/run-all.R 2>&1 | tee -a $LOGFILE FAILED=$((PIPESTATUS[0]||$FAILED)) if [[ $FAILED != 0 ]]; then |