diff options
| author | Davies Liu <davies.liu@gmail.com> | 2014-07-24 22:53:47 -0700 |
|---|---|---|
| committer | Matei Zaharia <matei@databricks.com> | 2014-07-24 22:53:47 -0700 |
| commit | 14174abd421318e71c16edd24224fd5094bdfed4 (patch) | |
| tree | 0af1cd49e09c80d8b7dd62d9de90ff03842db382 /python/run-tests | |
| parent | eff9714e1c88e39e28317358ca9ec87677f121dc (diff) | |
| download | spark-14174abd421318e71c16edd24224fd5094bdfed4.tar.gz spark-14174abd421318e71c16edd24224fd5094bdfed4.tar.bz2 spark-14174abd421318e71c16edd24224fd5094bdfed4.zip | |
[SPARK-2538] [PySpark] Hash based disk spilling aggregation
During aggregation in Python worker, if the memory usage is above spark.executor.memory, it will do disk spilling aggregation.
It will split the aggregation into multiple stage, in each stage, it will partition the aggregated data by hash and dump them into disks. After all the data are aggregated, it will merge all the stages together (partition by partition).
Author: Davies Liu <davies.liu@gmail.com>
Closes #1460 from davies/spill and squashes the following commits:
cad91bf [Davies Liu] call gc.collect() after data.clear() to release memory as much as possible.
37d71f7 [Davies Liu] balance the partitions
902f036 [Davies Liu] add shuffle.py into run-tests
dcf03a9 [Davies Liu] fix memory_info() of psutil
67e6eba [Davies Liu] comment for MAX_TOTAL_PARTITIONS
f6bd5d6 [Davies Liu] rollback next_limit() again, the performance difference is huge:
e74b785 [Davies Liu] fix code style and change next_limit to memory_limit
400be01 [Davies Liu] address all the comments
6178844 [Davies Liu] refactor and improve docs
fdd0a49 [Davies Liu] add long doc string for ExternalMerger
1a97ce4 [Davies Liu] limit used memory and size of objects in partitionBy()
e6cc7f9 [Davies Liu] Merge branch 'master' into spill
3652583 [Davies Liu] address comments
e78a0a0 [Davies Liu] fix style
24cec6a [Davies Liu] get local directory by SPARK_LOCAL_DIR
57ee7ef [Davies Liu] update docs
286aaff [Davies Liu] let spilled aggregation in Python configurable
e9a40f6 [Davies Liu] recursive merger
6edbd1f [Davies Liu] Hash based disk spilling aggregation
Diffstat (limited to 'python/run-tests')
| -rwxr-xr-x | python/run-tests | 1 |
1 files changed, 1 insertions, 0 deletions
diff --git a/python/run-tests b/python/run-tests index 9282aa47e8..29f755fc0d 100755 --- a/python/run-tests +++ b/python/run-tests @@ -61,6 +61,7 @@ run_test "pyspark/broadcast.py" run_test "pyspark/accumulators.py" run_test "pyspark/serializers.py" unset PYSPARK_DOC_TEST +run_test "pyspark/shuffle.py" run_test "pyspark/tests.py" run_test "pyspark/mllib/_common.py" run_test "pyspark/mllib/classification.py" |