diff options
| author | Davies Liu <davies.liu@gmail.com> | 2014-08-26 16:57:40 -0700 |
|---|---|---|
| committer | Matei Zaharia <matei@databricks.com> | 2014-08-26 16:57:40 -0700 |
| commit | f1e71d4c3ba678fc108effb05cf2d6101dadc0ce (patch) | |
| tree | ef5c761a9bf3a75c59b03148985a4a83e64a2c16 /tox.ini | |
| parent | c4787a3690a9ed3b8b2c6c294fc4a6915436b6f7 (diff) | |
| download | spark-f1e71d4c3ba678fc108effb05cf2d6101dadc0ce.tar.gz spark-f1e71d4c3ba678fc108effb05cf2d6101dadc0ce.tar.bz2 spark-f1e71d4c3ba678fc108effb05cf2d6101dadc0ce.zip | |
[SPARK-3073] [PySpark] use external sort in sortBy() and sortByKey()
Using external sort to support sort large datasets in reduce stage.
Author: Davies Liu <davies.liu@gmail.com>
Closes #1978 from davies/sort and squashes the following commits:
bbcd9ba [Davies Liu] check spilled bytes in tests
b125d2f [Davies Liu] add test for external sort in rdd
eae0176 [Davies Liu] choose different disks from different processes and instances
1f075ed [Davies Liu] Merge branch 'master' into sort
eb53ca6 [Davies Liu] Merge branch 'master' into sort
644abaf [Davies Liu] add license in LICENSE
19f7873 [Davies Liu] improve tests
55602ee [Davies Liu] use external sort in sortBy() and sortByKey()
Diffstat (limited to 'tox.ini')
| -rw-r--r-- | tox.ini | 2 |
1 files changed, 1 insertions, 1 deletions
@@ -15,4 +15,4 @@ [pep8] max-line-length=100 -exclude=cloudpickle.py +exclude=cloudpickle.py,heapq3.py |