diff options
| author | Davies Liu <davies@databricks.com> | 2016-02-23 15:00:10 -0800 |
|---|---|---|
| committer | Davies Liu <davies.liu@gmail.com> | 2016-02-23 15:00:10 -0800 |
| commit | 9cdd867da978629ea2f61f94e3c346fa0bfecf0e (patch) | |
| tree | b8aa4cceb32a21911a7e8ed40a9c50859df25266 /core | |
| parent | c481bdf512f09060c9b9f341a5ce9fce00427d08 (diff) | |
| download | spark-9cdd867da978629ea2f61f94e3c346fa0bfecf0e.tar.gz spark-9cdd867da978629ea2f61f94e3c346fa0bfecf0e.tar.bz2 spark-9cdd867da978629ea2f61f94e3c346fa0bfecf0e.zip | |
[SPARK-13373] [SQL] generate sort merge join
## What changes were proposed in this pull request?
Generates code for SortMergeJoin.
## How was the this patch tested?
Unit tests and manually tested with TPCDS Q72, which showed 70% performance improvements (from 42s to 25s), but micro benchmark only show minor improvements, it may depends the distribution of data and number of columns.
Author: Davies Liu <davies@databricks.com>
Closes #11248 from davies/gen_smj.
Diffstat (limited to 'core')
| -rw-r--r-- | core/src/main/scala/org/apache/spark/storage/DiskBlockObjectWriter.scala | 1 |
1 files changed, 1 insertions, 0 deletions
diff --git a/core/src/main/scala/org/apache/spark/storage/DiskBlockObjectWriter.scala b/core/src/main/scala/org/apache/spark/storage/DiskBlockObjectWriter.scala index c34d49c0d9..9cc4084497 100644 --- a/core/src/main/scala/org/apache/spark/storage/DiskBlockObjectWriter.scala +++ b/core/src/main/scala/org/apache/spark/storage/DiskBlockObjectWriter.scala @@ -203,6 +203,7 @@ private[spark] class DiskBlockObjectWriter( numRecordsWritten += 1 writeMetrics.incRecordsWritten(1) + // TODO: call updateBytesWritten() less frequently. if (numRecordsWritten % 32 == 0) { updateBytesWritten() } |