diff options
| author | Davies Liu <davies@databricks.com> | 2016-03-12 00:48:36 -0800 |
|---|---|---|
| committer | Davies Liu <davies.liu@gmail.com> | 2016-03-12 00:48:36 -0800 |
| commit | ba8c86d06f5968c1af4db8dd9a458005bc5f214c (patch) | |
| tree | fa6a7479cef0ba8c2f6b4574b0bbd180502bed85 /python | |
| parent | 2ef4c5963bff3574fe17e669d703b25ddd064e5d (diff) | |
| download | spark-ba8c86d06f5968c1af4db8dd9a458005bc5f214c.tar.gz spark-ba8c86d06f5968c1af4db8dd9a458005bc5f214c.tar.bz2 spark-ba8c86d06f5968c1af4db8dd9a458005bc5f214c.zip | |
[SPARK-13671] [SPARK-13311] [SQL] Use different physical plans for RDD and data sources
## What changes were proposed in this pull request?
This PR split the PhysicalRDD into two classes, PhysicalRDD and PhysicalScan. PhysicalRDD is used for DataFrames that is created from existing RDD. PhysicalScan is used for DataFrame that is created from data sources. This enable use to apply different optimization on both of them.
Also fix the problem for sameResult() on two DataSourceScan.
Also fix the equality check to toString for `In`. It's better to use Seq there, but we can't break this public API (sad).
## How was this patch tested?
Existing tests. Manually tested with TPCDS query Q59 and Q64, all those duplicated exchanges can be re-used now, also saw there are 40+% performance improvement (saving half of the scan).
Author: Davies Liu <davies@databricks.com>
Closes #11514 from davies/existing_rdd.
Diffstat (limited to 'python')
| -rw-r--r-- | python/pyspark/sql/dataframe.py | 3 |
1 files changed, 1 insertions, 2 deletions
diff --git a/python/pyspark/sql/dataframe.py b/python/pyspark/sql/dataframe.py index 99d665fafe..7008e8fadf 100644 --- a/python/pyspark/sql/dataframe.py +++ b/python/pyspark/sql/dataframe.py @@ -173,8 +173,7 @@ class DataFrame(object): >>> df.explain() == Physical Plan == - WholeStageCodegen - : +- Scan ExistingRDD[age#0,name#1] + Scan ExistingRDD[age#0,name#1] >>> df.explain(True) == Parsed Logical Plan == |