diff options
| author | Michael Armbrust <michael@databricks.com> | 2014-08-18 13:17:10 -0700 |
|---|---|---|
| committer | Michael Armbrust <michael@databricks.com> | 2014-08-18 13:17:27 -0700 |
| commit | cc4015d2fa3785b92e6ab079b3abcf17627f7c56 (patch) | |
| tree | 902d71e748e973b50b94610f383d2bc858a43b7c /python | |
| parent | 2ae2857986e94d5a8bd5f4660eabe5689463bd21 (diff) | |
| download | spark-cc4015d2fa3785b92e6ab079b3abcf17627f7c56.tar.gz spark-cc4015d2fa3785b92e6ab079b3abcf17627f7c56.tar.bz2 spark-cc4015d2fa3785b92e6ab079b3abcf17627f7c56.zip | |
[SPARK-2406][SQL] Initial support for using ParquetTableScan to read HiveMetaStore tables.
This PR adds an experimental flag `spark.sql.hive.convertMetastoreParquet` that when true causes the planner to detects tables that use Hive's Parquet SerDe and instead plans them using Spark SQL's native `ParquetTableScan`.
Author: Michael Armbrust <michael@databricks.com>
Author: Yin Huai <huai@cse.ohio-state.edu>
Closes #1819 from marmbrus/parquetMetastore and squashes the following commits:
1620079 [Michael Armbrust] Revert "remove hive parquet bundle"
cc30430 [Michael Armbrust] Merge remote-tracking branch 'origin/master' into parquetMetastore
4f3d54f [Michael Armbrust] fix style
41ebc5f [Michael Armbrust] remove hive parquet bundle
a43e0da [Michael Armbrust] Merge remote-tracking branch 'origin/master' into parquetMetastore
4c4dc19 [Michael Armbrust] Fix bug with tree splicing.
ebb267e [Michael Armbrust] include parquet hive to tests pass (Remove this later).
c0d9b72 [Michael Armbrust] Avoid creating a HadoopRDD per partition. Add dirty hacks to retrieve partition values from the InputSplit.
8cdc93c [Michael Armbrust] Merge pull request #8 from yhuai/parquetMetastore
a0baec7 [Yin Huai] Partitioning columns can be resolved.
1161338 [Michael Armbrust] Add a test to make sure conversion is actually happening
212d5cd [Michael Armbrust] Initial support for using ParquetTableScan to read HiveMetaStore tables.
(cherry picked from commit 3abd0c1cda09bb575adc99847a619bc84af37fd0)
Signed-off-by: Michael Armbrust <michael@databricks.com>
Diffstat (limited to 'python')
0 files changed, 0 insertions, 0 deletions