diff options
| author | Michael Armbrust <michael@databricks.com> | 2014-08-18 13:17:10 -0700 |
|---|---|---|
| committer | Michael Armbrust <michael@databricks.com> | 2014-08-18 13:17:10 -0700 |
| commit | 3abd0c1cda09bb575adc99847a619bc84af37fd0 (patch) | |
| tree | 7fbad5bd12995cbb047b77732f5212a05929cca1 /project | |
| parent | 9eb74c7d2cbe127dd4c32bf1a8318497b2fb55b6 (diff) | |
| download | spark-3abd0c1cda09bb575adc99847a619bc84af37fd0.tar.gz spark-3abd0c1cda09bb575adc99847a619bc84af37fd0.tar.bz2 spark-3abd0c1cda09bb575adc99847a619bc84af37fd0.zip | |
[SPARK-2406][SQL] Initial support for using ParquetTableScan to read HiveMetaStore tables.
This PR adds an experimental flag `spark.sql.hive.convertMetastoreParquet` that when true causes the planner to detects tables that use Hive's Parquet SerDe and instead plans them using Spark SQL's native `ParquetTableScan`.
Author: Michael Armbrust <michael@databricks.com>
Author: Yin Huai <huai@cse.ohio-state.edu>
Closes #1819 from marmbrus/parquetMetastore and squashes the following commits:
1620079 [Michael Armbrust] Revert "remove hive parquet bundle"
cc30430 [Michael Armbrust] Merge remote-tracking branch 'origin/master' into parquetMetastore
4f3d54f [Michael Armbrust] fix style
41ebc5f [Michael Armbrust] remove hive parquet bundle
a43e0da [Michael Armbrust] Merge remote-tracking branch 'origin/master' into parquetMetastore
4c4dc19 [Michael Armbrust] Fix bug with tree splicing.
ebb267e [Michael Armbrust] include parquet hive to tests pass (Remove this later).
c0d9b72 [Michael Armbrust] Avoid creating a HadoopRDD per partition. Add dirty hacks to retrieve partition values from the InputSplit.
8cdc93c [Michael Armbrust] Merge pull request #8 from yhuai/parquetMetastore
a0baec7 [Yin Huai] Partitioning columns can be resolved.
1161338 [Michael Armbrust] Add a test to make sure conversion is actually happening
212d5cd [Michael Armbrust] Initial support for using ParquetTableScan to read HiveMetaStore tables.
Diffstat (limited to 'project')
| -rw-r--r-- | project/SparkBuild.scala | 1 |
1 files changed, 0 insertions, 1 deletions
diff --git a/project/SparkBuild.scala b/project/SparkBuild.scala index 63a285b81a..49d52aefca 100644 --- a/project/SparkBuild.scala +++ b/project/SparkBuild.scala @@ -228,7 +228,6 @@ object SQL { object Hive { lazy val settings = Seq( - javaOptions += "-XX:MaxPermSize=1g", // Multiple queries rely on the TestHive singleton. See comments there for more details. parallelExecution in Test := false, |