diff options
| author | Yin Huai <yhuai@databricks.com> | 2016-06-15 11:50:54 -0700 |
|---|---|---|
| committer | Reynold Xin <rxin@databricks.com> | 2016-06-15 11:50:54 -0700 |
| commit | e1585cc74853c497271eecdc943c0eabe1aeb4c1 (patch) | |
| tree | 0cb8c511b9dc26fecca4b5a833298e551562dc01 /sql/hive/src/main | |
| parent | 9a5071996b968148f6b9aba12e0d3fe888d9acd8 (diff) | |
| download | spark-e1585cc74853c497271eecdc943c0eabe1aeb4c1.tar.gz spark-e1585cc74853c497271eecdc943c0eabe1aeb4c1.tar.bz2 spark-e1585cc74853c497271eecdc943c0eabe1aeb4c1.zip | |
[SPARK-15959][SQL] Add the support of hive.metastore.warehouse.dir back
## What changes were proposed in this pull request?
This PR adds the support of conf `hive.metastore.warehouse.dir` back. With this patch, the way of setting the warehouse dir is described as follows:
* If `spark.sql.warehouse.dir` is set, `hive.metastore.warehouse.dir` will be automatically set to the value of `spark.sql.warehouse.dir`. The warehouse dir is effectively set to the value of `spark.sql.warehouse.dir`.
* If `spark.sql.warehouse.dir` is not set but `hive.metastore.warehouse.dir` is set, `spark.sql.warehouse.dir` will be automatically set to the value of `hive.metastore.warehouse.dir`. The warehouse dir is effectively set to the value of `hive.metastore.warehouse.dir`.
* If neither `spark.sql.warehouse.dir` nor `hive.metastore.warehouse.dir` is set, `hive.metastore.warehouse.dir` will be automatically set to the default value of `spark.sql.warehouse.dir`. The warehouse dir is effectively set to the default value of `spark.sql.warehouse.dir`.
## How was this patch tested?
`set hive.metastore.warehouse.dir` in `HiveSparkSubmitSuite`.
JIRA: https://issues.apache.org/jira/browse/SPARK-15959
Author: Yin Huai <yhuai@databricks.com>
Closes #13679 from yhuai/hiveWarehouseDir.
Diffstat (limited to 'sql/hive/src/main')
| -rw-r--r-- | sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveSharedState.scala | 13 |
1 files changed, 2 insertions, 11 deletions
diff --git a/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveSharedState.scala b/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveSharedState.scala index 78b1ecbbea..6b7a333f2d 100644 --- a/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveSharedState.scala +++ b/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveSharedState.scala @@ -18,9 +18,8 @@ package org.apache.spark.sql.hive import org.apache.spark.SparkContext -import org.apache.spark.internal.Logging import org.apache.spark.sql.hive.client.HiveClient -import org.apache.spark.sql.internal.{SharedState, SQLConf} +import org.apache.spark.sql.internal.SharedState /** @@ -28,18 +27,10 @@ import org.apache.spark.sql.internal.{SharedState, SQLConf} * [[org.apache.spark.sql.SparkSession]] backed by Hive. */ private[hive] class HiveSharedState(override val sparkContext: SparkContext) - extends SharedState(sparkContext) with Logging { + extends SharedState(sparkContext) { // TODO: just share the IsolatedClientLoader instead of the client instance itself - { - // Set the Hive metastore warehouse path to the one we use - val tempConf = new SQLConf - sparkContext.conf.getAll.foreach { case (k, v) => tempConf.setConfString(k, v) } - sparkContext.conf.set("hive.metastore.warehouse.dir", tempConf.warehousePath) - logInfo(s"Setting Hive metastore warehouse path to '${tempConf.warehousePath}'") - } - /** * A Hive client used to interact with the metastore. */ |