diff options
author | Wenchen Fan <wenchen@databricks.com> | 2016-06-13 14:57:35 -0700 |
---|---|---|
committer | Yin Huai <yhuai@databricks.com> | 2016-06-13 14:57:35 -0700 |
commit | c4b1ad020962c42be804d3a1a55171d9b51b01e7 (patch) | |
tree | 567f143a02a0b657ddc29deb37c24701b7c59fcc /sql/core/src/test | |
parent | c654ae2140bc184adb407fd02072b653c5359ee5 (diff) | |
download | spark-c4b1ad020962c42be804d3a1a55171d9b51b01e7.tar.gz spark-c4b1ad020962c42be804d3a1a55171d9b51b01e7.tar.bz2 spark-c4b1ad020962c42be804d3a1a55171d9b51b01e7.zip |
[SPARK-15887][SQL] Bring back the hive-site.xml support for Spark 2.0
## What changes were proposed in this pull request?
Right now, Spark 2.0 does not load hive-site.xml. Based on users' feedback, it seems make sense to still load this conf file.
This PR adds a `hadoopConf` API in `SharedState`, which is `sparkContext.hadoopConfiguration` by default. When users are under hive context, `SharedState.hadoopConf` will load hive-site.xml and append its configs to `sparkContext.hadoopConfiguration`.
When we need to read hadoop config in spark sql, we should call `SessionState.newHadoopConf`, which contains `sparkContext.hadoopConfiguration`, hive-site.xml and sql configs.
## How was this patch tested?
new test in `HiveDataFrameSuite`
Author: Wenchen Fan <wenchen@databricks.com>
Closes #13611 from cloud-fan/hive-site.
Diffstat (limited to 'sql/core/src/test')
-rw-r--r-- | sql/core/src/test/resources/hive-site.xml | 26 | ||||
-rw-r--r-- | sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala | 4 |
2 files changed, 30 insertions, 0 deletions
diff --git a/sql/core/src/test/resources/hive-site.xml b/sql/core/src/test/resources/hive-site.xml new file mode 100644 index 0000000000..17297b3e22 --- /dev/null +++ b/sql/core/src/test/resources/hive-site.xml @@ -0,0 +1,26 @@ +<?xml version="1.0"?> +<?xml-stylesheet type="text/xsl" href="configuration.xsl"?> +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +<configuration> + <property> + <name>hive.in.test</name> + <value>true</value> + <description>Internal marker for test.</description> + </property> +</configuration> diff --git a/sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala index 90465b65bd..89f8685099 100644 --- a/sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala +++ b/sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala @@ -2843,4 +2843,8 @@ class SQLQuerySuite extends QueryTest with SharedSQLContext { sql(s"SELECT '$literal' AS DUMMY"), Row(s"$expected") :: Nil) } + + test("SPARK-15887: hive-site.xml should be loaded") { + assert(spark.sessionState.newHadoopConf().get("hive.in.test") == "true") + } } |