diff options
| author | jerryshao <sshao@hortonworks.com> | 2017-02-24 09:31:52 -0800 |
|---|---|---|
| committer | Marcelo Vanzin <vanzin@cloudera.com> | 2017-02-24 09:31:52 -0800 |
| commit | a920a4369434c84274866a09f61e402232c3b47c (patch) | |
| tree | ceab5c733ff40106f825ddbd9f2cbde18f6b1b04 /sql/hive | |
| parent | b0a8c16fecd4337f77bfbe4b45884254d7af52bd (diff) | |
| download | spark-a920a4369434c84274866a09f61e402232c3b47c.tar.gz spark-a920a4369434c84274866a09f61e402232c3b47c.tar.bz2 spark-a920a4369434c84274866a09f61e402232c3b47c.zip | |
[SPARK-19038][YARN] Avoid overwriting keytab configuration in yarn-client
## What changes were proposed in this pull request?
Because yarn#client will reset the `spark.yarn.keytab` configuration to point to the location in distributed file, so if user still uses the old `SparkConf` to create `SparkSession` with Hive enabled, it will read keytab from the path in distributed cached. This is OK for yarn cluster mode, but in yarn client mode where driver is running out of container, it will be failed to fetch the keytab.
So here we should avoid reseting this configuration in the `yarn#client` and only overwriting it for AM, so using `spark.yarn.keytab` could get correct keytab path no matter running in client (keytab in local fs) or cluster (keytab in distributed cache) mode.
## How was this patch tested?
Verified in security cluster.
Author: jerryshao <sshao@hortonworks.com>
Closes #16923 from jerryshao/SPARK-19038.
Diffstat (limited to 'sql/hive')
| -rw-r--r-- | sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala | 4 |
1 files changed, 0 insertions, 4 deletions
diff --git a/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala b/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala index dc9c3ff335..24dfd33bc3 100644 --- a/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala +++ b/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala @@ -106,10 +106,6 @@ private[hive] class HiveClientImpl( // Set up kerberos credentials for UserGroupInformation.loginUser within // current class loader - // Instead of using the spark conf of the current spark context, a new - // instance of SparkConf is needed for the original value of spark.yarn.keytab - // and spark.yarn.principal set in SparkSubmit, as yarn.Client resets the - // keytab configuration for the link name in distributed cache if (sparkConf.contains("spark.yarn.principal") && sparkConf.contains("spark.yarn.keytab")) { val principalName = sparkConf.get("spark.yarn.principal") val keytabFileName = sparkConf.get("spark.yarn.keytab") |