diff options
| author | Y.CORP.YAHOO.COM\tgraves <tgraves@thatenemy-lm.(none)> | 2013-07-02 21:18:59 -0500 |
|---|---|---|
| committer | Y.CORP.YAHOO.COM\tgraves <tgraves@thatenemy-lm.(none)> | 2013-07-02 21:18:59 -0500 |
| commit | 923cf929003c67963e273fcdcd5b01baf68df8b5 (patch) | |
| tree | 7049909ae9f19e9a22382e584c43dd8e3c4f2a8b /streaming | |
| parent | bad51c7cb4e15b68ce49ac4886e4631b9fb7e308 (diff) | |
| download | spark-923cf929003c67963e273fcdcd5b01baf68df8b5.tar.gz spark-923cf929003c67963e273fcdcd5b01baf68df8b5.tar.bz2 spark-923cf929003c67963e273fcdcd5b01baf68df8b5.zip | |
Rework from pull request. Removed --user option from Spark on Yarn Client, made the user of JAVA_HOME environment
variable conditional on if its set, and created addCredentials in each of the SparkHadoopUtil classes
to only add the credentials when the profile is hadoop2-yarn.
Diffstat (limited to 'streaming')
| -rw-r--r-- | streaming/src/main/scala/spark/streaming/PairDStreamFunctions.scala | 4 |
1 files changed, 0 insertions, 4 deletions
diff --git a/streaming/src/main/scala/spark/streaming/PairDStreamFunctions.scala b/streaming/src/main/scala/spark/streaming/PairDStreamFunctions.scala index 20ee1d3c5d..8d0a83d439 100644 --- a/streaming/src/main/scala/spark/streaming/PairDStreamFunctions.scala +++ b/streaming/src/main/scala/spark/streaming/PairDStreamFunctions.scala @@ -472,10 +472,6 @@ extends Serializable { outputFormatClass: Class[_ <: OutputFormat[_, _]], conf: JobConf = new JobConf ) { - // make sure to propogate any credentials from the current user to the jobConf - // for Hadoop security - val jobCreds = conf.getCredentials(); - jobCreds.mergeAll(UserGroupInformation.getCurrentUser().getCredentials()) val saveFunc = (rdd: RDD[(K, V)], time: Time) => { val file = rddToFileName(prefix, suffix, time) rdd.saveAsHadoopFile(file, keyClass, valueClass, outputFormatClass, conf) |