diff options
| author | Raymond Liu <raymond.liu@intel.com> | 2014-07-03 19:24:22 -0700 |
|---|---|---|
| committer | Aaron Davidson <aaron@databricks.com> | 2014-07-03 19:24:37 -0700 |
| commit | d2f25346780bbaf03bf17ce38d8a272336c2c814 (patch) | |
| tree | a637e5430c55efe73045667175bfee7bfbb53556 /core | |
| parent | ff6ec257218fd28869aeaefd41ef81fd39dce1fb (diff) | |
| download | spark-d2f25346780bbaf03bf17ce38d8a272336c2c814.tar.gz spark-d2f25346780bbaf03bf17ce38d8a272336c2c814.tar.bz2 spark-d2f25346780bbaf03bf17ce38d8a272336c2c814.zip | |
[SPARK-1097] Workaround Hadoop conf ConcurrentModification issue
Workaround Hadoop conf ConcurrentModification issue
Author: Raymond Liu <raymond.liu@intel.com>
Closes #1273 from colorant/hadoopRDD and squashes the following commits:
994e98b [Raymond Liu] Address comments
e2cda3d [Raymond Liu] Workaround Hadoop conf ConcurrentModification issue
(cherry picked from commit 5fa0a05763ab1d527efe20e3b10539ac5ffc36de)
Signed-off-by: Aaron Davidson <aaron@databricks.com>
Diffstat (limited to 'core')
| -rw-r--r-- | core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala | 4 |
1 files changed, 2 insertions, 2 deletions
diff --git a/core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala b/core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala index 2aa111d600..a55b2266f8 100644 --- a/core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala +++ b/core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala @@ -140,8 +140,8 @@ class HadoopRDD[K, V]( // local process. The local cache is accessed through HadoopRDD.putCachedMetadata(). // The caching helps minimize GC, since a JobConf can contain ~10KB of temporary objects. // synchronize to prevent ConcurrentModificationException (Spark-1097, Hadoop-10456) - broadcastedConf.synchronized { - val newJobConf = new JobConf(broadcastedConf.value.value) + conf.synchronized { + val newJobConf = new JobConf(conf) initLocalJobConfFuncOpt.map(f => f(newJobConf)) HadoopRDD.putCachedMetadata(jobConfCacheKey, newJobConf) newJobConf |