diff options
| author | zsxwing <zsxwing@gmail.com> | 2015-03-27 12:31:06 +0000 |
|---|---|---|
| committer | Sean Owen <sowen@cloudera.com> | 2015-03-27 12:31:06 +0000 |
| commit | da546b7ba03d84d7f6af97fe04471b12f5b3392f (patch) | |
| tree | 985b7d1988a66ece4f7adf68cea74215d22b2b2d /core | |
| parent | f43a61031fd7d9d4fab3d8ac584e7b4c7c5e1035 (diff) | |
| download | spark-da546b7ba03d84d7f6af97fe04471b12f5b3392f.tar.gz spark-da546b7ba03d84d7f6af97fe04471b12f5b3392f.tar.bz2 spark-da546b7ba03d84d7f6af97fe04471b12f5b3392f.zip | |
[SPARK-6556][Core] Fix wrong parsing logic of executorTimeoutMs and checkTimeoutIntervalMs in HeartbeatReceiver
The current reading logic of `executorTimeoutMs` is:
```Scala
private val executorTimeoutMs = sc.conf.getLong("spark.network.timeout",
sc.conf.getLong("spark.storage.blockManagerSlaveTimeoutMs", 120)) * 1000
```
So if `spark.storage.blockManagerSlaveTimeoutMs` is 10000 and `spark.network.timeout` is not set, executorTimeoutMs will be 10000 * 1000. But the correct value should have been 10000.
`checkTimeoutIntervalMs` has the same issue.
This PR fixes them.
Author: zsxwing <zsxwing@gmail.com>
Closes #5209 from zsxwing/SPARK-6556 and squashes the following commits:
6a0a411 [zsxwing] Fix docs
c7d5422 [zsxwing] Add comments for executorTimeoutMs and checkTimeoutIntervalMs
ccd5147 [zsxwing] Fix wrong parsing logic of executorTimeoutMs and checkTimeoutIntervalMs in HeartbeatReceiver
Diffstat (limited to 'core')
| -rw-r--r-- | core/src/main/scala/org/apache/spark/HeartbeatReceiver.scala | 17 |
1 files changed, 11 insertions, 6 deletions
diff --git a/core/src/main/scala/org/apache/spark/HeartbeatReceiver.scala b/core/src/main/scala/org/apache/spark/HeartbeatReceiver.scala index 715f292f03..548dcb93c3 100644 --- a/core/src/main/scala/org/apache/spark/HeartbeatReceiver.scala +++ b/core/src/main/scala/org/apache/spark/HeartbeatReceiver.scala @@ -49,12 +49,17 @@ private[spark] class HeartbeatReceiver(sc: SparkContext, scheduler: TaskSchedule // executor ID -> timestamp of when the last heartbeat from this executor was received private val executorLastSeen = new mutable.HashMap[String, Long] - - private val executorTimeoutMs = sc.conf.getLong("spark.network.timeout", - sc.conf.getLong("spark.storage.blockManagerSlaveTimeoutMs", 120)) * 1000 - - private val checkTimeoutIntervalMs = sc.conf.getLong("spark.network.timeoutInterval", - sc.conf.getLong("spark.storage.blockManagerTimeoutIntervalMs", 60)) * 1000 + + // "spark.network.timeout" uses "seconds", while `spark.storage.blockManagerSlaveTimeoutMs` uses + // "milliseconds" + private val executorTimeoutMs = sc.conf.getOption("spark.network.timeout").map(_.toLong * 1000). + getOrElse(sc.conf.getLong("spark.storage.blockManagerSlaveTimeoutMs", 120000)) + + // "spark.network.timeoutInterval" uses "seconds", while + // "spark.storage.blockManagerTimeoutIntervalMs" uses "milliseconds" + private val checkTimeoutIntervalMs = + sc.conf.getOption("spark.network.timeoutInterval").map(_.toLong * 1000). + getOrElse(sc.conf.getLong("spark.storage.blockManagerTimeoutIntervalMs", 60000)) private var timeoutCheckingTask: Cancellable = null |