diff options
| author | Sandy Ryza <sandy@cloudera.com> | 2014-08-01 11:08:39 -0700 |
|---|---|---|
| committer | Patrick Wendell <pwendell@gmail.com> | 2014-08-01 11:08:39 -0700 |
| commit | 8d338f64c4eda45d22ae33f61ef7928011cc2846 (patch) | |
| tree | e15459a74699b857b99ad80c86510b041a69bbff /docs | |
| parent | 5328c0aaa09911c848f9b3e1e1f2397bef932d0f (diff) | |
| download | spark-8d338f64c4eda45d22ae33f61ef7928011cc2846.tar.gz spark-8d338f64c4eda45d22ae33f61ef7928011cc2846.tar.bz2 spark-8d338f64c4eda45d22ae33f61ef7928011cc2846.zip | |
SPARK-2099. Report progress while task is running.
This is a sketch of a patch that allows the UI to show metrics for tasks that have not yet completed. It adds a heartbeat every 2 seconds from the executors to the driver, reporting metrics for all of the executor's tasks.
It still needs unit tests, polish, and cluster testing, but I wanted to put it up to get feedback on the approach.
Author: Sandy Ryza <sandy@cloudera.com>
Closes #1056 from sryza/sandy-spark-2099 and squashes the following commits:
93b9fdb [Sandy Ryza] Up heartbeat interval to 10 seconds and other tidying
132aec7 [Sandy Ryza] Heartbeat and HeartbeatResponse are already Serializable as case classes
38dffde [Sandy Ryza] Additional review feedback and restore test that was removed in BlockManagerSuite
51fa396 [Sandy Ryza] Remove hostname race, add better comments about threading, and some stylistic improvements
3084f10 [Sandy Ryza] Make TaskUIData a case class again
3bda974 [Sandy Ryza] Stylistic fixes
0dae734 [Sandy Ryza] SPARK-2099. Report progress while task is running.
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/configuration.md | 7 |
1 files changed, 7 insertions, 0 deletions
diff --git a/docs/configuration.md b/docs/configuration.md index ea69057b5b..2a71d7b820 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -541,6 +541,13 @@ Apart from these, the following properties are also available, and may be useful output directories. We recommend that users do not disable this except if trying to achieve compatibility with previous versions of Spark. Simply use Hadoop's FileSystem API to delete output directories by hand.</td> </tr> +<tr> + <td>spark.executor.heartbeatInterval</td> + <td>10000</td> + <td>Interval (milliseconds) between each executor's heartbeats to the driver. Heartbeats let + the driver know that the executor is still alive and update it with metrics for in-progress + tasks.</td> +</tr> </table> #### Networking |