diff options
author | Andrew Or <andrewor14@gmail.com> | 2014-04-22 14:27:49 -0700 |
---|---|---|
committer | Patrick Wendell <pwendell@gmail.com> | 2014-04-22 14:27:59 -0700 |
commit | 54c96c2343dff6e6defe4e2ad576ab8a82d45140 (patch) | |
tree | cca8cd7fa9e02b48f5a4d6adcc2bde6fbc38b8dd /core | |
parent | 898fc3480c408a7e88fa95f27ca8f72d660cdff7 (diff) | |
download | spark-54c96c2343dff6e6defe4e2ad576ab8a82d45140.tar.gz spark-54c96c2343dff6e6defe4e2ad576ab8a82d45140.tar.bz2 spark-54c96c2343dff6e6defe4e2ad576ab8a82d45140.zip |
[Fix #204] Eliminate delay between binding and log checking
**Bug**: In the existing history server, there is a `spark.history.updateInterval` seconds delay before application logs show up on the UI.
**Cause**: This is because the following events happen in this order: (1) The background thread that checks for logs starts, but realizes the server has not yet bound and so waits for N seconds, (2) server binds, (3) N seconds later the background thread finds that the server has finally bound to a port, and so finally checks for application logs.
**Fix**: This PR forces the log checking thread to start immediately after binding. It also documents two relevant environment variables that are currently missing.
Author: Andrew Or <andrewor14@gmail.com>
Closes #441 from andrewor14/history-server-fix and squashes the following commits:
b2eb46e [Andrew Or] Document SPARK_PUBLIC_DNS and SPARK_HISTORY_OPTS for the history server
e8d1fbc [Andrew Or] Eliminate delay between binding and checking for logs
(cherry picked from commit 745e496c59cfece2fcd6120ecc366dcab07b293a)
Signed-off-by: Patrick Wendell <pwendell@gmail.com>
Diffstat (limited to 'core')
-rw-r--r-- | core/src/main/scala/org/apache/spark/deploy/history/HistoryServer.scala | 5 |
1 files changed, 5 insertions, 0 deletions
diff --git a/core/src/main/scala/org/apache/spark/deploy/history/HistoryServer.scala b/core/src/main/scala/org/apache/spark/deploy/history/HistoryServer.scala index cf64700f90..b8f56234d3 100644 --- a/core/src/main/scala/org/apache/spark/deploy/history/HistoryServer.scala +++ b/core/src/main/scala/org/apache/spark/deploy/history/HistoryServer.scala @@ -98,6 +98,11 @@ class HistoryServer( def initialize() { attachPage(new HistoryPage(this)) attachHandler(createStaticHandler(STATIC_RESOURCE_DIR, "/static")) + } + + /** Bind to the HTTP server behind this web interface. */ + override def bind() { + super.bind() logCheckingThread.start() } |