diff options
author | Carson Wang <carson.wang@intel.com> | 2015-12-03 16:39:12 -0800 |
---|---|---|
committer | Marcelo Vanzin <vanzin@cloudera.com> | 2015-12-03 16:39:12 -0800 |
commit | b6e9963ee4bf0ffb62c8e9829a551bcdc31e12e3 (patch) | |
tree | 756c1e9cee472fb3f8125f1a367ede5251bc41cd /core/src/main/java | |
parent | f434f36d508eb4dcade70871611fc022ae0feb56 (diff) | |
download | spark-b6e9963ee4bf0ffb62c8e9829a551bcdc31e12e3.tar.gz spark-b6e9963ee4bf0ffb62c8e9829a551bcdc31e12e3.tar.bz2 spark-b6e9963ee4bf0ffb62c8e9829a551bcdc31e12e3.zip |
[SPARK-11206] Support SQL UI on the history server (resubmit)
Resubmit #9297 and #9991
On the live web UI, there is a SQL tab which provides valuable information for the SQL query. But once the workload is finished, we won't see the SQL tab on the history server. It will be helpful if we support SQL UI on the history server so we can analyze it even after its execution.
To support SQL UI on the history server:
1. I added an onOtherEvent method to the SparkListener trait and post all SQL related events to the same event bus.
2. Two SQL events SparkListenerSQLExecutionStart and SparkListenerSQLExecutionEnd are defined in the sql module.
3. The new SQL events are written to event log using Jackson.
4. A new trait SparkHistoryListenerFactory is added to allow the history server to feed events to the SQL history listener. The SQL implementation is loaded at runtime using java.util.ServiceLoader.
Author: Carson Wang <carson.wang@intel.com>
Closes #10061 from carsonwang/SqlHistoryUI.
Diffstat (limited to 'core/src/main/java')
-rw-r--r-- | core/src/main/java/org/apache/spark/JavaSparkListener.java | 3 | ||||
-rw-r--r-- | core/src/main/java/org/apache/spark/SparkFirehoseListener.java | 4 |
2 files changed, 7 insertions, 0 deletions
diff --git a/core/src/main/java/org/apache/spark/JavaSparkListener.java b/core/src/main/java/org/apache/spark/JavaSparkListener.java index fa9acf0a15..23bc9a2e81 100644 --- a/core/src/main/java/org/apache/spark/JavaSparkListener.java +++ b/core/src/main/java/org/apache/spark/JavaSparkListener.java @@ -82,4 +82,7 @@ public class JavaSparkListener implements SparkListener { @Override public void onBlockUpdated(SparkListenerBlockUpdated blockUpdated) { } + @Override + public void onOtherEvent(SparkListenerEvent event) { } + } diff --git a/core/src/main/java/org/apache/spark/SparkFirehoseListener.java b/core/src/main/java/org/apache/spark/SparkFirehoseListener.java index 1214d05ba6..e6b24afd88 100644 --- a/core/src/main/java/org/apache/spark/SparkFirehoseListener.java +++ b/core/src/main/java/org/apache/spark/SparkFirehoseListener.java @@ -118,4 +118,8 @@ public class SparkFirehoseListener implements SparkListener { onEvent(blockUpdated); } + @Override + public void onOtherEvent(SparkListenerEvent event) { + onEvent(event); + } } |