diff options
| author | Josh Rosen <joshrosen@databricks.com> | 2015-02-04 17:18:03 -0800 |
|---|---|---|
| committer | Patrick Wendell <patrick@databricks.com> | 2015-02-04 17:18:03 -0800 |
| commit | 9a7ce70eabc0ccaa036e142fc97bf0d37faa0b63 (patch) | |
| tree | 71d6035cf95af5f681aeb578ec65417962671a5e /docs/configuration.md | |
| parent | dc101b0e4e23dffddbc2f70d14a19fae5d87a328 (diff) | |
| download | spark-9a7ce70eabc0ccaa036e142fc97bf0d37faa0b63.tar.gz spark-9a7ce70eabc0ccaa036e142fc97bf0d37faa0b63.tar.bz2 spark-9a7ce70eabc0ccaa036e142fc97bf0d37faa0b63.zip | |
[SPARK-5411] Allow SparkListeners to be specified in SparkConf and loaded when creating SparkContext
This patch introduces a new configuration option, `spark.extraListeners`, that allows SparkListeners to be specified in SparkConf and registered before the SparkContext is initialized. From the configuration documentation:
> A comma-separated list of classes that implement SparkListener; when initializing SparkContext, instances of these classes will be created and registered with Spark's listener bus. If a class has a single-argument constructor that accepts a SparkConf, that constructor will be called; otherwise, a zero-argument constructor will be called. If no valid constructor can be found, the SparkContext creation will fail with an exception.
This motivation for this patch is to allow monitoring code to be easily injected into existing Spark programs without having to modify those programs' code.
Author: Josh Rosen <joshrosen@databricks.com>
Closes #4111 from JoshRosen/SPARK-5190-register-sparklistener-in-sc-constructor and squashes the following commits:
8370839 [Josh Rosen] Two minor fixes after merging with master
6e0122c [Josh Rosen] Merge remote-tracking branch 'origin/master' into SPARK-5190-register-sparklistener-in-sc-constructor
1a5b9a0 [Josh Rosen] Remove SPARK_EXTRA_LISTENERS environment variable.
2daff9b [Josh Rosen] Add a couple of explanatory comments for SPARK_EXTRA_LISTENERS.
b9973da [Josh Rosen] Add test to ensure that conf and env var settings are merged, not overriden.
d6f3113 [Josh Rosen] Use getConstructors() instead of try-catch to find right constructor.
d0d276d [Josh Rosen] Move code into setupAndStartListenerBus() method
b22b379 [Josh Rosen] Instantiate SparkListeners from classes listed in configurations.
9c0d8f1 [Josh Rosen] Revert "[SPARK-5190] Allow SparkListeners to be registered before SparkContext starts."
217ecc0 [Josh Rosen] Revert "Add addSparkListener to JavaSparkContext"
25988f3 [Josh Rosen] Add addSparkListener to JavaSparkContext
163ba19 [Josh Rosen] [SPARK-5190] Allow SparkListeners to be registered before SparkContext starts.
Diffstat (limited to 'docs/configuration.md')
| -rw-r--r-- | docs/configuration.md | 11 |
1 files changed, 11 insertions, 0 deletions
diff --git a/docs/configuration.md b/docs/configuration.md index 62d3fca937..8b1d7598c4 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -190,6 +190,17 @@ of the most common options to set are: Logs the effective SparkConf as INFO when a SparkContext is started. </td> </tr> +<tr> + <td><code>spark.extraListeners</code></td> + <td>(none)</td> + <td> + A comma-separated list of classes that implement <code>SparkListener</code>; when initializing + SparkContext, instances of these classes will be created and registered with Spark's listener + bus. If a class has a single-argument constructor that accepts a SparkConf, that constructor + will be called; otherwise, a zero-argument constructor will be called. If no valid constructor + can be found, the SparkContext creation will fail with an exception. + </td> +</tr> </table> Apart from these, the following properties are also available, and may be useful in some situations: |