diff options
author | Adrian Petrescu <apetresc@gmail.com> | 2016-09-20 10:49:02 +0100 |
---|---|---|
committer | Sean Owen <sowen@cloudera.com> | 2016-09-20 10:49:02 +0100 |
commit | 4a426ff8aea4faa31a3016a453dec5b7954578dd (patch) | |
tree | 037ee8e6df4d767aa5c001cd2101b93c3463d72c /sql/core | |
parent | f039d964d152c0aeb5b71eb5188a9a7fd4b5aef3 (diff) | |
download | spark-4a426ff8aea4faa31a3016a453dec5b7954578dd.tar.gz spark-4a426ff8aea4faa31a3016a453dec5b7954578dd.tar.bz2 spark-4a426ff8aea4faa31a3016a453dec5b7954578dd.zip |
[SPARK-17437] Add uiWebUrl to JavaSparkContext and pyspark.SparkContext
## What changes were proposed in this pull request?
The Scala version of `SparkContext` has a handy field called `uiWebUrl` that tells you which URL the SparkUI spawned by that instance lives at. This is often very useful because the value for `spark.ui.port` in the config is only a suggestion; if that port number is taken by another Spark instance on the same machine, Spark will just keep incrementing the port until it finds a free one. So, on a machine with a lot of running PySpark instances, you often have to start trying all of them one-by-one until you find your application name.
Scala users have a way around this with `uiWebUrl` but Java and Python users do not. This pull request fixes this in the most straightforward way possible, simply propagating this field through the `JavaSparkContext` and into pyspark through the Java gateway.
Please let me know if any additional documentation/testing is needed.
## How was this patch tested?
Existing tests were run to make sure there were no regressions, and a binary distribution was created and tested manually for the correct value of `sc.uiWebPort` in a variety of circumstances.
Author: Adrian Petrescu <apetresc@gmail.com>
Closes #15000 from apetresc/pyspark-uiweburl.
Diffstat (limited to 'sql/core')
0 files changed, 0 insertions, 0 deletions