diff options
author | Matei Zaharia <matei@databricks.com> | 2014-01-02 15:54:54 -0500 |
---|---|---|
committer | Matei Zaharia <matei@databricks.com> | 2014-01-02 15:54:54 -0500 |
commit | ca67909cd4b4accea64498b3bc1421c6d75e33a6 (patch) | |
tree | b6e13c3c0be95ea99c857429c97d3687c61050a3 /pagerank_data.txt | |
parent | 3713f8129a618a633a7aca8c944960c3e7ac9d3b (diff) | |
parent | fec01664a717c8ecf84eaf7a2523a62cd5d3b4b8 (diff) | |
download | spark-ca67909cd4b4accea64498b3bc1421c6d75e33a6.tar.gz spark-ca67909cd4b4accea64498b3bc1421c6d75e33a6.tar.bz2 spark-ca67909cd4b4accea64498b3bc1421c6d75e33a6.zip |
Merge pull request #311 from tmyklebu/master
SPARK-991: Report information gleaned from a Python stacktrace in the UI
Scala:
- Added setCallSite/clearCallSite to SparkContext and JavaSparkContext.
These functions mutate a LocalProperty called "externalCallSite."
- Add a wrapper, getCallSite, that checks for an externalCallSite and, if
none is found, calls the usual Utils.formatSparkCallSite.
- Change everything that calls Utils.formatSparkCallSite to call
getCallSite instead. Except getCallSite.
- Add wrappers to setCallSite/clearCallSite wrappers to JavaSparkContext.
Python:
- Add a gruesome hack to rdd.py that inspects the traceback and guesses
what you want to see in the UI.
- Add a RAII wrapper around said gruesome hack that calls
setCallSite/clearCallSite as appropriate.
- Wire said RAII wrapper up around three calls into the Scala code.
I'm not sure that I hit all the spots with the RAII wrapper. I'm also
not sure that my gruesome hack does exactly what we want.
One could also approach this change by refactoring
runJob/submitJob/runApproximateJob to take a call site, then threading
that parameter through everything that needs to know it.
One might object to the pointless-looking wrappers in JavaSparkContext.
Unfortunately, I can't directly access the SparkContext from
Python---or, if I can, I don't know how---so I need to wrap everything
that matters in JavaSparkContext.
Conflicts:
core/src/main/scala/org/apache/spark/api/java/JavaSparkContext.scala
Diffstat (limited to 'pagerank_data.txt')
0 files changed, 0 insertions, 0 deletions