aboutsummaryrefslogtreecommitdiff
path: root/docker
diff options
context:
space:
mode:
authorChip Senkbeil <rcsenkbe@us.ibm.com>2014-08-27 13:01:11 -0700
committerMatei Zaharia <matei@databricks.com>2014-08-27 13:01:11 -0700
commit191d7cf2a655d032f160b9fa181730364681d0e7 (patch)
treef7163dab0c75e1eb509315694d2fd381ce42db9e /docker
parent4238c17dc9e1f2f93cc9e6c768f92bd27bf1df66 (diff)
downloadspark-191d7cf2a655d032f160b9fa181730364681d0e7.tar.gz
spark-191d7cf2a655d032f160b9fa181730364681d0e7.tar.bz2
spark-191d7cf2a655d032f160b9fa181730364681d0e7.zip
[SPARK-3256] Added support for :cp <jar> that was broken in Scala 2.10.x for REPL
As seen with [SI-6502](https://issues.scala-lang.org/browse/SI-6502) of Scala, the _:cp_ command was broken in Scala 2.10.x. As the Spark shell is a friendly wrapper on top of the Scala REPL, it is also affected by this problem. My solution was to alter the internal classpath and invalidate any new entries. I also had to add the ability to add new entries to the parent classloader of the interpreter (SparkIMain's global). The advantage of this versus wiping the interpreter and replaying all of the commands is that you don't have to worry about rerunning heavy Spark-related commands (going to the cluster) or potentially reloading data that might have changed. Instead, you get to work from where you left off. Until this is fixed upstream for 2.10.x, I had to use reflection to alter the internal compiler classpath. The solution now looks like this: ![screen shot 2014-08-13 at 3 46 02 pm](https://cloud.githubusercontent.com/assets/2481802/3912625/f02b1440-232c-11e4-9bf6-bafb3e352d14.png) Author: Chip Senkbeil <rcsenkbe@us.ibm.com> Closes #1929 from rcsenkbeil/FixReplClasspathSupport and squashes the following commits: f420cbf [Chip Senkbeil] Added SparkContext.addJar calls to support executing code on remote clusters a826795 [Chip Senkbeil] Updated AddUrlsToClasspath to use 'new Run' suggestion over hackish compiler error 2ff1d86 [Chip Senkbeil] Added compilation failure on symbols hack to get Scala classes to load correctly a220639 [Chip Senkbeil] Added support for :cp <jar> that was broken in Scala 2.10.x for REPL
Diffstat (limited to 'docker')
0 files changed, 0 insertions, 0 deletions