diff options
author | CodingCat <zhunansjtu@gmail.com> | 2014-02-26 23:42:15 -0800 |
---|---|---|
committer | Patrick Wendell <pwendell@gmail.com> | 2014-02-26 23:42:15 -0800 |
commit | 345df5f4a9c16a6a87440afa2b09082fc3d224bd (patch) | |
tree | 9057ae06eab304637361c6c3546bd011e6a4aef6 /core | |
parent | 6ccd6c55bdfcf1a4f8f8fd1a9715d4a45ec1703a (diff) | |
download | spark-345df5f4a9c16a6a87440afa2b09082fc3d224bd.tar.gz spark-345df5f4a9c16a6a87440afa2b09082fc3d224bd.tar.bz2 spark-345df5f4a9c16a6a87440afa2b09082fc3d224bd.zip |
[SPARK-1089] fix the regression problem on ADD_JARS in 0.9
https://spark-project.atlassian.net/browse/SPARK-1089
copied from JIRA, reported by @ash211
"Using the ADD_JARS environment variable with spark-shell used to add the jar to both the shell and the various workers. Now it only adds to the workers and importing a custom class in the shell is broken.
The workaround is to add custom jars to both ADD_JARS and SPARK_CLASSPATH.
We should fix ADD_JARS so it works properly again.
See various threads on the user list:
https://mail-archives.apache.org/mod_mbox/incubator-spark-user/201402.mbox/%3CCAJbo4neMLiTrnm1XbyqomWmp0m+EUcg4yE-txuRGSVKOb5KLeA@mail.gmail.com%3E
(another one that doesn't appear in the archives yet titled "ADD_JARS not working on 0.9")"
The reason of this bug is two-folds
in the current implementation of SparkILoop.scala, the settings.classpath is not set properly when the process() method is invoked
the weird behaviour of Scala 2.10, (I personally thought it is a bug)
if we simply set value of a PathSettings object (like settings.classpath), the isDefault is not set to true (this is a flag showing if the variable is modified), so it makes the PathResolver loads the default CLASSPATH environment variable value to calculated the path (see https://github.com/scala/scala/blob/2.10.x/src/compiler/scala/tools/util/PathResolver.scala#L215)
what we have to do is to manually make this flag set, (https://github.com/CodingCat/incubator-spark/blob/e3991d97ddc33e77645e4559b13bf78b9e68239a/repl/src/main/scala/org/apache/spark/repl/SparkILoop.scala#L884)
Author: CodingCat <zhunansjtu@gmail.com>
Closes #13 from CodingCat/SPARK-1089 and squashes the following commits:
8af81e7 [CodingCat] impose non-null settings
9aa2125 [CodingCat] code cleaning
ce36676 [CodingCat] code cleaning
e045582 [CodingCat] fix the regression problem on ADD_JARS in 0.9
Diffstat (limited to 'core')
0 files changed, 0 insertions, 0 deletions