diff options
| author | Daniel Darabos <darabos.daniel@gmail.com> | 2014-07-12 00:07:42 -0700 |
|---|---|---|
| committer | Patrick Wendell <pwendell@gmail.com> | 2014-07-12 00:07:42 -0700 |
| commit | 2245c87af4f507cda361e16f322a14eac25b38fd (patch) | |
| tree | 39c017fd97b1fbd9f03b5a12e76aa4e009b136ee /graphx/src/test | |
| parent | d38887b8a0d00a11d7cf9393e7cb0918c3ec7a22 (diff) | |
| download | spark-2245c87af4f507cda361e16f322a14eac25b38fd.tar.gz spark-2245c87af4f507cda361e16f322a14eac25b38fd.tar.bz2 spark-2245c87af4f507cda361e16f322a14eac25b38fd.zip | |
Use the Executor's ClassLoader in sc.objectFile().
This makes it possible to read classes from the object file which were specified in the user-provided jars. (By default ObjectInputStream uses latestUserDefinedLoader, which may or may not be the right one.)
I created this because I ran into the following problem. I have x:RDD[X] with X being defined in the jar that I provide to SparkContext. I save it with x.saveAsObjectFile("x"). I try to load it with sc.objectFile\[X\]("x"). It fails with ClassNotFoundException.
After a good while of debugging I figured out that Utils.deserialize() most likely uses the ClassLoader of Utils. This is the bootstrap ClassLoader, so it is not aware of the dynamically added jars. This patch fixes the issue.
A more robust fix would be to always default to Thread.currentThread.getContextClassLoader. This would prevent this problem from biting anyone in the future. It would be a bit harder to test though. On the topic of testing, if you'd like to see tests for this, I will need some hand-holding. Thanks!
Author: Daniel Darabos <darabos.daniel@gmail.com>
Closes #181 from darabos/master and squashes the following commits:
45a011a [Daniel Darabos] Add test for SPARK-1877. (Fixed in 52eb54d.)
e13e090 [Daniel Darabos] Merge branch 'master' of https://github.com/apache/spark
61fe0d0 [Daniel Darabos] Fix style (line too long).
1b5df2c [Daniel Darabos] Use the Executor's ClassLoader in sc.objectFile(). This makes it possible to read classes from the object file which were specified in the user-provided jars. (By default ObjectInputStream uses latestUserDefinedLoader, which may or may not be the right one.)
Diffstat (limited to 'graphx/src/test')
0 files changed, 0 insertions, 0 deletions