diff options
| author | Matei Zaharia <matei@eecs.berkeley.edu> | 2013-10-30 12:03:44 -0700 |
|---|---|---|
| committer | Matei Zaharia <matei@eecs.berkeley.edu> | 2013-10-30 12:03:44 -0700 |
| commit | 618c1f6cf3008caae7a8c0202721a6bd77d29a0f (patch) | |
| tree | 58020a6617f7ee5c1b8c33d675ce89ebb6ead6a4 /examples | |
| parent | 745dc4290834e8c38d61600e23e4706b0ae47179 (diff) | |
| parent | de0285556ac3fa068e9b33d6db14af2168504c1f (diff) | |
| download | spark-618c1f6cf3008caae7a8c0202721a6bd77d29a0f.tar.gz spark-618c1f6cf3008caae7a8c0202721a6bd77d29a0f.tar.bz2 spark-618c1f6cf3008caae7a8c0202721a6bd77d29a0f.zip | |
Merge pull request #125 from velvia/2013-10/local-jar-uri
Add support for local:// URI scheme for addJars()
This PR adds support for a new URI scheme for SparkContext.addJars(): `local://file/path`.
The *local* scheme indicates that the `/file/path` exists on every worker node. The reason for its existence is for big library JARs, which would be really expensive to serve using the standard HTTP fileserver distribution method, especially for big clusters. Today the only inexpensive method (assuming such a file is on every host, via say NFS, rsync, etc.) of doing this is to add the JAR to the SPARK_CLASSPATH, but we want a method where the user does not need to modify the Spark configuration.
I would add something to the docs, but it's not obvious where to add it.
Oh, and it would be great if this could be merged in time for 0.8.1.
Diffstat (limited to 'examples')
0 files changed, 0 insertions, 0 deletions