diff options
author | Andrew Or <andrewor14@gmail.com> | 2014-07-29 23:52:09 -0700 |
---|---|---|
committer | Patrick Wendell <pwendell@gmail.com> | 2014-07-29 23:52:09 -0700 |
commit | 4ce92ccaf761e48a10fc4fe4927dbfca858ca22b (patch) | |
tree | af298364e8b7cff17d1fa93c7f60b4883b0fa88c /sbt | |
parent | 077f633b4720422c5efbf0382e869ead3dc49612 (diff) | |
download | spark-4ce92ccaf761e48a10fc4fe4927dbfca858ca22b.tar.gz spark-4ce92ccaf761e48a10fc4fe4927dbfca858ca22b.tar.bz2 spark-4ce92ccaf761e48a10fc4fe4927dbfca858ca22b.zip |
[SPARK-2260] Fix standalone-cluster mode, which was broken
The main thing was that spark configs were not propagated to the driver, and so applications that do not specify `master` or `appName` automatically failed. This PR fixes that and a couple of miscellaneous things that are related.
One thing that may or may not be an issue is that the jars must be available on the driver node. In `standalone-cluster` mode, this effectively means these jars must be available on all the worker machines, since the driver is launched on one of them. The semantics here are not the same as `yarn-cluster` mode, where all the relevant jars are uploaded to a distributed cache automatically and shipped to the containers. This is probably not a concern, but still worth a mention.
Author: Andrew Or <andrewor14@gmail.com>
Closes #1538 from andrewor14/standalone-cluster and squashes the following commits:
8c11a0d [Andrew Or] Clean up imports / comments (minor)
2678d13 [Andrew Or] Handle extraJavaOpts properly
7660547 [Andrew Or] Merge branch 'master' of github.com:apache/spark into standalone-cluster
6f64a9b [Andrew Or] Revert changes in YARN
2f2908b [Andrew Or] Fix tests
ed01491 [Andrew Or] Don't go overboard with escaping
8e105e1 [Andrew Or] Merge branch 'master' of github.com:apache/spark into standalone-cluster
b890949 [Andrew Or] Abstract usages of converting spark opts to java opts
79f63a3 [Andrew Or] Move sparkProps into javaOpts
78752f8 [Andrew Or] Fix tests
5a9c6c7 [Andrew Or] Fix line too long
c141a00 [Andrew Or] Don't display "unknown app" on driver log pages
d7e2728 [Andrew Or] Avoid deprecation warning in standalone Client
6ceb14f [Andrew Or] Allow relevant configs to propagate to standalone Driver
7f854bc [Andrew Or] Fix test
855256e [Andrew Or] Fix standalone-cluster mode
fd9da51 [Andrew Or] Formatting changes (minor)
Diffstat (limited to 'sbt')
0 files changed, 0 insertions, 0 deletions