diff options
| author | Cheng Lian <lian.cs.zju@gmail.com> | 2014-08-27 12:39:21 -0700 |
|---|---|---|
| committer | Reynold Xin <rxin@apache.org> | 2014-08-28 11:03:48 -0700 |
| commit | 6d392b36ee1dc6f7e5198dd436e4e62eb816a072 (patch) | |
| tree | 76c60c7186990e6b173807c8d2d0e1636b4ec322 /docs/configuration.md | |
| parent | 70d814665baa8b8ca868d3126452105ecfa5cbff (diff) | |
| download | spark-6d392b36ee1dc6f7e5198dd436e4e62eb816a072.tar.gz spark-6d392b36ee1dc6f7e5198dd436e4e62eb816a072.tar.bz2 spark-6d392b36ee1dc6f7e5198dd436e4e62eb816a072.zip | |
[SPARK-2608][Core] Fixed command line option passing issue over Mesos via SPARK_EXECUTOR_OPTS
This is another try after #2145 to fix [SPARK-2608](https://issues.apache.org/jira/browse/SPARK-2608).
The basic idea is to pass `extraJavaOpts` and `extraLibraryPath` together via environment variable `SPARK_EXECUTOR_OPTS`. This variable is recognized by `spark-class` and not used anywhere else. In this way, we still launch Mesos executors with `spark-class`/`spark-executor`, but avoids the executor side Spark home issue.
Quoted string with spaces is not allowed in either `extraJavaOpts` or `extraLibraryPath` when using Spark over Mesos. The reason is that Mesos passes the whole command line as a single string argument to `sh -c` to start the executor, and this makes shell string escaping non-trivial to handle. This should be fixed in a later release.
Classes in package `org.apache.spark.deploy` shouldn't be used as they assume Spark is deployed in standalone mode, and give wrong executor side Spark home directory. Please refer to comments in #2145 for more details.
Author: Cheng Lian <lian.cs.zju@gmail.com>
Closes #2161 from liancheng/mesos-fix-with-env-var and squashes the following commits:
ba59190 [Cheng Lian] Added fine grained Mesos executor support
1174076 [Cheng Lian] Draft fix for CoarseMesosSchedulerBackend
(cherry picked from commit 935bffe3bf6c91a42288bff8c1ec69fecb41a769)
Signed-off-by: Reynold Xin <rxin@apache.org>
Diffstat (limited to 'docs/configuration.md')
0 files changed, 0 insertions, 0 deletions