diff options
| author | Bouke van der Bijl <boukevanderbijl@gmail.com> | 2014-05-08 20:43:37 -0700 |
|---|---|---|
| committer | Patrick Wendell <pwendell@gmail.com> | 2014-05-08 20:43:49 -0700 |
| commit | ab912271a2f7ab267102c524755c30a9c1785cd0 (patch) | |
| tree | b2108b276086084e2f0d75912cb0a58b51225ee4 | |
| parent | 9ed17ff34503854c676986f1553a103a370d66c6 (diff) | |
| download | spark-ab912271a2f7ab267102c524755c30a9c1785cd0.tar.gz spark-ab912271a2f7ab267102c524755c30a9c1785cd0.tar.bz2 spark-ab912271a2f7ab267102c524755c30a9c1785cd0.zip | |
Include the sbin/spark-config.sh in spark-executor
This is needed because broadcast values are broken on pyspark on Mesos, it tries to import pyspark but can't, as the PYTHONPATH is not set due to changes in ff5be9a4
https://issues.apache.org/jira/browse/SPARK-1725
Author: Bouke van der Bijl <boukevanderbijl@gmail.com>
Closes #651 from bouk/include-spark-config-in-mesos-executor and squashes the following commits:
b2f1295 [Bouke van der Bijl] Inline PYTHONPATH in spark-executor
eedbbcc [Bouke van der Bijl] Include the sbin/spark-config.sh in spark-executor
(cherry picked from commit 2fd2752e572921a9010614eb1c1238c493d34a7c)
Signed-off-by: Patrick Wendell <pwendell@gmail.com>
| -rwxr-xr-x | sbin/spark-executor | 3 |
1 files changed, 3 insertions, 0 deletions
diff --git a/sbin/spark-executor b/sbin/spark-executor index de5bfab563..336549f29c 100755 --- a/sbin/spark-executor +++ b/sbin/spark-executor @@ -19,5 +19,8 @@ FWDIR="$(cd `dirname $0`/..; pwd)" +export PYTHONPATH=$FWDIR/python:$PYTHONPATH +export PYTHONPATH=$FWDIR/python/lib/py4j-0.8.1-src.zip:$PYTHONPATH + echo "Running spark-executor with framework dir = $FWDIR" exec $FWDIR/bin/spark-class org.apache.spark.executor.MesosExecutorBackend |