diff options
| author | Bouke van der Bijl <boukevanderbijl@gmail.com> | 2014-05-08 20:43:37 -0700 |
|---|---|---|
| committer | Patrick Wendell <pwendell@gmail.com> | 2014-05-08 20:43:37 -0700 |
| commit | 2fd2752e572921a9010614eb1c1238c493d34a7c (patch) | |
| tree | feee2ee7b2dd82ab62c761d776fba5d4035dba69 /sbin | |
| parent | 191279ce4edb940821d11a6b25cd33c8ad0af054 (diff) | |
| download | spark-2fd2752e572921a9010614eb1c1238c493d34a7c.tar.gz spark-2fd2752e572921a9010614eb1c1238c493d34a7c.tar.bz2 spark-2fd2752e572921a9010614eb1c1238c493d34a7c.zip | |
Include the sbin/spark-config.sh in spark-executor
This is needed because broadcast values are broken on pyspark on Mesos, it tries to import pyspark but can't, as the PYTHONPATH is not set due to changes in ff5be9a4
https://issues.apache.org/jira/browse/SPARK-1725
Author: Bouke van der Bijl <boukevanderbijl@gmail.com>
Closes #651 from bouk/include-spark-config-in-mesos-executor and squashes the following commits:
b2f1295 [Bouke van der Bijl] Inline PYTHONPATH in spark-executor
eedbbcc [Bouke van der Bijl] Include the sbin/spark-config.sh in spark-executor
Diffstat (limited to 'sbin')
| -rwxr-xr-x | sbin/spark-executor | 3 |
1 files changed, 3 insertions, 0 deletions
diff --git a/sbin/spark-executor b/sbin/spark-executor index de5bfab563..336549f29c 100755 --- a/sbin/spark-executor +++ b/sbin/spark-executor @@ -19,5 +19,8 @@ FWDIR="$(cd `dirname $0`/..; pwd)" +export PYTHONPATH=$FWDIR/python:$PYTHONPATH +export PYTHONPATH=$FWDIR/python/lib/py4j-0.8.1-src.zip:$PYTHONPATH + echo "Running spark-executor with framework dir = $FWDIR" exec $FWDIR/bin/spark-class org.apache.spark.executor.MesosExecutorBackend |