diff options
| author | Diana Carroll <dcarroll@cloudera.com> | 2014-04-03 15:48:42 -0700 |
|---|---|---|
| committer | Matei Zaharia <matei@databricks.com> | 2014-04-03 15:48:42 -0700 |
| commit | a599e43d6e0950f6b6b32150ce264a8c2711470c (patch) | |
| tree | 60cebed83c3e6ee3451f75f0fa503beabf26640d /bin | |
| parent | b8f534196f9a8c99f75728a06e62282d139dee28 (diff) | |
| download | spark-a599e43d6e0950f6b6b32150ce264a8c2711470c.tar.gz spark-a599e43d6e0950f6b6b32150ce264a8c2711470c.tar.bz2 spark-a599e43d6e0950f6b6b32150ce264a8c2711470c.zip | |
[SPARK-1134] Fix and document passing of arguments to IPython
This is based on @dianacarroll's previous pull request https://github.com/apache/spark/pull/227, and @joshrosen's comments on https://github.com/apache/spark/pull/38. Since we do want to allow passing arguments to IPython, this does the following:
* It documents that IPython can't be used with standalone jobs for now. (Later versions of IPython will deal with PYTHONSTARTUP properly and enable this, see https://github.com/ipython/ipython/pull/5226, but no released version has that fix.)
* If you run `pyspark` with `IPYTHON=1`, it passes your command-line arguments to it. This way you can do stuff like `IPYTHON=1 bin/pyspark notebook`.
* The old `IPYTHON_OPTS` remains, but I've removed it from the documentation. This is in case people read an old tutorial that uses it.
This is not a perfect solution and I'd also be okay with keeping things as they are today (ignoring `$@` for IPython and using IPYTHON_OPTS), and only doing the doc change. With this change though, when IPython fixes https://github.com/ipython/ipython/pull/5226, people will immediately be able to do `IPYTHON=1 bin/pyspark myscript.py` to run a standalone script and get all the benefits of running scripts in IPython (presumably better debugging and such). Without it, there will be no way to run scripts in IPython.
@joshrosen you should probably take the final call on this.
Author: Diana Carroll <dcarroll@cloudera.com>
Closes #294 from mateiz/spark-1134 and squashes the following commits:
747bb13 [Diana Carroll] SPARK-1134 bug with ipython prevents non-interactive use with spark; only call ipython if no command line arguments were supplied
Diffstat (limited to 'bin')
| -rwxr-xr-x | bin/pyspark | 3 |
1 files changed, 2 insertions, 1 deletions
diff --git a/bin/pyspark b/bin/pyspark index 67e1f61eeb..cad982bc33 100755 --- a/bin/pyspark +++ b/bin/pyspark @@ -55,7 +55,8 @@ if [ -n "$IPYTHON_OPTS" ]; then IPYTHON=1 fi -if [[ "$IPYTHON" = "1" ]] ; then +# Only use ipython if no command line arguments were provided [SPARK-1134] +if [[ "$IPYTHON" = "1" && $# = 0 ]] ; then exec ipython $IPYTHON_OPTS else exec "$PYSPARK_PYTHON" "$@" |