diff options
author | Josh Rosen <joshrosen@apache.org> | 2014-10-09 16:08:07 -0700 |
---|---|---|
committer | Josh Rosen <joshrosen@apache.org> | 2014-10-09 16:08:07 -0700 |
commit | 4e9b551a0b807f5a2cc6679165c8be4e88a3d077 (patch) | |
tree | d7d6be00b5dbb04d38eab034c9dc18e73b5002de /docs | |
parent | ac302052870a650d56f2d3131c27755bb2960ad7 (diff) | |
download | spark-4e9b551a0b807f5a2cc6679165c8be4e88a3d077.tar.gz spark-4e9b551a0b807f5a2cc6679165c8be4e88a3d077.tar.bz2 spark-4e9b551a0b807f5a2cc6679165c8be4e88a3d077.zip |
[SPARK-3772] Allow `ipython` to be used by Pyspark workers; IPython support improvements:
This pull request addresses a few issues related to PySpark's IPython support:
- Fix the remaining uses of the '-u' flag, which IPython doesn't support (see SPARK-3772).
- Change PYSPARK_PYTHON_OPTS to PYSPARK_DRIVER_PYTHON_OPTS, so that the old name is reserved in case we ever want to allow the worker Python options to be customized (this variable was introduced in #2554 and hasn't landed in a release yet, so this doesn't break any compatibility).
- Introduce a PYSPARK_DRIVER_PYTHON option that allows the driver to use `ipython` while the workers use a different Python version.
- Attempt to use Python 2.7 by default if PYSPARK_PYTHON is not specified.
- Retain the old semantics for IPYTHON=1 and IPYTHON_OPTS (to avoid breaking existing example programs).
There are more details in a block comment in `bin/pyspark`.
Author: Josh Rosen <joshrosen@apache.org>
Closes #2651 from JoshRosen/SPARK-3772 and squashes the following commits:
7b8eb86 [Josh Rosen] More changes to PySpark python executable configuration:
c4f5778 [Josh Rosen] [SPARK-3772] Allow ipython to be used by Pyspark workers; IPython fixes:
Diffstat (limited to 'docs')
-rw-r--r-- | docs/programming-guide.md | 8 |
1 files changed, 4 insertions, 4 deletions
diff --git a/docs/programming-guide.md b/docs/programming-guide.md index 8e8cc1dd98..18420afb27 100644 --- a/docs/programming-guide.md +++ b/docs/programming-guide.md @@ -211,17 +211,17 @@ For a complete list of options, run `pyspark --help`. Behind the scenes, It is also possible to launch the PySpark shell in [IPython](http://ipython.org), the enhanced Python interpreter. PySpark works with IPython 1.0.0 and later. To -use IPython, set the `PYSPARK_PYTHON` variable to `ipython` when running `bin/pyspark`: +use IPython, set the `PYSPARK_DRIVER_PYTHON` variable to `ipython` when running `bin/pyspark`: {% highlight bash %} -$ PYSPARK_PYTHON=ipython ./bin/pyspark +$ PYSPARK_DRIVER_PYTHON=ipython ./bin/pyspark {% endhighlight %} -You can customize the `ipython` command by setting `PYSPARK_PYTHON_OPTS`. For example, to launch +You can customize the `ipython` command by setting `PYSPARK_DRIVER_PYTHON_OPTS`. For example, to launch the [IPython Notebook](http://ipython.org/notebook.html) with PyLab plot support: {% highlight bash %} -$ PYSPARK_PYTHON=ipython PYSPARK_PYTHON_OPTS="notebook --pylab inline" ./bin/pyspark +$ PYSPARK_DRIVER_PYTHON=ipython PYSPARK_DRIVER_PYTHON_OPTS="notebook --pylab inline" ./bin/pyspark {% endhighlight %} </div> |