diff options
| author | Josh Rosen <joshrosen@apache.org> | 2014-10-09 16:08:07 -0700 |
|---|---|---|
| committer | Josh Rosen <joshrosen@apache.org> | 2014-10-09 16:08:07 -0700 |
| commit | 4e9b551a0b807f5a2cc6679165c8be4e88a3d077 (patch) | |
| tree | d7d6be00b5dbb04d38eab034c9dc18e73b5002de /sql | |
| parent | ac302052870a650d56f2d3131c27755bb2960ad7 (diff) | |
| download | spark-4e9b551a0b807f5a2cc6679165c8be4e88a3d077.tar.gz spark-4e9b551a0b807f5a2cc6679165c8be4e88a3d077.tar.bz2 spark-4e9b551a0b807f5a2cc6679165c8be4e88a3d077.zip | |
[SPARK-3772] Allow `ipython` to be used by Pyspark workers; IPython support improvements:
This pull request addresses a few issues related to PySpark's IPython support:
- Fix the remaining uses of the '-u' flag, which IPython doesn't support (see SPARK-3772).
- Change PYSPARK_PYTHON_OPTS to PYSPARK_DRIVER_PYTHON_OPTS, so that the old name is reserved in case we ever want to allow the worker Python options to be customized (this variable was introduced in #2554 and hasn't landed in a release yet, so this doesn't break any compatibility).
- Introduce a PYSPARK_DRIVER_PYTHON option that allows the driver to use `ipython` while the workers use a different Python version.
- Attempt to use Python 2.7 by default if PYSPARK_PYTHON is not specified.
- Retain the old semantics for IPYTHON=1 and IPYTHON_OPTS (to avoid breaking existing example programs).
There are more details in a block comment in `bin/pyspark`.
Author: Josh Rosen <joshrosen@apache.org>
Closes #2651 from JoshRosen/SPARK-3772 and squashes the following commits:
7b8eb86 [Josh Rosen] More changes to PySpark python executable configuration:
c4f5778 [Josh Rosen] [SPARK-3772] Allow ipython to be used by Pyspark workers; IPython fixes:
Diffstat (limited to 'sql')
0 files changed, 0 insertions, 0 deletions