diff options
| author | Jeff Zhang <zjffdu@apache.org> | 2016-08-11 20:08:25 -0700 |
|---|---|---|
| committer | Marcelo Vanzin <vanzin@cloudera.com> | 2016-08-11 20:08:39 -0700 |
| commit | 7a9e25c38380e6c62080d62ad38a4830e44fe753 (patch) | |
| tree | c7ee1437a3dec8abc0fef57b10406a5bd0b72987 /docs | |
| parent | ea0bf91b4a2ca3ef472906e50e31fd6268b6f53e (diff) | |
| download | spark-7a9e25c38380e6c62080d62ad38a4830e44fe753.tar.gz spark-7a9e25c38380e6c62080d62ad38a4830e44fe753.tar.bz2 spark-7a9e25c38380e6c62080d62ad38a4830e44fe753.zip | |
[SPARK-13081][PYSPARK][SPARK_SUBMIT] Allow set pythonExec of driver and executor through conf…
Before this PR, user have to export environment variable to specify the python of driver & executor which is not so convenient for users. This PR is trying to allow user to specify python through configuration "--pyspark-driver-python" & "--pyspark-executor-python"
Manually test in local & yarn mode for pyspark-shell and pyspark batch mode.
Author: Jeff Zhang <zjffdu@apache.org>
Closes #13146 from zjffdu/SPARK-13081.
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/configuration.md | 21 |
1 files changed, 19 insertions, 2 deletions
diff --git a/docs/configuration.md b/docs/configuration.md index e33094b062..ae753189b5 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -427,6 +427,21 @@ Apart from these, the following properties are also available, and may be useful with <code>spark.jars.packages</code>. </td> </tr> +<tr> + <td><code>spark.pyspark.driver.python</code></td> + <td></td> + <td> + Python binary executable to use for PySpark in driver. + (default is <code>spark.pyspark.python</code>) + </td> +</tr> +<tr> + <td><code>spark.pyspark.python</code></td> + <td></td> + <td> + Python binary executable to use for PySpark in both driver and executors. + </td> +</tr> </table> #### Shuffle Behavior @@ -1786,11 +1801,13 @@ The following variables can be set in `spark-env.sh`: </tr> <tr> <td><code>PYSPARK_PYTHON</code></td> - <td>Python binary executable to use for PySpark in both driver and workers (default is <code>python2.7</code> if available, otherwise <code>python</code>).</td> + <td>Python binary executable to use for PySpark in both driver and workers (default is <code>python2.7</code> if available, otherwise <code>python</code>). + Property <code>spark.pyspark.python</code> take precedence if it is set</td> </tr> <tr> <td><code>PYSPARK_DRIVER_PYTHON</code></td> - <td>Python binary executable to use for PySpark in driver only (default is <code>PYSPARK_PYTHON</code>).</td> + <td>Python binary executable to use for PySpark in driver only (default is <code>PYSPARK_PYTHON</code>). + Property <code>spark.pyspark.driver.python</code> take precedence if it is set</td> </tr> <tr> <td><code>SPARKR_DRIVER_R</code></td> |