diff options
author | Josh Rosen <joshrosen@eecs.berkeley.edu> | 2012-12-27 22:47:37 -0800 |
---|---|---|
committer | Josh Rosen <joshrosen@eecs.berkeley.edu> | 2012-12-27 22:47:37 -0800 |
commit | 665466dfff4f89196627a0777eabd3d3894cd296 (patch) | |
tree | 7fa580209756c5fdbb0a52930f30959bbbbc2ba3 /pyspark/README | |
parent | ac32447cd38beac8f6bc7a90be9fd24666bb46ad (diff) | |
download | spark-665466dfff4f89196627a0777eabd3d3894cd296.tar.gz spark-665466dfff4f89196627a0777eabd3d3894cd296.tar.bz2 spark-665466dfff4f89196627a0777eabd3d3894cd296.zip |
Simplify PySpark installation.
- Bundle Py4J binaries, since it's hard to install
- Uses Spark's `run` script to launch the Py4J
gateway, inheriting the settings in spark-env.sh
With these changes, (hopefully) nothing more than
running `sbt/sbt package` will be necessary to run
PySpark.
Diffstat (limited to 'pyspark/README')
-rw-r--r-- | pyspark/README | 23 |
1 files changed, 2 insertions, 21 deletions
diff --git a/pyspark/README b/pyspark/README index 461176de7d..d8d521c72c 100644 --- a/pyspark/README +++ b/pyspark/README @@ -32,30 +32,11 @@ The `pyspark/pyspark/examples` directory contains a few complete examples. ## Installing PySpark - -PySpark requires a development version of Py4J, a Python library for -interacting with Java processes. It can be installed from -https://github.com/bartdag/py4j; make sure to install a version that -contains at least the commits through b7924aabe9. - -PySpark requires the `argparse` module, which is included in Python 2.7 -and is is available for Python 2.6 through `pip` or `easy_install`. - -PySpark uses the `PYTHONPATH` environment variable to search for Python -classes; Py4J should be on this path, along with any libraries used by -PySpark programs. `PYTHONPATH` will be automatically shipped to worker -machines, but the files that it points to must be present on each -machine. - -PySpark requires the Spark assembly JAR, which can be created by running -`sbt/sbt assembly` in the Spark directory. - -Additionally, `SPARK_HOME` should be set to the location of the Spark +# +To use PySpark, `SPARK_HOME` should be set to the location of the Spark package. ## Running PySpark The easiest way to run PySpark is to use the `run-pyspark` and `pyspark-shell` scripts, which are included in the `pyspark` directory. -These scripts automatically load the `spark-conf.sh` file, set -`SPARK_HOME`, and add the `pyspark` package to the `PYTHONPATH`. |