diff options
| author | Holden Karau <holden@us.ibm.com> | 2016-05-10 10:29:38 -0700 |
|---|---|---|
| committer | Marcelo Vanzin <vanzin@cloudera.com> | 2016-05-10 10:29:38 -0700 |
| commit | 488863d87340e0fb68e81ba0be6df7d7c9723992 (patch) | |
| tree | 91e6853d3ab612c0da411019f902de1068a3ea66 /docs/building-spark.md | |
| parent | 2646265368aab0f0b800d3052e557dea7c40c2d6 (diff) | |
| download | spark-488863d87340e0fb68e81ba0be6df7d7c9723992.tar.gz spark-488863d87340e0fb68e81ba0be6df7d7c9723992.tar.bz2 spark-488863d87340e0fb68e81ba0be6df7d7c9723992.zip | |
[SPARK-13382][DOCS][PYSPARK] Update pyspark testing notes in build docs
## What changes were proposed in this pull request?
The current build documents don't specify that for PySpark tests we need to include Hive in the assembly otherwise the ORC tests fail.
## How was the this patch tested?
Manually built the docs locally. Ran the provided build command follow by the PySpark SQL tests.
Author: Holden Karau <holden@us.ibm.com>
Closes #11278 from holdenk/SPARK-13382-update-pyspark-testing-notes-r2.
Diffstat (limited to 'docs/building-spark.md')
| -rw-r--r-- | docs/building-spark.md | 15 |
1 files changed, 15 insertions, 0 deletions
diff --git a/docs/building-spark.md b/docs/building-spark.md index 13c95e4fcb..63532c742e 100644 --- a/docs/building-spark.md +++ b/docs/building-spark.md @@ -123,6 +123,21 @@ To produce a Spark package compiled with Scala 2.10, use the `-Dscala-2.10` prop ./dev/change-scala-version.sh 2.10 mvn -Pyarn -Phadoop-2.4 -Dscala-2.10 -DskipTests clean package +# PySpark Tests with Maven + +If you are building PySpark and wish to run the PySpark tests you will need to build Spark with hive support. + +{% highlight bash %} +build/mvn -DskipTests clean package -Phive +./python/run-tests +{% endhighlight %} + +The run-tests script also can be limited to a specific Python version or a specific module + + ./python/run-tests --python-executables=python --modules=pyspark-sql + +**Note:** You can also run Python tests with an sbt build, provided you build Spark with hive support. + # Spark Tests in Maven Tests are run by default via the [ScalaTest Maven plugin](http://www.scalatest.org/user_guide/using_the_scalatest_maven_plugin). |