diff options
author | Patrick Wendell <pwendell@apache.org> | 2015-04-17 05:52:53 +0000 |
---|---|---|
committer | Patrick Wendell <pwendell@apache.org> | 2015-04-17 05:52:53 +0000 |
commit | d49935af658ee525496c7ac598f7352c7793d6f8 (patch) | |
tree | 1813e84df9974a07d595ba24f0fe0e832f182f05 /site/docs/1.3.1/README.md | |
parent | 9695c4519404b689f29bc0fbc1a2eefa3eb33806 (diff) | |
download | spark-website-d49935af658ee525496c7ac598f7352c7793d6f8.tar.gz spark-website-d49935af658ee525496c7ac598f7352c7793d6f8.tar.bz2 spark-website-d49935af658ee525496c7ac598f7352c7793d6f8.zip |
Adding docs for 1.2.2 and 1.3.1
Diffstat (limited to 'site/docs/1.3.1/README.md')
-rw-r--r-- | site/docs/1.3.1/README.md | 76 |
1 files changed, 76 insertions, 0 deletions
diff --git a/site/docs/1.3.1/README.md b/site/docs/1.3.1/README.md new file mode 100644 index 000000000..8a54724c4 --- /dev/null +++ b/site/docs/1.3.1/README.md @@ -0,0 +1,76 @@ +Welcome to the Spark documentation! + +This readme will walk you through navigating and building the Spark documentation, which is included +here with the Spark source code. You can also find documentation specific to release versions of +Spark at http://spark.apache.org/documentation.html. + +Read on to learn more about viewing documentation in plain text (i.e., markdown) or building the +documentation yourself. Why build it yourself? So that you have the docs that corresponds to +whichever version of Spark you currently have checked out of revision control. + +## Generating the Documentation HTML + +We include the Spark documentation as part of the source (as opposed to using a hosted wiki, such as +the github wiki, as the definitive documentation) to enable the documentation to evolve along with +the source code and be captured by revision control (currently git). This way the code automatically +includes the version of the documentation that is relevant regardless of which version or release +you have checked out or downloaded. + +In this directory you will find textfiles formatted using Markdown, with an ".md" suffix. You can +read those text files directly if you want. Start with index.md. + +The markdown code can be compiled to HTML using the [Jekyll tool](http://jekyllrb.com). +`Jekyll` and a few dependencies must be installed for this to work. We recommend +installing via the Ruby Gem dependency manager. Since the exact HTML output +varies between versions of Jekyll and its dependencies, we list specific versions here +in some cases: + + $ sudo gem install jekyll + $ sudo gem install jekyll-redirect-from + +Execute `jekyll` from the `docs/` directory. Compiling the site with Jekyll will create a directory +called `_site` containing index.html as well as the rest of the compiled files. + +You can modify the default Jekyll build as follows: + + # Skip generating API docs (which takes a while) + $ SKIP_API=1 jekyll build + # Serve content locally on port 4000 + $ jekyll serve --watch + # Build the site with extra features used on the live page + $ PRODUCTION=1 jekyll build + +## Pygments + +We also use pygments (http://pygments.org) for syntax highlighting in documentation markdown pages, +so you will also need to install that (it requires Python) by running `sudo pip install Pygments`. + +To mark a block of code in your markdown to be syntax highlighted by jekyll during the compile +phase, use the following sytax: + + {% highlight scala %} + // Your scala code goes here, you can replace scala with many other + // supported languages too. + {% endhighlight %} + +## Sphinx + +We use Sphinx to generate Python API docs, so you will need to install it by running +`sudo pip install sphinx`. + +## API Docs (Scaladoc and Sphinx) + +You can build just the Spark scaladoc by running `build/sbt doc` from the SPARK_PROJECT_ROOT directory. + +Similarly, you can build just the PySpark docs by running `make html` from the +SPARK_PROJECT_ROOT/python/docs directory. Documentation is only generated for classes that are listed as +public in `__init__.py`. + +When you run `jekyll` in the `docs` directory, it will also copy over the scaladoc for the various +Spark subprojects into the `docs` directory (and then also into the `_site` directory). We use a +jekyll plugin to run `build/sbt doc` before building the site so if you haven't run it (recently) it +may take some time as it generates all of the scaladoc. The jekyll plugin also generates the +PySpark docs [Sphinx](http://sphinx-doc.org/). + +NOTE: To skip the step of building and copying over the Scala and Python API docs, run `SKIP_API=1 +jekyll`. |