aboutsummaryrefslogtreecommitdiff
path: root/docs/quick-start.md
Commit message (Collapse)AuthorAgeFilesLines
* [SPARK-1439, SPARK-1440] Generate unified Scaladoc across projects and JavadocsMatei Zaharia2014-04-211-2/+4
| | | | | | | | | | | | | | | | | | | | | | I used the sbt-unidoc plugin (https://github.com/sbt/sbt-unidoc) to create a unified Scaladoc of our public packages, and generate Javadocs as well. One limitation is that I haven't found an easy way to exclude packages in the Javadoc; there is a SBT task that identifies Java sources to run javadoc on, but it's been very difficult to modify it from outside to change what is set in the unidoc package. Some SBT-savvy people should help with this. The Javadoc site also lacks package-level descriptions and things like that, so we may want to look into that. We may decide not to post these right now if it's too limited compared to the Scala one. Example of the built doc site: http://people.csail.mit.edu/matei/spark-unified-docs/ Author: Matei Zaharia <matei@databricks.com> This patch had conflicts when merged, resolved by Committer: Patrick Wendell <pwendell@gmail.com> Closes #457 from mateiz/better-docs and squashes the following commits: a63d4a3 [Matei Zaharia] Skip Java/Scala API docs for Python package 5ea1f43 [Matei Zaharia] Fix links to Java classes in Java guide, fix some JS for scrolling to anchors on page load f05abc0 [Matei Zaharia] Don't include java.lang package names 995e992 [Matei Zaharia] Skip internal packages and class names with $ in JavaDoc a14a93c [Matei Zaharia] typo 76ce64d [Matei Zaharia] Add groups to Javadoc index page, and a first package-info.java ed6f994 [Matei Zaharia] Generate JavaDoc as well, add titles, update doc site to use unified docs acb993d [Matei Zaharia] Add Unidoc plugin for the projects we want Unidoced
* Clean up and simplify Spark configurationPatrick Wendell2014-04-211-95/+83
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Over time as we've added more deployment modes, this have gotten a bit unwieldy with user-facing configuration options in Spark. Going forward we'll advise all users to run `spark-submit` to launch applications. This is a WIP patch but it makes the following improvements: 1. Improved `spark-env.sh.template` which was missing a lot of things users now set in that file. 2. Removes the shipping of SPARK_CLASSPATH, SPARK_JAVA_OPTS, and SPARK_LIBRARY_PATH to the executors on the cluster. This was an ugly hack. Instead it introduces config variables spark.executor.extraJavaOpts, spark.executor.extraLibraryPath, and spark.executor.extraClassPath. 3. Adds ability to set these same variables for the driver using `spark-submit`. 4. Allows you to load system properties from a `spark-defaults.conf` file when running `spark-submit`. This will allow setting both SparkConf options and other system properties utilized by `spark-submit`. 5. Made `SPARK_LOCAL_IP` an environment variable rather than a SparkConf property. This is more consistent with it being set on each node. Author: Patrick Wendell <pwendell@gmail.com> Closes #299 from pwendell/config-cleanup and squashes the following commits: 127f301 [Patrick Wendell] Improvements to testing a006464 [Patrick Wendell] Moving properties file template. b4b496c [Patrick Wendell] spark-defaults.properties -> spark-defaults.conf 0086939 [Patrick Wendell] Minor style fixes af09e3e [Patrick Wendell] Mention config file in docs and clean-up docs b16e6a2 [Patrick Wendell] Cleanup of spark-submit script and Scala quick start guide af0adf7 [Patrick Wendell] Automatically add user jar a56b125 [Patrick Wendell] Responses to Tom's review d50c388 [Patrick Wendell] Merge remote-tracking branch 'apache/master' into config-cleanup a762901 [Patrick Wendell] Fixing test failures ffa00fe [Patrick Wendell] Review feedback fda0301 [Patrick Wendell] Note 308f1f6 [Patrick Wendell] Properly escape quotes and other clean-up for YARN e83cd8f [Patrick Wendell] Changes to allow re-use of test applications be42f35 [Patrick Wendell] Handle case where SPARK_HOME is not set c2a2909 [Patrick Wendell] Test compile fixes 4ee6f9d [Patrick Wendell] Making YARN doc changes consistent afc9ed8 [Patrick Wendell] Cleaning up line limits and two compile errors. b08893b [Patrick Wendell] Additional improvements. ace4ead [Patrick Wendell] Responses to review feedback. b72d183 [Patrick Wendell] Review feedback for spark env file 46555c1 [Patrick Wendell] Review feedback and import clean-ups 437aed1 [Patrick Wendell] Small fix 761ebcd [Patrick Wendell] Library path and classpath for drivers 7cc70e4 [Patrick Wendell] Clean up terminology inside of spark-env script 5b0ba8e [Patrick Wendell] Don't ship executor envs 84cc5e5 [Patrick Wendell] Small clean-up 1f75238 [Patrick Wendell] SPARK_JAVA_OPTS --> SPARK_MASTER_OPTS for master settings 4982331 [Patrick Wendell] Remove SPARK_LIBRARY_PATH 6eaf7d0 [Patrick Wendell] executorJavaOpts 0faa3b6 [Patrick Wendell] Stash of adding config options in submit script and YARN ac2d65e [Patrick Wendell] Change spark.local.dir -> SPARK_LOCAL_DIRS
* small fix ( proogram -> program )Prabeesh K2014-04-041-1/+1
| | | | | | | | Author: Prabeesh K <prabsmails@gmail.com> Closes #331 from prabeesh/patch-3 and squashes the following commits: 9399eb5 [Prabeesh K] small fix(proogram -> program)
* [SPARK-1105] fix site scala version error in docsCodingCat2014-02-191-2/+2
| | | | | | | | | | | | | https://spark-project.atlassian.net/browse/SPARK-1105 fix site scala version error Author: CodingCat <zhunansjtu@gmail.com> Closes #618 from CodingCat/doc_version and squashes the following commits: 39bb8aa [CodingCat] more fixes 65bedb0 [CodingCat] fix site scala version error in doc
* Code review feedbackHolden Karau2014-01-051-4/+4
|
* Merge remote-tracking branch 'apache-github/master' into remove-binariesPatrick Wendell2014-01-031-4/+4
|\ | | | | | | | | | | Conflicts: core/src/test/scala/org/apache/spark/DriverSuite.scala docs/python-programming-guide.md
| * pyspark -> bin/pysparkPrashant Sharma2014-01-021-2/+2
| |
| * spark-shell -> bin/spark-shellPrashant Sharma2014-01-021-2/+2
| |
* | Merge branch 'master' into spark-1002-remove-jarsPrashant Sharma2014-01-031-14/+38
|\|
| * Updated docs for SparkConf and handled review commentsMatei Zaharia2013-12-301-14/+38
| |
* | Removed sbt folder and changed docs accordinglyPrashant Sharma2014-01-021-1/+1
|/
* Fix some review commentsMatei Zaharia2013-09-081-1/+1
|
* More fair scheduler docs and property names.Matei Zaharia2013-09-081-35/+35
| | | | | Also changed uses of "job" terminology to "application" when they referred to an entire Spark program, to avoid confusion.
* Move some classes to more appropriate packages:Matei Zaharia2013-09-011-1/+1
| | | | | | * RDD, *RDDFunctions -> org.apache.spark.rdd * Utils, ClosureCleaner, SizeEstimator -> org.apache.spark.util * JavaSerializer, KryoSerializer -> org.apache.spark.serializer
* Update docs for new packageMatei Zaharia2013-09-011-5/+5
|
* Add assmebly plug in linksPatrick Wendell2013-09-011-6/+9
|
* Better docsPatrick Wendell2013-08-311-8/+28
|
* Fixes suggested by PatrickMatei Zaharia2013-08-311-8/+2
|
* Update docs about HDFS versionsMatei Zaharia2013-08-301-1/+19
|
* Update some build instructions because only sbt assembly and mvn packageMatei Zaharia2013-08-291-1/+1
| | | | are now needed
* SPARK-739 Have quickstart standlone job use READMEPatrick Wendell2013-04-251-10/+11
|
* Update quick-start.md heading on Operations (not just Transformations).Andy Konwinski2013-04-121-2/+2
|
* Don't use deprecated Application in exampleAndrew Ash2013-03-281-8/+10
| | | | | As of 2.9.0 extending from Application is not recommended http://www.scala-lang.org/api/2.9.3/index.html#scala.Application
* Fix another broken link in quick start.Andy Konwinski2013-03-131-1/+1
|
* Fix broken link in Quick Start.Andy Konwinski2013-03-131-1/+1
|
* Switch docs to use Akka repo instead of TypesafeMatei Zaharia2013-02-251-3/+3
|
* Merge pull request #346 from JoshRosen/python-apiMatei Zaharia2013-01-121-1/+39
|\ | | | | Python API (PySpark)
| * Add `pyspark` script to replace the other scripts.Josh Rosen2013-01-011-2/+2
| | | | | | Expand the PySpark programming guide.
| * Merge remote-tracking branch 'origin/master' into python-apiJosh Rosen2012-12-291-11/+11
| |\ | | | | | | | | | | | | Conflicts: docs/quick-start.md
| * | Add documentation for Python API.Josh Rosen2012-12-281-1/+39
| | |
* | | add repositories section to simple job pom.xmlMichael Heuer2013-01-111-0/+10
| |/ |/|
* | Adding multi-jar constructor in quickstartPatrick Wendell2012-11-271-2/+2
| |
* | Fix minor typos in quick start guide.Josh Rosen2012-10-231-9/+9
|/
* Adding dependency repos in quickstart examplePatrick Wendell2012-10-141-1/+5
|
* Removing reference to publish-local in the quickstartPatrick Wendell2012-10-091-11/+4
|
* Updates to documentation:Matei Zaharia2012-10-091-49/+45
| | | | | | | | - Edited quick start and tuning guide to simplify them a little - Simplified top menu bar - Made private a SparkContext constructor parameter that was left as public - Various small fixes
* Updating lots of docs to use the new special version number variables,Andy Konwinski2012-10-081-5/+5
| | | | | also adding the version to the navbar so it is easy to tell which version of Spark these docs were compiled for.
* Merge remote-tracking branch 'public-spark/dev' into add-version-vars-to-docsAndy Konwinski2012-10-081-1/+1
|\ | | | | | | | | Conflicts: docs/quick-start.md
| * Removing one link in quickstartPatrick Wendell2012-10-081-1/+1
| |
* | Adds liquid variables to docs templating system so that they can be usedAndy Konwinski2012-10-081-7/+7
|/ | | | | | | | | throughout the docs: SPARK_VERSION, SCALA_VERSION, and MESOS_VERSION. To use them, e.g. use {{site.SPARK_VERSION}}. Also removes uses of {{HOME_PATH}} which were being resolved to "" by the templating system anyway.
* Responding to Matei's commentsPatrick Wendell2012-10-021-17/+57
|
* A Spark "Quick Start" examplePatrick Wendell2012-10-011-0/+199
This commit includes a quick start example that covers: 1) Basic usage of the Spark shell 2) A simple Spark job in Scala 3) A simple Spark job in Java
generated by cgit v1.2.3 (git 2.39.1) at 2024-08-10 05:14:26 +0000