aboutsummaryrefslogtreecommitdiff
path: root/docs
Commit message (Collapse)AuthorAgeFilesLines
* Merge pull request #97 from ewencp/pyspark-system-propertiesMatei Zaharia2013-10-221-0/+11
|\ | | | | | | | | | | | | | | | | | | | | Add classmethod to SparkContext to set system properties. Add a new classmethod to SparkContext to set system properties like is possible in Scala/Java. Unlike the Java/Scala implementations, there's no access to System until the JVM bridge is created. Since SparkContext handles that, move the initialization of the JVM connection to a separate classmethod that can safely be called repeatedly as long as the same instance (or no instance) is provided.
| * Add notes to python documentation about using SparkContext.setSystemProperty.Ewen Cheslack-Postava2013-10-221-0/+11
| |
* | Docs: Fix links to RDD API documentationAaron Davidson2013-10-221-3/+3
|/
* Merge pull request #76 from pwendell/masterReynold Xin2013-10-181-1/+1
|\ | | | | | | | | | | | | Clarify compression property. Clarifies that this governs compression of internal data, not input data or output data.
| * Clarify compression property.Patrick Wendell2013-10-181-1/+1
| | | | | | | | | | Clarifies that this governs compression of internal data, not input data or output data.
* | Code styling. Updated doc.Mosharaf Chowdhury2013-10-171-0/+8
|/
* Merge remote-tracking branch 'tgravescs/sparkYarnDistCache'Matei Zaharia2013-10-101-1/+8
|\ | | | | | | | | | | | | | | Closes #11 Conflicts: docs/running-on-yarn.md yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala
| * Adding in the --addJars option to make SparkContext.addJar work on yarn and ↵tgravescs2013-10-031-0/+2
| | | | | | | | | | | | cleanup the classpaths
| * Support distributed cache files and archives on spark on yarn and attempt to ↵Y.CORP.YAHOO.COM\tgraves2013-09-231-1/+6
| | | | | | | | cleanup the staging directory on exit
* | Merge pull request #19 from aarondav/master-zkMatei Zaharia2013-10-103-4/+78
|\ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Standalone Scheduler fault tolerance using ZooKeeper This patch implements full distributed fault tolerance for standalone scheduler Masters. There is only one master Leader at a time, which is actively serving scheduling requests. If this Leader crashes, another master will eventually be elected, reconstruct the state from the first Master, and continue serving scheduling requests. Leader election is performed using the ZooKeeper leader election pattern. We try to minimize the use of ZooKeeper and the assumptions about ZooKeeper's behavior, so there is a layer of retries and session monitoring on top of the ZooKeeper client. Master failover follows directly from the single-node Master recovery via the file system (patch d5a96fe), save that the Master state is stored in ZooKeeper instead. Configuration: By default, no recovery mechanism is enabled (spark.deploy.recoveryMode = NONE). By setting spark.deploy.recoveryMode to ZOOKEEPER and setting spark.deploy.zookeeper.url to an appropriate ZooKeeper URL, ZooKeeper recovery mode is enabled. By setting spark.deploy.recoveryMode to FILESYSTEM and setting spark.deploy.recoveryDirectory to an appropriate directory accessible by the Master, we will keep the behavior of from d5a96fe. Additionally, places where a Master could be specificied by a spark:// url can now take comma-delimited lists to specify backup masters. Note that this is only used for registration of NEW Workers and application Clients. Once a Worker or Client has registered with the Master Leader, it is "in the system" and will never need to register again.
| * | Minor clarification and cleanup to spark-standalone.mdAaron Davidson2013-10-101-10/+33
| | |
| * | Address Matei's comments on documentationAaron Davidson2013-10-101-14/+21
| | | | | | | | | | | | Updates to the documentation and changing some logError()s to logWarning()s.
| * | Add docs for standalone scheduler fault toleranceAaron Davidson2013-10-083-4/+48
| | | | | | | | | | | | Also fix a couple HTML/Markdown issues in other files.
* | | Fix PySpark docs and an overly long line of code after fdbae41eMatei Zaharia2013-10-091-1/+1
| | |
* | | Merge branch 'master' into implicit-alsNick Pentreath2013-10-071-2/+2
|\ \ \
| * | | Merging build changes in from 0.8Patrick Wendell2013-10-051-2/+2
| | | |
* | | | Adding implicit feedback ALS to MLlib user guideNick Pentreath2013-10-041-4/+20
|/ / /
* / / Allow users to set the application name for Spark on Yarntgravescs2013-10-021-0/+1
|/ /
* / Update build version in masterPatrick Wendell2013-09-241-2/+2
|/
* Fix typo in Maven build docsJey Kottalam2013-09-151-2/+2
|
* Merge pull request #932 from pwendell/mesos-versionPatrick Wendell2013-09-151-1/+1
|\ | | | | Bumping Mesos version to 0.13.0
| * Bumping Mesos version to 0.13.0Patrick Wendell2013-09-151-1/+1
| |
* | Explain yarn.version in Maven build docsPatrick Wendell2013-09-151-3/+3
|/
* More updates to Spark on Mesos documentation.Benjamin Hindman2013-09-111-2/+2
|
* Updated Spark on Mesos documentation.Benjamin Hindman2013-09-111-17/+16
|
* Change port from 3030 to 4040Patrick Wendell2013-09-114-8/+8
|
* Update Python API featuresMatei Zaharia2013-09-101-1/+1
|
* Document fortran dependency for MLBasePatrick Wendell2013-09-091-0/+7
|
* Small tweaks to MLlib docsMatei Zaharia2013-09-081-10/+8
|
* Merge pull request #905 from mateiz/docs2Matei Zaharia2013-09-0817-138/+472
|\ | | | | Job scheduling and cluster mode docs
| * Fix some review commentsMatei Zaharia2013-09-082-2/+2
| |
| * Updated cluster diagram to show cachesMatei Zaharia2013-09-082-0/+0
| |
| * Review commentsMatei Zaharia2013-09-082-1/+48
| |
| * Some tweaks to CDH/HDP docMatei Zaharia2013-09-081-10/+52
| |
| * Added cluster overview doc, made logo higher-resolution, and added moreMatei Zaharia2013-09-087-15/+88
| | | | | | | | details on monitoring
| * More fair scheduler docs and property names.Matei Zaharia2013-09-087-76/+164
| | | | | | | | | | Also changed uses of "job" terminology to "application" when they referred to an entire Spark program, to avoid confusion.
| * Work in progress:Matei Zaharia2013-09-086-44/+128
| | | | | | | | | | | | | | - Add job scheduling docs - Rename some fair scheduler properties - Organize intro page better - Link to Apache wiki for "contributing to Spark"
* | respose to PR commentsAmeet Talwalkar2013-09-081-25/+30
| |
* | Merge remote-tracking branch 'upstream/master'Ameet Talwalkar2013-09-086-13/+179
|\ \
| * \ Merge pull request #906 from pwendell/ganglia-sinkPatrick Wendell2013-09-081-0/+9
| |\ \ | | |/ | |/| Clean-up of Metrics Code/Docs and Add Ganglia Sink
| | * Adding more docs and some code cleanupPatrick Wendell2013-09-081-0/+9
| | |
| * | Merge pull request #900 from pwendell/cdh-docsMatei Zaharia2013-09-082-0/+77
| |\ \ | | | | | | | | Provide docs to describe running on CDH/HDP cluster.
| | * | File renamePatrick Wendell2013-09-072-4/+2
| | | |
| | * | Changes based on feedbackPatrick Wendell2013-09-071-12/+24
| | | |
| | * | Provide docs to describe running on CDH/HDP cluster.Patrick Wendell2013-09-062-0/+67
| | | | | | | | | | | | | | | | This doc consolidates information relevant to CDH/HDP users in a single place.
| * | | Merge pull request #901 from ooyala/2013-09/0.8-doc-changesMatei Zaharia2013-09-072-4/+21
| |\ \ \ | | | | | | | | | | 0.8 Doc changes for make-distribution.sh
| | * | | CR feedback from MateiEvan Chan2013-09-072-7/+1
| | | | |
| | * | | Add references to make-distribution.shEvan Chan2013-09-062-0/+19
| | | | |
| | * | | "launch" scripts is more accurate terminologyEvan Chan2013-09-061-2/+2
| | | | |
| | * | | Easier way to start the masterEvan Chan2013-09-061-1/+1
| | | | |
generated by cgit v1.2.3 (git 2.39.1) at 2024-09-09 04:17:15 +0000