aboutsummaryrefslogtreecommitdiff
Commit message (Collapse)AuthorAgeFilesLines
* Fixing SPARK-602: PythonPartitionerAndre Schumacher2013-10-046-10/+44
| | | | | | | Currently PythonPartitioner determines partition ID by hashing a byte-array representation of PySpark's key. This PR lets PythonPartitioner use the actual partition ID, which is required e.g. for sorting via PySpark.
* Merge pull request #26 from Du-Li/masterMatei Zaharia2013-10-032-1/+4
|\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | fixed a wildcard bug in make-distribution.sh; ask sbt to check local maven repo in project/SparkBuild.scala (1) fixed a wildcard bug in make-distribution.sh: with the wildcard * in quotes, this cp command failed. it worked after moving the wildcard out quotes. (2) ask sbt to check local maven repo in SparkBuild.scala: To build Spark (0.9.0-SNAPSHOT) with the HEAD of mesos (0.15.0), I must do "make maven-install" under mesos/build, which publishes the java .jar file under ~/.m2. However, when building Spark (after pointing mesos to version 0.15.0), sbt uses ivy which by default only checks ~/.ivy2. This change is to tell sbt to also check ~/.m2.
| * ask ivy/sbt to check local maven repo under ~/.m2Du Li2013-10-011-0/+3
| |
| * fixed a bug of using wildcard in quotesDu Li2013-10-011-1/+1
| |
* | Merge pull request #25 from CruncherBigData/masterMatei Zaharia2013-10-031-1/+1
|\ \ | | | | | | | | | Update README: updated the link
| * | Update READMECruncherBigData2013-10-011-1/+1
| |/
* | Merge pull request #28 from tgravescs/sparYarnAppNameMatei Zaharia2013-10-033-1/+8
|\ \ | | | | | | | | | Allow users to set the application name for Spark on Yarn
| * | Add default value to usage statementtgravescs2013-10-031-1/+1
| | |
| * | Allow users to set the application name for Spark on Yarntgravescs2013-10-023-1/+8
| |/
* | Merge pull request #10 from kayousterhout/results_through-bmMatei Zaharia2013-10-0219-167/+496
|\ \ | |/ |/| | | | | | | | | | | | | | | | | | | | | | | Send Task results through the block manager when larger than Akka frame size (fixes SPARK-669). This change requires adding an extra failure mode: tasks can complete successfully, but the result gets lost or flushed from the block manager before it's been fetched. This change also moves the deserialization of tasks into a separate thread, so it's no longer part of the DAG scheduler's tight loop. This should improve scheduler throughput, particularly when tasks are sending back large results. Thanks Josh for writing the original version of this patch! This is duplicated from the mesos/spark repo: https://github.com/mesos/spark/pull/835
| * Added additional unit test for repeated task failuresKay Ousterhout2013-09-301-1/+28
| |
| * Fixed compilation errors and broken test.Kay Ousterhout2013-09-304-13/+11
| |
| * Merge remote-tracking branch 'upstream/master' into results_through-bmKay Ousterhout2013-09-3060-195/+368
| |\ | |/ |/| | | | | | | | | Conflicts: core/src/main/scala/org/apache/spark/scheduler/cluster/ClusterScheduler.scala core/src/main/scala/org/apache/spark/scheduler/cluster/ClusterTaskSetManager.scala core/src/main/scala/org/apache/spark/scheduler/local/LocalTaskSetManager.scala
* | Merge pull request #17 from rxin/optimizeReynold Xin2013-09-262-2/+1
|\ \ | | | | | | | | | Remove -optimize flag
| * | Removed scala -optimize flag.Reynold Xin2013-09-262-2/+1
| | |
* | | Merge pull request #16 from pwendell/masterReynold Xin2013-09-261-1/+1
|\ \ \ | | | | | | | | | | | | Bug fix in master build
| * | | Bug fix in master buildPatrick Wendell2013-09-261-1/+1
| | | |
* | | | Merge pull request #14 from kayousterhout/untangle_schedulerReynold Xin2013-09-2634-71/+62
|\ \ \ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Improved organization of scheduling packages. This commit does not change any code -- only file organization. Please let me know if there was some masterminded strategy behind the existing organization that I failed to understand! There are two components of this change: (1) Moving files out of the cluster package, and down a level to the scheduling package. These files are all used by the local scheduler in addition to the cluster scheduler(s), so should not be in the cluster package. As a result of this change, none of the files in the local package reference files in the cluster package. (2) Moving the mesos package to within the cluster package. The mesos scheduling code is for a cluster, and represents a specific case of cluster scheduling (the Mesos-related classes often subclass cluster scheduling classes). Thus, the most logical place for it seems to be within the cluster package. The one thing about the scheduling code that seems a little funny to me is the naming of the SchedulerBackends. The StandaloneSchedulerBackend is not just for Standalone mode, but instead is used by Mesos coarse grained mode and Yarn, and the backend that *is* just for Standalone mode is instead called SparkDeploySchedulerBackend. I didn't change this because I wasn't sure if there was a reason for this naming that I'm just not aware of.
| * | | | Improved organization of scheduling packages.Kay Ousterhout2013-09-2534-71/+62
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This commit does not change any code -- only file organization. There are two components of this change: (1) Moving files out of the cluster package, and down a level to the scheduling package. These files are all used by the local scheduler in addition to the cluster scheduler(s), so should not be in the cluster package. As a result of this change, none of the files in the local package reference files in the cluster package. (2) Moving the mesos package to within the cluster package. The mesos scheduling code is for a cluster, and represents a specific case of cluster scheduling (the Mesos-related classes often subclass cluster scheduling classes). Thus, the most logical place for it is within the cluster package.
* | | | | Merge pull request #670 from jey/ec2-ssh-improvementsReynold Xin2013-09-261-26/+80
|\ \ \ \ \ | |_|_|/ / |/| | | | EC2 SSH improvements
| * | | | Clarify error messages on SSH failureJey Kottalam2013-09-111-6/+21
| | | | |
| * | | | Generate new SSH key for the cluster, make "--identity-file" optionalJey Kottalam2013-09-061-21/+37
| | | | |
| * | | | Construct shell commands as sequences for safety and composabilityJey Kottalam2013-09-061-11/+34
| | | | |
* | | | | Merge pull request #930 from holdenk/masterReynold Xin2013-09-262-1/+11
|\ \ \ \ \ | | | | | | | | | | | | Add mapPartitionsWithIndex
| * | | | | Fix formatting :)Holden Karau2013-09-231-4/+5
| | | | | |
| * | | | | Switch indent from 2 to 4 spacesHolden Karau2013-09-221-2/+2
| | | | | |
| * | | | | Fix build on ubuntuHolden Karau2013-09-141-1/+1
| | | | | |
| * | | | | Merge branch 'master' of https://github.com/mesos/sparkHolden Karau2013-09-142-5/+12
| |\ \ \ \ \
| * | | | | | Make mapPartitionsWithIndex work with JavaRDD'sHolden Karau2013-09-141-2/+3
| | | | | | |
| * | | | | | Start of working on SPARK-615Holden Karau2013-09-111-0/+8
| | | | | | |
* | | | | | | Merge pull request #7 from wannabeast/memorystore-fixesReynold Xin2013-09-261-6/+8
|\ \ \ \ \ \ \ | |_|_|_|_|/ / |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | some minor fixes to MemoryStore This is a repeat of #5, moved to its own branch in my repo. This makes all updates to on ; it skips on synchronizing the reads where it can get away with it.
| * | | | | | Synchronize on "entries" the remaining update to "currentMemory".Mike2013-09-191-3/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Make "currentMemory" @volatile, so that it's reads in ensureFreeSpace() are atomic and up-to-date--i.e., currentMemory can't increase while putLock is held (though it could decrease, which would only help ensureFreeSpace()).
| * | | | | | Set currentMemory to 0 in clear().Mike2013-09-111-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Remove unnecessary entries.get() call.
| * | | | | | Remove MemoryStore$Entry.dropPending, unused as of 42e0a68082.Mike2013-09-101-1/+1
| | | | | | |
* | | | | | | Merge pull request #9 from rxin/limitPatrick Wendell2013-09-262-10/+66
|\ \ \ \ \ \ \ | | | | | | | | | | | | | | | | | | | | | | | | Smarter take/limit implementation.
| * | | | | | | Smarter take/limit implementation.Reynold Xin2013-09-202-10/+66
| | | | | | | |
* | | | | | | | Merge remote-tracking branch 'apache-github/pr/13' into HEADPatrick Wendell2013-09-2414-15/+15
|\ \ \ \ \ \ \ \ | |_|_|_|_|_|/ / |/| | | | | | |
| * | | | | | | Update build version in masterPatrick Wendell2013-09-2414-15/+15
|/ / / / / / /
* | | | | | | Merge remote-tracking branch 'pr/12'Reynold Xin2013-09-232-4/+6
|\ \ \ \ \ \ \ | | | | | | | | | | | | | | | | | | | | | | | | Fix spacing so java.io.tmpdir doesn't run on with SPARK_JAVA_OPTS
| * | | | | | | Fix spacing so that the java.io.tmpdir doesn't run on with SPARK_JAVA_OPTSY.CORP.YAHOO.COM\tgraves2013-09-232-4/+6
| |/ / / / / /
* | | | | | | Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/incubator-sparkReynold Xin2013-09-230-0/+0
|\| | | | | |
* | | | | | | Merge branch 'master' of github.com:markhamstra/incubator-sparkReynold Xin2013-09-231-1/+0
|\ \ \ \ \ \ \
| * | | | | | | Removed repetative import; fixes hidden definition compiler warning.Mark Hamstra2013-09-031-1/+0
| | |/ / / / / | |/| | | | |
* | | | | | | Merge branch 'master' of github.com:mesos/sparkReynold Xin2013-09-237-64/+123
|\ \ \ \ \ \ \
| * \ \ \ \ \ \ Merge pull request #928 from jerryshao/fairscheduler-refactorReynold Xin2013-09-221-43/+56
| |\ \ \ \ \ \ \ | | | | | | | | | | | | | | | | | | Refactor FairSchedulableBuilder
| | * | | | | | | Change Exception to NoSuchElementException and minor style fixjerryshao2013-09-221-6/+7
| | | | | | | | |
| | * | | | | | | Remove infix style and othersjerryshao2013-09-221-10/+8
| | | | | | | | |
| | * | | | | | | Refactor FairSchedulableBuilder:jerryshao2013-09-221-39/+53
| |/ / / / / / / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 1. Configuration can be read from classpath if not set explicitly. 2. Add missing close handler.
| * | | | | | | Merge pull request #937 from jerryshao/localProperties-fixReynold Xin2013-09-212-2/+50
| |\ \ \ \ \ \ \ | | | | | | | | | | | | | | | | | | Fix PR926 local properties issues in Spark Streaming like scenarios
| | * | | | | | | Add barrier for local properties unit test and fix some stylesjerryshao2013-09-222-3/+11
| | | | | | | | |
generated by cgit v1.2.3 (git 2.39.1) at 2024-08-06 21:24:50 +0000