aboutsummaryrefslogtreecommitdiff
path: root/core
Commit message (Collapse)AuthorAgeFilesLines
* Fixed bug in CheckpointRDD to prevent exception when the original RDD had ↵Tathagata Das2013-02-102-2/+12
| | | | zero splits.
* Merge pull request #450 from stephenh/inlinemergepairMatei Zaharia2013-02-051-6/+4
|\ | | | | Inline mergePair to look more like the narrow dep branch.
| * Inline mergePair to look more like the narrow dep branch.Stephen Haberman2013-02-051-6/+4
| | | | | | | | | | | | | | No functionality changes, I think this is just more consistent given mergePair isn't called multiple times/recursive. Also added a comment to explain the usual case of having two parent RDDs.
* | Merge branch 'master' into fixdeathpactexceptionStephen Haberman2013-02-0523-237/+965
|\ \ | | | | | | | | | | | | Conflicts: core/src/main/scala/spark/deploy/worker/Worker.scala
| * \ Merge pull request #449 from stephenh/longerdriversuiteMatei Zaharia2013-02-051-1/+2
| |\ \ | | | | | | | | Increase DriverSuite timeout.
| | * | Increase DriverSuite timeout.Stephen Haberman2013-02-051-1/+2
| | |/
| * | Small fix to test for distinctMatei Zaharia2013-02-041-1/+1
| | |
| * | Fix failing testMatei Zaharia2013-02-041-2/+1
| | |
| * | Merge pull request #445 from JoshRosen/pyspark_fixesMatei Zaharia2013-02-031-2/+9
| |\ \ | | | | | | | | Fix exit status in PySpark unit tests; fix/optimize PySpark's RDD.take()
| | * | Fetch fewer objects in PySpark's take() method.Josh Rosen2013-02-031-2/+9
| | |/
| * | Merge pull request #379 from stephenh/sparkmemMatei Zaharia2013-02-026-35/+17
| |\ \ | | | | | | | | Add spark.executor.memory to differentiate executor memory from spark-shell
| | * | Fix dangling old variable names.Stephen Haberman2013-02-021-2/+2
| | | |
| | * | Move executorMemory up into SchedulerBackend.Stephen Haberman2013-02-024-29/+12
| | | |
| | * | Merge branch 'master' into sparkmemStephen Haberman2013-02-02136-1920/+3738
| | |\|
| | * | Fix SPARK_MEM in ExecutorRunner.Stephen Haberman2013-01-221-1/+1
| | | |
| | * | Restore SPARK_MEM in executorEnvs.Stephen Haberman2013-01-221-2/+3
| | | |
| | * | Add spark.executor.memory to differentiate executor memory from spark-shell ↵Stephen Haberman2013-01-153-10/+8
| | | | | | | | | | | | | | | | memory.
| * | | Merge pull request #422 from squito/blockmanager_infoMatei Zaharia2013-02-026-38/+59
| |\ \ \ | | | | | | | | | | RDDInfo available from SparkContext
| | * | | remove unneeded (and unused) filter on block infoImran Rashid2013-02-011-2/+0
| | | | |
| | * | | track total partitions, in addition to cached partitions; use scala string ↵Imran Rashid2013-02-013-9/+13
| | | | | | | | | | | | | | | | | | | | formatting
| | * | | fixup merge (master -> driver renaming)Imran Rashid2013-02-011-1/+1
| | | | |
| | * | | Merge branch 'master' into blockmanager_infoImran Rashid2013-01-3030-227/+264
| | |\ \ \ | | | | | | | | | | | | | | | | | | | | | | | | Conflicts: core/src/main/scala/spark/storage/BlockManagerMaster.scala
| | * | | | rename Slaves --> ExecutorImran Rashid2013-01-302-5/+5
| | | | | |
| | * | | | Merge branch 'master' into blockmanager_infoImran Rashid2013-01-2921-187/+202
| | |\ \ \ \
| | * | | | | better formatting for RDDInfoImran Rashid2013-01-281-3/+9
| | | | | | |
| | * | | | | expose RDD & storage info directly via SparkContextImran Rashid2013-01-284-28/+41
| | | | | | |
| * | | | | | Merge pull request #436 from stephenh/removeextraloopMatei Zaharia2013-02-021-13/+10
| |\ \ \ \ \ \ | | | | | | | | | | | | | | | | Once we find a split with no block, we don't have to look for more.
| | * | | | | | Further simplify checking for Nil.Stephen Haberman2013-02-021-3/+1
| | | | | | | |
| | * | | | | | Once we find a split with no block, we don't have to look for more.Stephen Haberman2013-01-311-12/+11
| | | |_|/ / / | | |/| | | |
| * | | | | | Merge pull request #442 from stephenh/fixsystemnamesMatei Zaharia2013-02-026-73/+68
| |\ \ \ \ \ \ | | | | | | | | | | | | | | | | Fix createActorSystem not actually using the systemName parameter.
| | * | | | | | Fix createActorSystem not actually using the systemName parameter.Stephen Haberman2013-02-026-73/+68
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This meant all system names were "spark", which worked, but didn't lead to the most intuitive log output. This fixes createActorSystem to use the passed system name, and refactors Master/Worker to encapsulate their system/actor names instead of having the clients guess at them. Note that the driver system name, "spark", is left as is, and is still repeated a few times, but that seems like a separate issue.
| * | | | | | | FormattingMatei Zaharia2013-02-021-1/+2
| | | | | | | |
| * | | | | | | FormattingMatei Zaharia2013-02-021-6/+9
| | | | | | | |
| * | | | | | | Merge remote-tracking branch 'base/master' into dag-sched-testsCharles Reiss2013-02-0250-395/+504
| |\ \ \ \ \ \ \ | | | |_|_|_|_|/ | | |/| | | | | | | | | | | | | | | | | | | | | Conflicts: core/src/main/scala/spark/scheduler/DAGScheduler.scala
| * | | | | | | Code review changes: add sc.stop; style of multiline comments; parens on ↵Charles Reiss2013-02-011-22/+47
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | procedure calls.
| * | | | | | | Comment at top of DAGSchedulerSuiteCharles Reiss2013-01-301-1/+14
| | | | | | | |
| * | | | | | | Change DAGSchedulerSuite to run DAGScheduler in the same Thread.Charles Reiss2013-01-301-249/+319
| | | | | | | |
| * | | | | | | Refactor DAGScheduler more to allow testing without a separate thread.Charles Reiss2013-01-301-65/+111
| | | | | | | |
| * | | | | | | Clear spark.master.port to cleanup for other testsCharles Reiss2013-01-291-0/+1
| | | | | | | |
| * | | | | | | Add DAGScheduler tests.Charles Reiss2013-01-291-0/+540
| | | | | | | |
| * | | | | | | Refactoring to DAGScheduler to aid testingCharles Reiss2013-01-292-12/+18
| | | | | | | |
| * | | | | | | Add easymock to POMsCharles Reiss2013-01-291-0/+5
| | |_|_|_|/ / | |/| | | | |
* | | | | | | Handle Terminated to avoid endless DeathPactExceptions.Stephen Haberman2013-02-052-19/+13
| |/ / / / / |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Credit to Roland Kuhn, Akka's tech lead, for pointing out this various obvious fix, but StandaloneExecutorBackend.preStart's catch block would never (ever) get hit, because all of the operation's in preStart are async. So, the System.exit in the catch block was skipped, and instead Akka was sending Terminated messages which, since we didn't handle, it turned into DeathPactException, which started a postRestart/preStart infinite loop.
* | | | | | Add back test for distinct without parensMatei Zaharia2013-02-011-1/+2
| | | | | |
* | | | | | Reduce the amount of duplicate logging Akka does to stdout.Stephen Haberman2013-02-011-0/+1
| |/ / / / |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Given we have Akka logging go through SLF4j to log4j, we don't need all the extra noise of Akka's stdout logger that is supposedly only used during Akka init time but seems to continue logging lots of noisy network events that we either don't care about or are in the log4j logs anyway. See: http://doc.akka.io/docs/akka/2.0/general/configuration.html # Log level for the very basic logger activated during AkkaApplication startup # Options: ERROR, WARNING, INFO, DEBUG # stdout-loglevel = "WARNING"
* | | | | Reduced the memory usage of reduce and similar operationsMatei Zaharia2013-02-019-46/+107
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | These operations used to wait for all the results to be available in an array on the driver program before merging them. They now merge values incrementally as they arrive.
* | | | | Merge branch 'master' of github.com:mesos/sparkMatei Zaharia2013-02-018-59/+49
|\ \ \ \ \
| * \ \ \ \ Merge pull request #432 from stephenh/moreprivacyMatei Zaharia2013-02-018-59/+49
| |\ \ \ \ \ | | | | | | | | | | | | | | Add more private declarations.
| | * | | | | Add more private declarations.Stephen Haberman2013-01-318-59/+49
| | | |/ / / | | |/| | |
* | / | | | formattingMatei Zaharia2013-02-012-3/+3
|/ / / / /
generated by cgit v1.2.3 (git 2.39.1) at 2024-09-19 12:12:33 +0000