aboutsummaryrefslogtreecommitdiff
Commit message (Collapse)AuthorAgeFilesLines
...
| * | | | | | | Merge branch 'master' into blockmanager_infoImran Rashid2013-01-2923-192/+207
| |\ \ \ \ \ \ \
| * | | | | | | | better formatting for RDDInfoImran Rashid2013-01-281-3/+9
| | | | | | | | |
| * | | | | | | | expose RDD & storage info directly via SparkContextImran Rashid2013-01-284-28/+41
| | | | | | | | |
* | | | | | | | | Merge pull request #436 from stephenh/removeextraloopMatei Zaharia2013-02-021-13/+10
|\ \ \ \ \ \ \ \ \ | | | | | | | | | | | | | | | | | | | | Once we find a split with no block, we don't have to look for more.
| * | | | | | | | | Further simplify checking for Nil.Stephen Haberman2013-02-021-3/+1
| | | | | | | | | |
| * | | | | | | | | Once we find a split with no block, we don't have to look for more.Stephen Haberman2013-01-311-12/+11
| | |_|/ / / / / / | |/| | | | | | |
* | | | | | | | | Merge pull request #442 from stephenh/fixsystemnamesMatei Zaharia2013-02-026-73/+68
|\ \ \ \ \ \ \ \ \ | | | | | | | | | | | | | | | | | | | | Fix createActorSystem not actually using the systemName parameter.
| * | | | | | | | | Fix createActorSystem not actually using the systemName parameter.Stephen Haberman2013-02-026-73/+68
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This meant all system names were "spark", which worked, but didn't lead to the most intuitive log output. This fixes createActorSystem to use the passed system name, and refactors Master/Worker to encapsulate their system/actor names instead of having the clients guess at them. Note that the driver system name, "spark", is left as is, and is still repeated a few times, but that seems like a separate issue.
* | | | | | | | | | FormattingMatei Zaharia2013-02-021-1/+2
| | | | | | | | | |
* | | | | | | | | | FormattingMatei Zaharia2013-02-021-6/+9
| | | | | | | | | |
* | | | | | | | | | Merge pull request #427 from woggling/dag-sched-testsMatei Zaharia2013-02-026-73/+802
|\ \ \ \ \ \ \ \ \ \ | |_|_|_|_|_|/ / / / |/| | | | | | | | | Tests for DAGScheduler
| * | | | | | | | | Merge remote-tracking branch 'base/master' into dag-sched-testsCharles Reiss2013-02-0273-511/+584
| |\ \ \ \ \ \ \ \ \ | |/ / / / / / / / / |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | Conflicts: core/src/main/scala/spark/scheduler/DAGScheduler.scala
| * | | | | | | | | Code review changes: add sc.stop; style of multiline comments; parens on ↵Charles Reiss2013-02-011-22/+47
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | procedure calls.
| * | | | | | | | | Comment at top of DAGSchedulerSuiteCharles Reiss2013-01-301-1/+14
| | | | | | | | | |
| * | | | | | | | | Change DAGSchedulerSuite to run DAGScheduler in the same Thread.Charles Reiss2013-01-301-249/+319
| | | | | | | | | |
| * | | | | | | | | Refactor DAGScheduler more to allow testing without a separate thread.Charles Reiss2013-01-301-65/+111
| | | | | | | | | |
| * | | | | | | | | Clear spark.master.port to cleanup for other testsCharles Reiss2013-01-291-0/+1
| | | | | | | | | |
| * | | | | | | | | Add DAGScheduler tests.Charles Reiss2013-01-291-0/+540
| | | | | | | | | |
| * | | | | | | | | Refactoring to DAGScheduler to aid testingCharles Reiss2013-01-292-12/+18
| | | | | | | | | |
| * | | | | | | | | Add easymock to SBT configuration.Charles Reiss2013-01-291-1/+2
| | | | | | | | | |
| * | | | | | | | | Add easymock to POMsCharles Reiss2013-01-292-0/+11
| | |_|_|/ / / / / | |/| | | | | | |
| | | | | | * | | Handle Terminated to avoid endless DeathPactExceptions.Stephen Haberman2013-02-052-19/+13
| |_|_|_|_|/ / / |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Credit to Roland Kuhn, Akka's tech lead, for pointing out this various obvious fix, but StandaloneExecutorBackend.preStart's catch block would never (ever) get hit, because all of the operation's in preStart are async. So, the System.exit in the catch block was skipped, and instead Akka was sending Terminated messages which, since we didn't handle, it turned into DeathPactException, which started a postRestart/preStart infinite loop.
* | | | | | | | Add back test for distinct without parensMatei Zaharia2013-02-011-1/+2
| | | | | | | |
* | | | | | | | Merge pull request #441 from stephenh/lessnoisyakkaMatei Zaharia2013-02-011-0/+1
|\ \ \ \ \ \ \ \ | |_|/ / / / / / |/| | | | | | | Reduce the amount of duplicate logging Akka does to stdout.
| * | | | | | | Reduce the amount of duplicate logging Akka does to stdout.Stephen Haberman2013-02-011-0/+1
|/ / / / / / / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Given we have Akka logging go through SLF4j to log4j, we don't need all the extra noise of Akka's stdout logger that is supposedly only used during Akka init time but seems to continue logging lots of noisy network events that we either don't care about or are in the log4j logs anyway. See: http://doc.akka.io/docs/akka/2.0/general/configuration.html # Log level for the very basic logger activated during AkkaApplication startup # Options: ERROR, WARNING, INFO, DEBUG # stdout-loglevel = "WARNING"
* | | | | | | Reduced the memory usage of reduce and similar operationsMatei Zaharia2013-02-019-46/+107
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | These operations used to wait for all the results to be available in an array on the driver program before merging them. They now merge values incrementally as they arrive.
* | | | | | | Merge branch 'master' of github.com:mesos/sparkMatei Zaharia2013-02-018-59/+49
|\ \ \ \ \ \ \
| * \ \ \ \ \ \ Merge pull request #432 from stephenh/moreprivacyMatei Zaharia2013-02-018-59/+49
| |\ \ \ \ \ \ \ | | | | | | | | | | | | | | | | | | Add more private declarations.
| | * | | | | | | Add more private declarations.Stephen Haberman2013-01-318-59/+49
| | | |/ / / / / | | |/| | | | |
* | / | | | | | formattingMatei Zaharia2013-02-012-3/+3
|/ / / / / / /
* | | | | | | Merge pull request #437 from stephenh/cancelmetacleanerMatei Zaharia2013-02-011-0/+1
|\ \ \ \ \ \ \ | | | | | | | | | | | | | | | | Stop BlockManagers metadataCleaner.
| * | | | | | | Stop BlockManagers metadataCleaner.Stephen Haberman2013-02-011-0/+1
| |/ / / / / /
* | | | | | | Merge pull request #439 from JoshRosen/spark-580Matei Zaharia2013-02-012-10/+9
|\ \ \ \ \ \ \ | | | | | | | | | | | | | | | | Use spark.local.dir for PySpark temp files (SPARK-580).
| * | | | | | | Use spark.local.dir for PySpark temp files (SPARK-580).Josh Rosen2013-02-012-10/+9
|/ / / / / / /
* | | | | | | Merge pull request #438 from JoshRosen/spark-674Matei Zaharia2013-02-014-18/+25
|\ \ \ \ \ \ \ | | | | | | | | | | | | | | | | Do not launch JavaGateways on workers (SPARK-674).
| * | | | | | | Do not launch JavaGateways on workers (SPARK-674).Josh Rosen2013-02-014-18/+25
|/ / / / / / / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The problem was that the gateway was being initialized whenever the pyspark.context module was loaded. The fix uses lazy initialization that occurs only when SparkContext instances are actually constructed. I also made the gateway and jvm variables private. This change results in ~3-4x performance improvement when running the PySpark unit tests.
* | | | | | | Merge pull request #433 from rxin/masterMatei Zaharia2013-02-012-20/+24
|\ \ \ \ \ \ \ | | | | | | | | | | | | | | | | Changed PartitionPruningRDD's split to make sure it returns the correct split index.
| * | | | | | | Moved PruneDependency into PartitionPruningRDD.scala.Reynold Xin2013-02-012-26/+22
| | | | | | | |
| * | | | | | | Removed the TODO comment from PartitionPruningRDD.Reynold Xin2013-01-311-2/+0
| | | | | | | |
| * | | | | | | Changed PartitionPruningRDD's split to make sure it returns the correctReynold Xin2013-01-312-1/+11
| |/ / / / / / | | | | | | | | | | | | | | | | | | | | | split index.
* | | | | | | Merge pull request #435 from JoshRosen/pyspark_stdout_fixMatei Zaharia2013-02-012-2/+12
|\ \ \ \ \ \ \ | | | | | | | | | | | | | | | | Fix stdout redirection in PySpark.
| * | | | | | | Fix stdout redirection in PySpark.Josh Rosen2013-02-012-2/+12
|/ / / / / / /
* | | | | | | Merge pull request #434 from pwendell/python-exceptionsMatei Zaharia2013-01-312-17/+32
|\ \ \ \ \ \ \ | | | | | | | | | | | | | | | | SPARK-673: Capture and re-throw Python exceptions
| * | | | | | | Small fix from last commitPatrick Wendell2013-01-311-1/+1
| | | | | | | |
| * | | | | | | Some style cleanupPatrick Wendell2013-01-311-7/+4
| | | | | | | |
| * | | | | | | SPARK-673: Capture and re-throw Python exceptionsPatrick Wendell2013-01-312-16/+34
| |/ / / / / / | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch alters the Python <-> executor protocol to pass on exception data when they occur in user Python code.
* | | | | | | Merge pull request #431 from mbautin/revert_default_profileMatei Zaharia2013-01-317-77/+0
|\ \ \ \ \ \ \ | |/ / / / / / |/| | | | | | Remove activation of profiles by default
| * | | | | | Remove activation of profiles by defaultMikhail Bautin2013-01-317-77/+0
|/ / / / / / | | | | | | | | | | | | | | | | | | | | | | | | See the discussion at https://github.com/mesos/spark/pull/355 for why default profile activation is a problem.
* | | | | | Merge pull request #430 from pwendell/pyspark-guideMatei Zaharia2013-01-302-2/+10
|\ \ \ \ \ \ | | | | | | | | | | | | | | Minor improvements to PySpark docs
| * | | | | | Make module help available in python shell.Patrick Wendell2013-01-302-0/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Also, adds a line in doc explaining how to use.
generated by cgit v1.2.3 (git 2.39.1) at 2024-07-28 14:11:59 +0000