aboutsummaryrefslogtreecommitdiff
Commit message (Collapse)AuthorAgeFilesLines
* Adding an example with an OLAP roll-upPatrick Wendell2013-02-041-0/+66
|
* Small fix to test for distinctMatei Zaharia2013-02-041-1/+1
|
* Fix failing testMatei Zaharia2013-02-041-2/+1
|
* Merge pull request #445 from JoshRosen/pyspark_fixesMatei Zaharia2013-02-035-22/+19
|\ | | | | Fix exit status in PySpark unit tests; fix/optimize PySpark's RDD.take()
| * Remove unnecessary doctest __main__ methods.Josh Rosen2013-02-032-18/+0
| |
| * Fetch fewer objects in PySpark's take() method.Josh Rosen2013-02-032-2/+13
| |
| * Fix reporting of PySpark doctest failures.Josh Rosen2013-02-032-2/+6
| |
* | Merge pull request #379 from stephenh/sparkmemMatei Zaharia2013-02-026-35/+17
|\ \ | | | | | | Add spark.executor.memory to differentiate executor memory from spark-shell
| * | Fix dangling old variable names.Stephen Haberman2013-02-021-2/+2
| | |
| * | Move executorMemory up into SchedulerBackend.Stephen Haberman2013-02-024-29/+12
| | |
| * | Merge branch 'master' into sparkmemStephen Haberman2013-02-02254-2124/+13947
| |\|
| * | Fix SPARK_MEM in ExecutorRunner.Stephen Haberman2013-01-221-1/+1
| | |
| * | Restore SPARK_MEM in executorEnvs.Stephen Haberman2013-01-221-2/+3
| | |
| * | Add spark.executor.memory to differentiate executor memory from spark-shell ↵Stephen Haberman2013-01-153-10/+8
| | | | | | | | | | | | memory.
* | | Merge pull request #422 from squito/blockmanager_infoMatei Zaharia2013-02-026-38/+59
|\ \ \ | | | | | | | | RDDInfo available from SparkContext
| * | | remove unneeded (and unused) filter on block infoImran Rashid2013-02-011-2/+0
| | | |
| * | | track total partitions, in addition to cached partitions; use scala string ↵Imran Rashid2013-02-013-9/+13
| | | | | | | | | | | | | | | | formatting
| * | | fixup merge (master -> driver renaming)Imran Rashid2013-02-011-1/+1
| | | |
| * | | Merge branch 'master' into blockmanager_infoImran Rashid2013-01-3043-246/+291
| |\ \ \ | | | | | | | | | | | | | | | | | | | | Conflicts: core/src/main/scala/spark/storage/BlockManagerMaster.scala
| * | | | rename Slaves --> ExecutorImran Rashid2013-01-302-5/+5
| | | | |
| * | | | Merge branch 'master' into blockmanager_infoImran Rashid2013-01-2923-192/+207
| |\ \ \ \
| * | | | | better formatting for RDDInfoImran Rashid2013-01-281-3/+9
| | | | | |
| * | | | | expose RDD & storage info directly via SparkContextImran Rashid2013-01-284-28/+41
| | | | | |
* | | | | | Merge pull request #436 from stephenh/removeextraloopMatei Zaharia2013-02-021-13/+10
|\ \ \ \ \ \ | | | | | | | | | | | | | | Once we find a split with no block, we don't have to look for more.
| * | | | | | Further simplify checking for Nil.Stephen Haberman2013-02-021-3/+1
| | | | | | |
| * | | | | | Once we find a split with no block, we don't have to look for more.Stephen Haberman2013-01-311-12/+11
| | |_|/ / / | |/| | | |
* | | | | | Merge pull request #442 from stephenh/fixsystemnamesMatei Zaharia2013-02-026-73/+68
|\ \ \ \ \ \ | | | | | | | | | | | | | | Fix createActorSystem not actually using the systemName parameter.
| * | | | | | Fix createActorSystem not actually using the systemName parameter.Stephen Haberman2013-02-026-73/+68
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This meant all system names were "spark", which worked, but didn't lead to the most intuitive log output. This fixes createActorSystem to use the passed system name, and refactors Master/Worker to encapsulate their system/actor names instead of having the clients guess at them. Note that the driver system name, "spark", is left as is, and is still repeated a few times, but that seems like a separate issue.
* | | | | | | FormattingMatei Zaharia2013-02-021-1/+2
| | | | | | |
* | | | | | | FormattingMatei Zaharia2013-02-021-6/+9
| | | | | | |
* | | | | | | Merge pull request #427 from woggling/dag-sched-testsMatei Zaharia2013-02-026-73/+802
|\ \ \ \ \ \ \ | |_|_|_|_|_|/ |/| | | | | | Tests for DAGScheduler
| * | | | | | Merge remote-tracking branch 'base/master' into dag-sched-testsCharles Reiss2013-02-0273-511/+584
| |\ \ \ \ \ \ | |/ / / / / / |/| | | | | | | | | | | | | | | | | | | | Conflicts: core/src/main/scala/spark/scheduler/DAGScheduler.scala
* | | | | | | Add back test for distinct without parensMatei Zaharia2013-02-011-1/+2
| | | | | | |
* | | | | | | Merge pull request #441 from stephenh/lessnoisyakkaMatei Zaharia2013-02-011-0/+1
|\ \ \ \ \ \ \ | |_|/ / / / / |/| | | | | | Reduce the amount of duplicate logging Akka does to stdout.
| * | | | | | Reduce the amount of duplicate logging Akka does to stdout.Stephen Haberman2013-02-011-0/+1
|/ / / / / / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Given we have Akka logging go through SLF4j to log4j, we don't need all the extra noise of Akka's stdout logger that is supposedly only used during Akka init time but seems to continue logging lots of noisy network events that we either don't care about or are in the log4j logs anyway. See: http://doc.akka.io/docs/akka/2.0/general/configuration.html # Log level for the very basic logger activated during AkkaApplication startup # Options: ERROR, WARNING, INFO, DEBUG # stdout-loglevel = "WARNING"
* | | | | | Reduced the memory usage of reduce and similar operationsMatei Zaharia2013-02-019-46/+107
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | These operations used to wait for all the results to be available in an array on the driver program before merging them. They now merge values incrementally as they arrive.
* | | | | | Merge branch 'master' of github.com:mesos/sparkMatei Zaharia2013-02-018-59/+49
|\ \ \ \ \ \
| * \ \ \ \ \ Merge pull request #432 from stephenh/moreprivacyMatei Zaharia2013-02-018-59/+49
| |\ \ \ \ \ \ | | | | | | | | | | | | | | | | Add more private declarations.
| | * | | | | | Add more private declarations.Stephen Haberman2013-01-318-59/+49
| | | |/ / / / | | |/| | | |
* | / | | | | formattingMatei Zaharia2013-02-012-3/+3
|/ / / / / /
* | | | | | Merge pull request #437 from stephenh/cancelmetacleanerMatei Zaharia2013-02-011-0/+1
|\ \ \ \ \ \ | | | | | | | | | | | | | | Stop BlockManagers metadataCleaner.
| * | | | | | Stop BlockManagers metadataCleaner.Stephen Haberman2013-02-011-0/+1
| |/ / / / /
* | | | | | Merge pull request #439 from JoshRosen/spark-580Matei Zaharia2013-02-012-10/+9
|\ \ \ \ \ \ | | | | | | | | | | | | | | Use spark.local.dir for PySpark temp files (SPARK-580).
| * | | | | | Use spark.local.dir for PySpark temp files (SPARK-580).Josh Rosen2013-02-012-10/+9
|/ / / / / /
* | | | | | Merge pull request #438 from JoshRosen/spark-674Matei Zaharia2013-02-014-18/+25
|\ \ \ \ \ \ | | | | | | | | | | | | | | Do not launch JavaGateways on workers (SPARK-674).
| * | | | | | Do not launch JavaGateways on workers (SPARK-674).Josh Rosen2013-02-014-18/+25
|/ / / / / / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The problem was that the gateway was being initialized whenever the pyspark.context module was loaded. The fix uses lazy initialization that occurs only when SparkContext instances are actually constructed. I also made the gateway and jvm variables private. This change results in ~3-4x performance improvement when running the PySpark unit tests.
* | | | | | Merge pull request #433 from rxin/masterMatei Zaharia2013-02-012-20/+24
|\ \ \ \ \ \ | | | | | | | | | | | | | | Changed PartitionPruningRDD's split to make sure it returns the correct split index.
| * | | | | | Moved PruneDependency into PartitionPruningRDD.scala.Reynold Xin2013-02-012-26/+22
| | | | | | |
| * | | | | | Removed the TODO comment from PartitionPruningRDD.Reynold Xin2013-01-311-2/+0
| | | | | | |
| * | | | | | Changed PartitionPruningRDD's split to make sure it returns the correctReynold Xin2013-01-312-1/+11
| |/ / / / / | | | | | | | | | | | | | | | | | | split index.
generated by cgit v1.2.3 (git 2.39.1) at 2024-09-20 18:48:40 +0000