Commit message (Collapse) | Author | Age | Files | Lines | |
---|---|---|---|---|---|
* | Adding an example with an OLAP roll-up | Patrick Wendell | 2013-02-04 | 1 | -0/+66 |
| | |||||
* | Small fix to test for distinct | Matei Zaharia | 2013-02-04 | 1 | -1/+1 |
| | |||||
* | Fix failing test | Matei Zaharia | 2013-02-04 | 1 | -2/+1 |
| | |||||
* | Merge pull request #445 from JoshRosen/pyspark_fixes | Matei Zaharia | 2013-02-03 | 5 | -22/+19 |
|\ | | | | | Fix exit status in PySpark unit tests; fix/optimize PySpark's RDD.take() | ||||
| * | Remove unnecessary doctest __main__ methods. | Josh Rosen | 2013-02-03 | 2 | -18/+0 |
| | | |||||
| * | Fetch fewer objects in PySpark's take() method. | Josh Rosen | 2013-02-03 | 2 | -2/+13 |
| | | |||||
| * | Fix reporting of PySpark doctest failures. | Josh Rosen | 2013-02-03 | 2 | -2/+6 |
| | | |||||
* | | Merge pull request #379 from stephenh/sparkmem | Matei Zaharia | 2013-02-02 | 6 | -35/+17 |
|\ \ | | | | | | | Add spark.executor.memory to differentiate executor memory from spark-shell | ||||
| * | | Fix dangling old variable names. | Stephen Haberman | 2013-02-02 | 1 | -2/+2 |
| | | | |||||
| * | | Move executorMemory up into SchedulerBackend. | Stephen Haberman | 2013-02-02 | 4 | -29/+12 |
| | | | |||||
| * | | Merge branch 'master' into sparkmem | Stephen Haberman | 2013-02-02 | 254 | -2124/+13947 |
| |\| | |||||
| * | | Fix SPARK_MEM in ExecutorRunner. | Stephen Haberman | 2013-01-22 | 1 | -1/+1 |
| | | | |||||
| * | | Restore SPARK_MEM in executorEnvs. | Stephen Haberman | 2013-01-22 | 1 | -2/+3 |
| | | | |||||
| * | | Add spark.executor.memory to differentiate executor memory from spark-shell ↵ | Stephen Haberman | 2013-01-15 | 3 | -10/+8 |
| | | | | | | | | | | | | memory. | ||||
* | | | Merge pull request #422 from squito/blockmanager_info | Matei Zaharia | 2013-02-02 | 6 | -38/+59 |
|\ \ \ | | | | | | | | | RDDInfo available from SparkContext | ||||
| * | | | remove unneeded (and unused) filter on block info | Imran Rashid | 2013-02-01 | 1 | -2/+0 |
| | | | | |||||
| * | | | track total partitions, in addition to cached partitions; use scala string ↵ | Imran Rashid | 2013-02-01 | 3 | -9/+13 |
| | | | | | | | | | | | | | | | | formatting | ||||
| * | | | fixup merge (master -> driver renaming) | Imran Rashid | 2013-02-01 | 1 | -1/+1 |
| | | | | |||||
| * | | | Merge branch 'master' into blockmanager_info | Imran Rashid | 2013-01-30 | 43 | -246/+291 |
| |\ \ \ | | | | | | | | | | | | | | | | | | | | | Conflicts: core/src/main/scala/spark/storage/BlockManagerMaster.scala | ||||
| * | | | | rename Slaves --> Executor | Imran Rashid | 2013-01-30 | 2 | -5/+5 |
| | | | | | |||||
| * | | | | Merge branch 'master' into blockmanager_info | Imran Rashid | 2013-01-29 | 23 | -192/+207 |
| |\ \ \ \ | |||||
| * | | | | | better formatting for RDDInfo | Imran Rashid | 2013-01-28 | 1 | -3/+9 |
| | | | | | | |||||
| * | | | | | expose RDD & storage info directly via SparkContext | Imran Rashid | 2013-01-28 | 4 | -28/+41 |
| | | | | | | |||||
* | | | | | | Merge pull request #436 from stephenh/removeextraloop | Matei Zaharia | 2013-02-02 | 1 | -13/+10 |
|\ \ \ \ \ \ | | | | | | | | | | | | | | | Once we find a split with no block, we don't have to look for more. | ||||
| * | | | | | | Further simplify checking for Nil. | Stephen Haberman | 2013-02-02 | 1 | -3/+1 |
| | | | | | | | |||||
| * | | | | | | Once we find a split with no block, we don't have to look for more. | Stephen Haberman | 2013-01-31 | 1 | -12/+11 |
| | |_|/ / / | |/| | | | | |||||
* | | | | | | Merge pull request #442 from stephenh/fixsystemnames | Matei Zaharia | 2013-02-02 | 6 | -73/+68 |
|\ \ \ \ \ \ | | | | | | | | | | | | | | | Fix createActorSystem not actually using the systemName parameter. | ||||
| * | | | | | | Fix createActorSystem not actually using the systemName parameter. | Stephen Haberman | 2013-02-02 | 6 | -73/+68 |
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This meant all system names were "spark", which worked, but didn't lead to the most intuitive log output. This fixes createActorSystem to use the passed system name, and refactors Master/Worker to encapsulate their system/actor names instead of having the clients guess at them. Note that the driver system name, "spark", is left as is, and is still repeated a few times, but that seems like a separate issue. | ||||
* | | | | | | | Formatting | Matei Zaharia | 2013-02-02 | 1 | -1/+2 |
| | | | | | | | |||||
* | | | | | | | Formatting | Matei Zaharia | 2013-02-02 | 1 | -6/+9 |
| | | | | | | | |||||
* | | | | | | | Merge pull request #427 from woggling/dag-sched-tests | Matei Zaharia | 2013-02-02 | 6 | -73/+802 |
|\ \ \ \ \ \ \ | |_|_|_|_|_|/ |/| | | | | | | Tests for DAGScheduler | ||||
| * | | | | | | Merge remote-tracking branch 'base/master' into dag-sched-tests | Charles Reiss | 2013-02-02 | 73 | -511/+584 |
| |\ \ \ \ \ \ | |/ / / / / / |/| | | | | | | | | | | | | | | | | | | | | Conflicts: core/src/main/scala/spark/scheduler/DAGScheduler.scala | ||||
* | | | | | | | Add back test for distinct without parens | Matei Zaharia | 2013-02-01 | 1 | -1/+2 |
| | | | | | | | |||||
* | | | | | | | Merge pull request #441 from stephenh/lessnoisyakka | Matei Zaharia | 2013-02-01 | 1 | -0/+1 |
|\ \ \ \ \ \ \ | |_|/ / / / / |/| | | | | | | Reduce the amount of duplicate logging Akka does to stdout. | ||||
| * | | | | | | Reduce the amount of duplicate logging Akka does to stdout. | Stephen Haberman | 2013-02-01 | 1 | -0/+1 |
|/ / / / / / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Given we have Akka logging go through SLF4j to log4j, we don't need all the extra noise of Akka's stdout logger that is supposedly only used during Akka init time but seems to continue logging lots of noisy network events that we either don't care about or are in the log4j logs anyway. See: http://doc.akka.io/docs/akka/2.0/general/configuration.html # Log level for the very basic logger activated during AkkaApplication startup # Options: ERROR, WARNING, INFO, DEBUG # stdout-loglevel = "WARNING" | ||||
* | | | | | | Reduced the memory usage of reduce and similar operations | Matei Zaharia | 2013-02-01 | 9 | -46/+107 |
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | These operations used to wait for all the results to be available in an array on the driver program before merging them. They now merge values incrementally as they arrive. | ||||
* | | | | | | Merge branch 'master' of github.com:mesos/spark | Matei Zaharia | 2013-02-01 | 8 | -59/+49 |
|\ \ \ \ \ \ | |||||
| * \ \ \ \ \ | Merge pull request #432 from stephenh/moreprivacy | Matei Zaharia | 2013-02-01 | 8 | -59/+49 |
| |\ \ \ \ \ \ | | | | | | | | | | | | | | | | | Add more private declarations. | ||||
| | * | | | | | | Add more private declarations. | Stephen Haberman | 2013-01-31 | 8 | -59/+49 |
| | | |/ / / / | | |/| | | | | |||||
* | / | | | | | formatting | Matei Zaharia | 2013-02-01 | 2 | -3/+3 |
|/ / / / / / | |||||
* | | | | | | Merge pull request #437 from stephenh/cancelmetacleaner | Matei Zaharia | 2013-02-01 | 1 | -0/+1 |
|\ \ \ \ \ \ | | | | | | | | | | | | | | | Stop BlockManagers metadataCleaner. | ||||
| * | | | | | | Stop BlockManagers metadataCleaner. | Stephen Haberman | 2013-02-01 | 1 | -0/+1 |
| |/ / / / / | |||||
* | | | | | | Merge pull request #439 from JoshRosen/spark-580 | Matei Zaharia | 2013-02-01 | 2 | -10/+9 |
|\ \ \ \ \ \ | | | | | | | | | | | | | | | Use spark.local.dir for PySpark temp files (SPARK-580). | ||||
| * | | | | | | Use spark.local.dir for PySpark temp files (SPARK-580). | Josh Rosen | 2013-02-01 | 2 | -10/+9 |
|/ / / / / / | |||||
* | | | | | | Merge pull request #438 from JoshRosen/spark-674 | Matei Zaharia | 2013-02-01 | 4 | -18/+25 |
|\ \ \ \ \ \ | | | | | | | | | | | | | | | Do not launch JavaGateways on workers (SPARK-674). | ||||
| * | | | | | | Do not launch JavaGateways on workers (SPARK-674). | Josh Rosen | 2013-02-01 | 4 | -18/+25 |
|/ / / / / / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The problem was that the gateway was being initialized whenever the pyspark.context module was loaded. The fix uses lazy initialization that occurs only when SparkContext instances are actually constructed. I also made the gateway and jvm variables private. This change results in ~3-4x performance improvement when running the PySpark unit tests. | ||||
* | | | | | | Merge pull request #433 from rxin/master | Matei Zaharia | 2013-02-01 | 2 | -20/+24 |
|\ \ \ \ \ \ | | | | | | | | | | | | | | | Changed PartitionPruningRDD's split to make sure it returns the correct split index. | ||||
| * | | | | | | Moved PruneDependency into PartitionPruningRDD.scala. | Reynold Xin | 2013-02-01 | 2 | -26/+22 |
| | | | | | | | |||||
| * | | | | | | Removed the TODO comment from PartitionPruningRDD. | Reynold Xin | 2013-01-31 | 1 | -2/+0 |
| | | | | | | | |||||
| * | | | | | | Changed PartitionPruningRDD's split to make sure it returns the correct | Reynold Xin | 2013-01-31 | 2 | -1/+11 |
| |/ / / / / | | | | | | | | | | | | | | | | | | | split index. |