aboutsummaryrefslogtreecommitdiff
path: root/core
Commit message (Collapse)AuthorAgeFilesLines
* Merge pull request #516 from squito/fix_local_metricsMatei Zaharia2013-03-158-12/+104
|\ | | | | Fix local metrics
| * increase sleep timeImran Rashid2013-03-101-1/+1
| |
| * add a small wait to one task to make sure some task runtime really is non-zeroImran Rashid2013-03-101-4/+10
| |
| * enable task metrics in local mode, add testsImran Rashid2013-03-092-2/+88
| |
| * rename remoteFetchWaitTime to fetchWaitTime, since it also includes time ↵Imran Rashid2013-03-096-10/+10
| | | | | | | | from local fetches
* | Add a log4j compile dependency to fix build in IntelliJMikhail Bautin2013-03-151-1/+5
| | | | | | | | | | Also rename parent project to spark-parent (otherwise it shows up as "parent" in IntelliJ, which is very confusing).
* | Merge pull request #521 from stephenh/earlycloseMatei Zaharia2013-03-134-51/+153
|\ \ | | | | | | Close the reader in HadoopRDD as soon as iteration end.
| * | Add a test for NextIterator.Stephen Haberman2013-03-131-0/+68
| | |
| * | Add NextIterator.closeIfNeeded.Stephen Haberman2013-03-132-2/+16
| | |
| * | Remove NextIterator.close default implementation.Stephen Haberman2013-03-122-4/+7
| | |
| * | More quickly call close in HadoopRDD.Stephen Haberman2013-03-113-52/+69
| | | | | | | | | | | | | | | This also refactors out the common "gotNext" iterator pattern into a shared utility class.
* | | Send block sizes as longs.Charles Reiss2013-03-111-4/+4
|/ /
* | Merge remote-tracking branch 'woggling/dag-sched-driver-port'Matei Zaharia2013-03-101-5/+4
|\ \ | | | | | | | | | | | | Conflicts: core/src/test/scala/spark/scheduler/DAGSchedulerSuite.scala
| * | Prevent DAGSchedulerSuite from corrupting driver.port.Charles Reiss2013-03-091-4/+5
| |/ | | | | | | | | Use the LocalSparkContext abstraction to properly manage clearing spark.driver.port.
* | Merge pull request #512 from patelh/fix-kryo-serializerMatei Zaharia2013-03-102-11/+19
|\ \ | | | | | | Fix reference bug in Kryo serializer, add test, update version
| * | Fix reference bug in Kryo serializer, add test, update versionHiral Patel2013-03-072-11/+19
| |/
* | Merge pull request #515 from woggling/deploy-app-deathMatei Zaharia2013-03-103-7/+17
|\ \ | | | | | | Notify standalone deploy client of application death.
| * | Notify standalone deploy client of application death.Charles Reiss2013-03-093-7/+17
| |/ | | | | | | | | | | | | | | | | Usually, this isn't necessary since the application will be removed as a result of the deploy client disconnecting, but occassionally, the standalone deploy master removes an application otherwise. Also mark applications as FAILED instead of FINISHED when they are killed as a result of their executors failing too many times.
* | Merge remote-tracking branch 'stephenh/nomocks'Matei Zaharia2013-03-109-536/+275
|\ \ | |/ |/| | | | | | | Conflicts: core/src/main/scala/spark/storage/BlockManagerMaster.scala core/src/test/scala/spark/scheduler/DAGSchedulerSuite.scala
| * Fix MapOutputTrackerSuite.Stephen Haberman2013-02-261-2/+4
| |
| * Override DAGScheduler.runLocally so we can remove the Thread.sleep.Stephen Haberman2013-02-252-19/+27
| |
| * Merge branch 'master' into nomocksStephen Haberman2013-02-2595-890/+1411
| |\ | | | | | | | | | | | | Conflicts: core/src/test/scala/spark/scheduler/DAGSchedulerSuite.scala
| * | Use stubs instead of mocks for DAGSchedulerSuite.Stephen Haberman2013-02-098-527/+253
| | |
* | | Fix TaskMetrics not being serializableMatei Zaharia2013-03-041-6/+13
| | |
* | | Merge pull request #506 from rxin/spark-706Matei Zaharia2013-03-032-51/+120
|\ \ \ | | | | | | | | Fixed SPARK-706: Failures in block manager put leads to read task hanging.
| * | | Fixed SPARK-706: Failures in block manager put leads to read taskReynold Xin2013-02-282-51/+120
| | | | | | | | | | | | | | | | hanging.
* | | | minor cleanup based on feedback in review requestImran Rashid2013-03-034-24/+24
| | | |
* | | | change CleanupIterator to CompletionIteratorImran Rashid2013-03-033-27/+27
| | | |
* | | | refactoring of TaskMetricsImran Rashid2013-03-036-59/+110
| | | |
* | | | Merge branch 'master' into stageInfoImran Rashid2013-03-0329-107/+474
|\ \ \ \
| * \ \ \ Merge pull request #504 from mosharaf/masterMatei Zaharia2013-03-022-2/+2
| |\ \ \ \ | | | | | | | | | | | | Worker address was getting removed when removing an app.
| | * | | | Fixed master datastructure updates after removing an application; and a typo.Mosharaf Chowdhury2013-02-272-2/+2
| | | | | |
| * | | | | bump version to 0.7.1-SNAPSHOT in the subproject poms to keep the maven ↵Mark Hamstra2013-02-281-1/+1
| |/ / / / | | | | | | | | | | | | | | | build building.
| * | | | Fix a problem with no hosts being counted as alive in the first jobMatei Zaharia2013-02-261-3/+3
| | | | |
| * | | | Fix overly large thread names in PySparkMatei Zaharia2013-02-261-2/+2
| | | | |
| * | | | Fixed replication bug in BlockManagerTathagata Das2013-02-252-3/+17
| | | | |
| * | | | Allow passing sparkHome and JARs to StreamingContext constructorMatei Zaharia2013-02-252-3/+3
| | | | | | | | | | | | | | | | | | | | | | | | | Also warns if spark.cleaner.ttl is not set in the version where you pass your own SparkContext.
| * | | | Set spark.deploy.spreadOut to true by default in 0.7 (improves locality)Matei Zaharia2013-02-251-1/+1
| | | | |
| * | | | Add a config property for Akka lifecycle event loggingMatei Zaharia2013-02-251-2/+4
| | | | |
| * | | | Merge pull request #498 from pwendell/shutup-akkaMatei Zaharia2013-02-251-1/+1
| |\ \ \ \ | | | | | | | | | | | | Disable remote lifecycle logging from Akka.
| | * | | | Disable remote lifecycle logging from Akka.Patrick Wendell2013-02-251-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | This changes the default setting to `off` for remote lifecycle events. When this is on, it is very chatty at the INFO level. It also prints out several ERROR messages sometimes when sc.stop() is called.
| * | | | | Get spark.default.paralellism on each call to defaultPartitioner,Matei Zaharia2013-02-251-4/+1
| | |_|_|/ | |/| | | | | | | | | | | | | instead of only once, in case the user changes it across Spark uses
| * | | | Merge pull request #459 from stephenh/bettersplitsMatei Zaharia2013-02-258-38/+90
| |\ \ \ \ | | |/ / / | |/| | | Change defaultPartitioner to use upstream split size.
| | * | | Use default parallelism if its set.Stephen Haberman2013-02-242-6/+19
| | | | |
| | * | | Merge branch 'master' into bettersplitsStephen Haberman2013-02-2492-793/+1260
| | |\ \ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Conflicts: core/src/main/scala/spark/RDD.scala core/src/main/scala/spark/scheduler/cluster/StandaloneSchedulerBackend.scala core/src/test/scala/spark/ShuffleSuite.scala
| | * | | | Update more javadocs.Stephen Haberman2013-02-162-15/+17
| | | | | |
| | * | | | Tweak test names.Stephen Haberman2013-02-161-2/+2
| | | | | |
| | * | | | Remove fileServerSuite.txt.Stephen Haberman2013-02-161-1/+0
| | | | | |
| | * | | | Update default.parallelism docs, have StandaloneSchedulerBackend use it.Stephen Haberman2013-02-167-24/+39
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Only brand new RDDs (e.g. parallelize and makeRDD) now use default parallelism, everything else uses their largest parent's partitioner or partition size.
| | * | | | Change defaultPartitioner to use upstream split size.Stephen Haberman2013-02-103-6/+29
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Previously it used the SparkContext.defaultParallelism, which occassionally ended up being a very bad guess. Looking at upstream RDDs seems to make better use of the context. Also sorted the upstream RDDs by partition size first, as if we have a hugely-partitioned RDD and tiny-partitioned RDD, it is unlikely we want the resulting RDD to be tiny-partitioned.
generated by cgit v1.2.3 (git 2.39.1) at 2024-09-19 01:27:06 +0000