aboutsummaryrefslogtreecommitdiff
path: root/core
Commit message (Collapse)AuthorAgeFilesLines
* Refactored a whole lot to push all DStreams into the spark.streaming.dstream ↵Tathagata Das2012-12-301-0/+1
| | | | package.
* Improved jekyll and scala docs. Made many classes and method private to ↵Tathagata Das2012-12-291-1/+0
| | | | remove them from scala docs.
* Modifications to make sure LocalScheduler terminate cleanly without errors ↵Tathagata Das2012-12-274-13/+21
| | | | when SparkContext is shutdown, to minimize spurious exception during master failure tests.
* Merge branch 'mesos' into dev-mergeTathagata Das2012-12-265-142/+90
|\
| * fall back to filter-map-collect when calling lookup() on an RDD without a ↵Mark Hamstra2012-12-242-1/+12
| | | | | | | | partitioner
| * Allow distinct() to be called without parentheses when using the default ↵Mark Hamstra2012-12-242-5/+11
| | | | | | | | number of splits.
| * Remove the call to close input stream in Kryo serializer.Reynold Xin2012-12-211-1/+1
| |
| * Merge pull request #341 from rxin/4a3fb06ac2d11125feb08acbbd4df76d1e91b677Matei Zaharia2012-12-211-137/+68
| |\ | | | | | | Kryo2 update against Spark master
| | * Updated Kryo to 2.20.Reynold Xin2012-10-161-137/+68
| | |
* | | Merge branch 'dev-checkpoint' of github.com:radlab/spark into dev-mergeTathagata Das2012-12-2626-470/+921
|\ \ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Conflicts: core/src/main/scala/spark/ParallelCollection.scala core/src/main/scala/spark/RDD.scala core/src/main/scala/spark/rdd/BlockRDD.scala core/src/main/scala/spark/rdd/CartesianRDD.scala core/src/main/scala/spark/rdd/CoGroupedRDD.scala core/src/main/scala/spark/rdd/CoalescedRDD.scala core/src/main/scala/spark/rdd/FilteredRDD.scala core/src/main/scala/spark/rdd/FlatMappedRDD.scala core/src/main/scala/spark/rdd/GlommedRDD.scala core/src/main/scala/spark/rdd/HadoopRDD.scala core/src/main/scala/spark/rdd/MapPartitionsRDD.scala core/src/main/scala/spark/rdd/MapPartitionsWithSplitRDD.scala core/src/main/scala/spark/rdd/MappedRDD.scala core/src/main/scala/spark/rdd/PipedRDD.scala core/src/main/scala/spark/rdd/SampledRDD.scala core/src/main/scala/spark/rdd/ShuffledRDD.scala core/src/main/scala/spark/rdd/UnionRDD.scala core/src/main/scala/spark/scheduler/ResultTask.scala core/src/test/scala/spark/CheckpointSuite.scala
| * \ \ Merge branch 'dev' of github.com:radlab/spark into dev-checkpointTathagata Das2012-12-204-10/+29
| |\ \ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Conflicts: core/src/main/scala/spark/ParallelCollection.scala core/src/test/scala/spark/CheckpointSuite.scala streaming/src/main/scala/spark/streaming/DStream.scala
| * | | | Fixed bugs in CheckpointRDD and spark.CheckpointSuite.Tathagata Das2012-12-203-12/+9
| | | | |
| * | | | Changed checkpoint writing and reading process.Tathagata Das2012-12-203-25/+124
| | | | |
| * | | | Introduced getSpits, getDependencies, and getPreferredLocations in RDD and ↵Tathagata Das2012-12-1822-113/+134
| | | | | | | | | | | | | | | | | | | | RDDCheckpointData.
| * | | | Converted CheckpointState in RDDCheckpointData to use scala Enumeration.Tathagata Das2012-12-171-26/+22
| | | | |
| * | | | Made checkpoint data in RDDs optional to further reduce serialized size.Tathagata Das2012-12-113-14/+28
| | | | |
| * | | | Fixed bug in UnionRDD and CoGroupedRDDTathagata Das2012-12-113-115/+10
| | | | |
| * | | | Bunch of bug fixes related to checkpointing in RDDs. RDDCheckpointData ↵Tathagata Das2012-12-1013-90/+389
| | | | | | | | | | | | | | | | | | | | object is used to lock all serialization and dependency changes for checkpointing. ResultTask converted to Externalizable and serialized RDD is cached like ShuffleMapTask.
| * | | | Modified checkpoint testsuite to more comprehensively test checkpointing of ↵Tathagata Das2012-12-0714-110/+285
| | | | | | | | | | | | | | | | | | | | various RDDs. Fixed checkpoint bug (splits referring to parent RDDs or parent splits) in UnionRDD and CoalescedRDD. Fixed bug in testing ShuffledRDD. Removed unnecessary and useless map-side combining step for narrow dependencies in CoGroupedRDD. Removed unncessary WeakReference stuff from many other RDDs.
| * | | | Refactored RDD checkpointing to minimize extra fields in RDD class.Tathagata Das2012-12-0411-191/+140
| | | | |
* | | | | Merge branch 'master' of github.com:mesos/spark into devReynold Xin2012-12-2071-850/+1940
|\ \ \ \ \ | |_|/ / / |/| | / / | | |/ / | |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Conflicts: core/src/main/scala/spark/MapOutputTracker.scala core/src/main/scala/spark/PairRDDFunctions.scala core/src/main/scala/spark/ParallelCollection.scala core/src/main/scala/spark/RDD.scala core/src/main/scala/spark/rdd/BlockRDD.scala core/src/main/scala/spark/rdd/CartesianRDD.scala core/src/main/scala/spark/rdd/CoGroupedRDD.scala core/src/main/scala/spark/rdd/CoalescedRDD.scala core/src/main/scala/spark/rdd/FilteredRDD.scala core/src/main/scala/spark/rdd/FlatMappedRDD.scala core/src/main/scala/spark/rdd/GlommedRDD.scala core/src/main/scala/spark/rdd/HadoopRDD.scala core/src/main/scala/spark/rdd/MapPartitionsRDD.scala core/src/main/scala/spark/rdd/MapPartitionsWithSplitRDD.scala core/src/main/scala/spark/rdd/MappedRDD.scala core/src/main/scala/spark/rdd/PipedRDD.scala core/src/main/scala/spark/rdd/SampledRDD.scala core/src/main/scala/spark/rdd/ShuffledRDD.scala core/src/main/scala/spark/rdd/UnionRDD.scala core/src/main/scala/spark/storage/BlockManager.scala core/src/main/scala/spark/storage/BlockManagerId.scala core/src/main/scala/spark/storage/BlockManagerMaster.scala core/src/main/scala/spark/storage/StorageLevel.scala core/src/main/scala/spark/util/MetadataCleaner.scala core/src/main/scala/spark/util/TimeStampedHashMap.scala core/src/test/scala/spark/storage/BlockManagerSuite.scala run
| * | | Merge pull request #327 from rxin/spark-633Matei Zaharia2012-12-2016-769/+1096
| |\ \ \ | | | | | | | | | | Added the ability in block manager to remove blocks.
| | * | | Let the slave notify the master block removal.Reynold Xin2012-12-204-92/+83
| | | | |
| | * | | Moved BlockManager's IdGenerator into BlockManager object. Removed someReynold Xin2012-12-195-36/+22
| | | | | | | | | | | | | | | | | | | | excessive debug messages.
| | * | | Merge branch 'spark-633' of github.com:rxin/spark into spark-633Reynold Xin2012-12-144-7/+23
| | |\ \ \
| | | * | | Return success or failure in BlockStore.remove().Josh Rosen2012-12-134-7/+23
| | | | | |
| | * | | | Fixed conflicts from merging Charles' and TD's block manager changes.Reynold Xin2012-12-143-173/+158
| | | | | |
| | * | | | Merge branch 'master' of github.com:mesos/spark into spark-633Reynold Xin2012-12-131-7/+9
| | |\ \ \ \
| | * | | | | Merged TD's block manager refactoring.Reynold Xin2012-12-139-647/+805
| | | | | | |
| | * | | | | Merge branch 'master' of github.com:mesos/spark into spark-633Reynold Xin2012-12-138-15/+118
| | |\ \ \ \ \
| | * \ \ \ \ \ Merge branch 'master' of github.com:mesos/spark into spark-633Reynold Xin2012-12-1326-208/+207
| | |\ \ \ \ \ \ | | | |_|_|/ / / | | |/| | | | |
| | * | | | | | Added the ability in block manager to remove blocks.Reynold Xin2012-12-139-170/+361
| | | | | | | |
| * | | | | | | SPARK-616: Logging dead workers in Web UI.Patrick Wendell2012-12-175-3/+19
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch keeps track of which workers have died and marks them as such in the master web UI. It also handles workers which die and re-register using different actor ID's.
| * | | | | | | Merge pull request #332 from JoshRosen/spark-607Matei Zaharia2012-12-141-12/+17
| |\ \ \ \ \ \ \ | | | | | | | | | | | | | | | | | | Add try-finally to handle MapOutputTracker timeouts
| | * | | | | | | Add try-finally to handle MapOutputTracker timeouts.Josh Rosen2012-12-131-12/+17
| | |/ / / / / /
| * | | | | | | Code review cleanup.Charles Reiss2012-12-131-1/+4
| | | | | | | |
| * | | | | | | Have standalone cluster report exit codes to clients. Addresses SPARK-639.Charles Reiss2012-12-138-24/+20
| | |_|_|/ / / | |/| | | | |
| * | | | | | Fixed the broken Java unit test from SPARK-635.Reynold Xin2012-12-131-7/+9
| | |_|/ / / | |/| | | |
| * | | | | Merge pull request #329 from woggling/executor-status-codesMatei Zaharia2012-12-138-15/+118
| |\ \ \ \ \ | | |_|/ / / | |/| | | | Executor exit status codes
| | * | | | ExecutorLostReason -> ExecutorLossReasonCharles Reiss2012-12-134-6/+6
| | | | | |
| | * | | | Extra whitespace in ExecutorExitCodeCharles Reiss2012-12-131-0/+3
| | | | | |
| | * | | | Explain slaveLost calls made by StandaloneSchedulerBackendCharles Reiss2012-12-131-5/+5
| | | | | |
| | * | | | Log duplicate slaveLost() calls in ClusterScheduler.Charles Reiss2012-12-131-1/+7
| | | | | |
| | * | | | Normalize executor exit statuses and report them to the user.Charles Reiss2012-12-138-11/+94
| | | | | |
| | * | | | Call slaveLost on executor death for standalone clusters.Charles Reiss2012-12-121-1/+12
| | |/ / /
| * | | | SPARK-635: Pass a TaskContext object to compute() interface and useReynold Xin2012-12-1320-49/+46
| | | | | | | | | | | | | | | | | | | | that to close Hadoop input stream. Incorporated Matei's command.
| * | | | SPARK-635: Pass a TaskContext object to compute() interface and use thatReynold Xin2012-12-1326-205/+207
| |/ / / | | | | | | | | | | | | to close Hadoop input stream.
| * | | Suppress shuffle block updates when a slave node comes back.Reynold Xin2012-12-102-12/+17
| | | |
| * | | Merge pull request #317 from woggling/block-manager-heartbeatMatei Zaharia2012-12-106-62/+318
| |\ \ \ | | | | | | | | | | Implement block manager heartbeat
| | * | | Decrease BlockManagerMaster logging verbosityCharles Reiss2012-12-101-1/+2
| | | | |
generated by cgit v1.2.3 (git 2.39.1) at 2024-09-13 02:15:41 +0000