aboutsummaryrefslogtreecommitdiff
path: root/core
Commit message (Collapse)AuthorAgeFilesLines
* Merge branch 'master' into scala-2.9Matei Zaharia2011-07-141-1/+1
|\
| * Lowered default number of splits for filesMatei Zaharia2011-07-141-1/+1
| |
* | Merge branch 'master' into scala-2.9Matei Zaharia2011-07-142-0/+9
|\|
| * Set class loader for remote actors to fix a bug that happens in 2.9Matei Zaharia2011-07-142-0/+9
| |
* | Merge branch 'master' into scala-2.9Matei Zaharia2011-07-146-47/+48
|\|
| * Renamed ParallelArray to ParallelCollectionMatei Zaharia2011-07-143-30/+27
| |
| * Remove RDD.toString because it looked confusingMatei Zaharia2011-07-141-4/+0
| |
| * Fix tracking of updates in accumulators to solve an issue that would ↵Matei Zaharia2011-07-142-13/+21
| | | | | | | | manifest in the 2.9 interpreter
* | Merge branch 'master' into scala-2.9Matei Zaharia2011-07-141-0/+170
|\|
| * Forgot to add a fileMatei Zaharia2011-07-141-0/+170
| |
* | Merge branch 'master' into scala-2.9Matei Zaharia2011-07-1410-275/+114
|\|
| * Cleaned up a few issues to do with default parallelism levels. AlsoMatei Zaharia2011-07-1410-275/+114
| | | | | | | | | | renamed HadoopFileWriter to HadoopWriter (since it's not only for files) and fixed a bug for lookup().
* | Merge branch 'master' into scala-2.9Matei Zaharia2011-07-143-8/+77
|\|
| * Simplified and documented code a little and added testMatei Zaharia2011-07-143-31/+70
| |
| * Merge branch 'master' into implicit-sequencefileMatei Zaharia2011-07-1317-395/+714
| |\
| * | Initial work to make stuff like sequenceFile[Int, Int] work withoutMatei Zaharia2011-06-281-7/+37
| | | | | | | | | | | | | | | requiring the user to provide a Writable type. The approach here might not be the best but it seems to work correctly.
* | | Merge branch 'master' into scala-2.9Matei Zaharia2011-07-137-93/+162
|\ \ \ | | |/ | |/| | | | | | | Conflicts: core/src/main/scala/spark/HadoopFileWriter.scala
| * | Updated save code to allow non-file-based OutputFormats and added a testMatei Zaharia2011-07-136-91/+160
| | | | | | | | | | | | for file-related stuff
| * | Increase default value of spark.locality.wait a littleMatei Zaharia2011-07-131-1/+1
| | |
* | | Merge branch 'master' into scala-2.9Matei Zaharia2011-07-132-10/+49
|\| |
| * | Added mapPartitions operation and a bunch of tests for RDD opsMatei Zaharia2011-07-132-10/+49
| | |
* | | Merge branch 'master' into scala-2.9Matei Zaharia2011-07-1112-313/+516
|\| | | | | | | | | | | | | | Conflicts: core/src/main/scala/spark/RDD.scala
| * | Simplified parallel shuffle fetcher to use URLConnectionMatei Zaharia2011-07-111-44/+24
| | |
| * | Moved PairRDD and SequenceFileRDD functions to separate source filesMatei Zaharia2011-07-104-292/+371
| | |
| * | bug fixMatei Zaharia2011-07-091-32/+34
| | |
| * | Register byte[] with Kryo serializerMatei Zaharia2011-07-091-1/+1
| | |
| * | Added parallel shuffle fetcherMatei Zaharia2011-07-098-20/+162
| | |
| * | Support for non-filesystem-based Hadoop data sourcesMatei Zaharia2011-07-062-24/+32
| |/
* | Support for non-filesystem-based Hadoop data sourcesMatei Zaharia2011-07-062-24/+32
| |
* | Merge remote-tracking branch 'origin/master' into scala-2.9Matei Zaharia2011-06-271-1/+2
|\|
| * Don't pass a null context when running tasks locallyMatei Zaharia2011-06-271-1/+2
| |
* | Fixed HadoopFileWriter to compile for Scala 2.9Matei Zaharia2011-06-271-1/+2
| |
* | Merge branch 'master' into scala-2.9Matei Zaharia2011-06-2710-17/+399
|\|
| * Fix a compile errorMatei Zaharia2011-06-271-1/+1
| |
| * Merge branch 'master' into td-rdd-saveTathagata Das2011-06-2717-2275/+3381
| |\ | | | | | | | | | | | | Conflicts: core/src/main/scala/spark/SparkContext.scala
| * \ Merge branch 'master' into td-rdd-saveTathagata Das2011-06-272-0/+84
| |\ \ | | | | | | | | | | | | | | | | Conflicts: core/src/main/scala/spark/RDD.scala
| * | | Further changes to HadoopFileWriter. Implemented ability to save RDDs as ↵Tathagata Das2011-06-243-80/+213
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | SequenceFiles and ObjectFiles. 1> HadoopFileWriter changed to take class types as constructor parameters (no more generic type) 2> Multiple types of RDD.saveAsHadoopFile() implemented to provide more saving options 3> RDD.saveAsSequenceFile() automatically converts basic types to Writable types before saving as SequenceFile 4> RDD.saveAsObjectFile() serializes objects and saves them to a ObjectFile 5> SparkContext.objectFile() opens the saved ObjectFiles
| * | | Improved HadoopFileWriter (saves key and value classes to jobconf)Tathagata Das2011-06-231-5/+11
| | | |
| * | | Cleaner reimplementation of HadoopFileWriter. Introduced TaskContext.Tathagata Das2011-06-168-97/+184
| | | | | | | | | | | | | | | | | | | | | | | | | | | | 1> HadoopFileWriter works correctly with task failures 2> It can also take an user specified JobConf object for configuration settings 3> A Task can now get information like stage ID, split ID, and attempt ID using TaskContext class 4> Minor changes in SparkContext, DAGScheduler and subclasses to allow specification of TaskContext as a parameter
| * | | Implemented TaskContext to hold contextual information (jobID, taskID, ↵Tathagata Das2011-06-106-8/+34
| | | | | | | | | | | | | | | | attemptID) of a task
| * | | HadoopFileWriter changed to use Hadoop's OutputCommitterTathagata Das2011-06-092-78/+42
| | | |
| * | | First-cut implementation of RDD.SaveAsTextTathagata Das2011-06-052-0/+165
| | | |
* | | | Merge branch 'master' into scala-2.9Matei Zaharia2011-06-2617-2275/+3397
|\ \ \ \ | | |_|/ | |/| | | | | | | | | | Conflicts: repl/src/main/scala/spark/repl/SparkInterpreterLoop.scala
| * | | Merge branch 'mos-bt'Matei Zaharia2011-06-2615-2274/+3285
| |\ \ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This merge keeps only the broadcast work in mos-bt because the structure of shuffle has changed with the new RDD design. We still need some kind of parallel shuffle but that will be added later. Conflicts: core/src/main/scala/spark/BitTorrentBroadcast.scala core/src/main/scala/spark/ChainedBroadcast.scala core/src/main/scala/spark/RDD.scala core/src/main/scala/spark/SparkContext.scala core/src/main/scala/spark/Utils.scala core/src/main/scala/spark/shuffle/BasicLocalFileShuffle.scala core/src/main/scala/spark/shuffle/DfsShuffle.scala
| | * | | Issue #42 fixed.Mosharaf Chowdhury2011-04-287-13/+19
| | | | |
| | * | | Refactoring: daemonThreadFactories have all been moved to the UtilsMosharaf Chowdhury2011-04-279-86/+54
| | | | | | | | | | | | | | | | | | | | | | | | | object instead of having multiple copies in Broadcast and Shuffle objects.
| | * | | Cleanup + refactoring...Mosharaf Chowdhury2011-04-276-112/+41
| | | | |
| | * | | Shuffle is also working from its own subpackage.Mosharaf Chowdhury2011-04-2710-12/+30
| | | | |
| | * | | Removed some shuffle implementations. Remaining ones all use local filesMosharaf Chowdhury2011-04-279-5723/+0
| | | | | | | | | | | | | | | | | | | | to write map outputs.
| | * | | Merge branch 'mos-shuffle-tracked' into mos-btMosharaf Chowdhury2011-04-2715-12/+7444
| | |\ \ \ | | | | | | | | | | | | | | | | | | | | | | | | Conflicts: core/src/main/scala/spark/Broadcast.scala
generated by cgit v1.2.3 (git 2.39.1) at 2024-09-11 13:22:56 +0000