Commit message (Collapse) | Author | Age | Files | Lines | |
---|---|---|---|---|---|
* | Merge branch 'master' into scala-2.9 | Matei Zaharia | 2011-07-14 | 1 | -1/+1 |
|\ | |||||
| * | Lowered default number of splits for files | Matei Zaharia | 2011-07-14 | 1 | -1/+1 |
| | | |||||
* | | Merge branch 'master' into scala-2.9 | Matei Zaharia | 2011-07-14 | 2 | -0/+9 |
|\| | |||||
| * | Set class loader for remote actors to fix a bug that happens in 2.9 | Matei Zaharia | 2011-07-14 | 2 | -0/+9 |
| | | |||||
* | | Merge branch 'master' into scala-2.9 | Matei Zaharia | 2011-07-14 | 6 | -47/+48 |
|\| | |||||
| * | Renamed ParallelArray to ParallelCollection | Matei Zaharia | 2011-07-14 | 3 | -30/+27 |
| | | |||||
| * | Remove RDD.toString because it looked confusing | Matei Zaharia | 2011-07-14 | 1 | -4/+0 |
| | | |||||
| * | Fix tracking of updates in accumulators to solve an issue that would ↵ | Matei Zaharia | 2011-07-14 | 2 | -13/+21 |
| | | | | | | | | manifest in the 2.9 interpreter | ||||
* | | Merge branch 'master' into scala-2.9 | Matei Zaharia | 2011-07-14 | 1 | -0/+170 |
|\| | |||||
| * | Forgot to add a file | Matei Zaharia | 2011-07-14 | 1 | -0/+170 |
| | | |||||
* | | Merge branch 'master' into scala-2.9 | Matei Zaharia | 2011-07-14 | 10 | -275/+114 |
|\| | |||||
| * | Cleaned up a few issues to do with default parallelism levels. Also | Matei Zaharia | 2011-07-14 | 10 | -275/+114 |
| | | | | | | | | | | renamed HadoopFileWriter to HadoopWriter (since it's not only for files) and fixed a bug for lookup(). | ||||
* | | Merge branch 'master' into scala-2.9 | Matei Zaharia | 2011-07-14 | 3 | -8/+77 |
|\| | |||||
| * | Simplified and documented code a little and added test | Matei Zaharia | 2011-07-14 | 3 | -31/+70 |
| | | |||||
| * | Merge branch 'master' into implicit-sequencefile | Matei Zaharia | 2011-07-13 | 17 | -395/+714 |
| |\ | |||||
| * | | Initial work to make stuff like sequenceFile[Int, Int] work without | Matei Zaharia | 2011-06-28 | 1 | -7/+37 |
| | | | | | | | | | | | | | | | requiring the user to provide a Writable type. The approach here might not be the best but it seems to work correctly. | ||||
* | | | Merge branch 'master' into scala-2.9 | Matei Zaharia | 2011-07-13 | 7 | -93/+162 |
|\ \ \ | | |/ | |/| | | | | | | | Conflicts: core/src/main/scala/spark/HadoopFileWriter.scala | ||||
| * | | Updated save code to allow non-file-based OutputFormats and added a test | Matei Zaharia | 2011-07-13 | 6 | -91/+160 |
| | | | | | | | | | | | | for file-related stuff | ||||
| * | | Increase default value of spark.locality.wait a little | Matei Zaharia | 2011-07-13 | 1 | -1/+1 |
| | | | |||||
* | | | Merge branch 'master' into scala-2.9 | Matei Zaharia | 2011-07-13 | 2 | -10/+49 |
|\| | | |||||
| * | | Added mapPartitions operation and a bunch of tests for RDD ops | Matei Zaharia | 2011-07-13 | 2 | -10/+49 |
| | | | |||||
* | | | Merge branch 'master' into scala-2.9 | Matei Zaharia | 2011-07-11 | 12 | -313/+516 |
|\| | | | | | | | | | | | | | | Conflicts: core/src/main/scala/spark/RDD.scala | ||||
| * | | Simplified parallel shuffle fetcher to use URLConnection | Matei Zaharia | 2011-07-11 | 1 | -44/+24 |
| | | | |||||
| * | | Moved PairRDD and SequenceFileRDD functions to separate source files | Matei Zaharia | 2011-07-10 | 4 | -292/+371 |
| | | | |||||
| * | | bug fix | Matei Zaharia | 2011-07-09 | 1 | -32/+34 |
| | | | |||||
| * | | Register byte[] with Kryo serializer | Matei Zaharia | 2011-07-09 | 1 | -1/+1 |
| | | | |||||
| * | | Added parallel shuffle fetcher | Matei Zaharia | 2011-07-09 | 8 | -20/+162 |
| | | | |||||
| * | | Support for non-filesystem-based Hadoop data sources | Matei Zaharia | 2011-07-06 | 2 | -24/+32 |
| |/ | |||||
* | | Support for non-filesystem-based Hadoop data sources | Matei Zaharia | 2011-07-06 | 2 | -24/+32 |
| | | |||||
* | | Merge remote-tracking branch 'origin/master' into scala-2.9 | Matei Zaharia | 2011-06-27 | 1 | -1/+2 |
|\| | |||||
| * | Don't pass a null context when running tasks locally | Matei Zaharia | 2011-06-27 | 1 | -1/+2 |
| | | |||||
* | | Fixed HadoopFileWriter to compile for Scala 2.9 | Matei Zaharia | 2011-06-27 | 1 | -1/+2 |
| | | |||||
* | | Merge branch 'master' into scala-2.9 | Matei Zaharia | 2011-06-27 | 10 | -17/+399 |
|\| | |||||
| * | Fix a compile error | Matei Zaharia | 2011-06-27 | 1 | -1/+1 |
| | | |||||
| * | Merge branch 'master' into td-rdd-save | Tathagata Das | 2011-06-27 | 17 | -2275/+3381 |
| |\ | | | | | | | | | | | | | Conflicts: core/src/main/scala/spark/SparkContext.scala | ||||
| * \ | Merge branch 'master' into td-rdd-save | Tathagata Das | 2011-06-27 | 2 | -0/+84 |
| |\ \ | | | | | | | | | | | | | | | | | Conflicts: core/src/main/scala/spark/RDD.scala | ||||
| * | | | Further changes to HadoopFileWriter. Implemented ability to save RDDs as ↵ | Tathagata Das | 2011-06-24 | 3 | -80/+213 |
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | SequenceFiles and ObjectFiles. 1> HadoopFileWriter changed to take class types as constructor parameters (no more generic type) 2> Multiple types of RDD.saveAsHadoopFile() implemented to provide more saving options 3> RDD.saveAsSequenceFile() automatically converts basic types to Writable types before saving as SequenceFile 4> RDD.saveAsObjectFile() serializes objects and saves them to a ObjectFile 5> SparkContext.objectFile() opens the saved ObjectFiles | ||||
| * | | | Improved HadoopFileWriter (saves key and value classes to jobconf) | Tathagata Das | 2011-06-23 | 1 | -5/+11 |
| | | | | |||||
| * | | | Cleaner reimplementation of HadoopFileWriter. Introduced TaskContext. | Tathagata Das | 2011-06-16 | 8 | -97/+184 |
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | 1> HadoopFileWriter works correctly with task failures 2> It can also take an user specified JobConf object for configuration settings 3> A Task can now get information like stage ID, split ID, and attempt ID using TaskContext class 4> Minor changes in SparkContext, DAGScheduler and subclasses to allow specification of TaskContext as a parameter | ||||
| * | | | Implemented TaskContext to hold contextual information (jobID, taskID, ↵ | Tathagata Das | 2011-06-10 | 6 | -8/+34 |
| | | | | | | | | | | | | | | | | attemptID) of a task | ||||
| * | | | HadoopFileWriter changed to use Hadoop's OutputCommitter | Tathagata Das | 2011-06-09 | 2 | -78/+42 |
| | | | | |||||
| * | | | First-cut implementation of RDD.SaveAsText | Tathagata Das | 2011-06-05 | 2 | -0/+165 |
| | | | | |||||
* | | | | Merge branch 'master' into scala-2.9 | Matei Zaharia | 2011-06-26 | 17 | -2275/+3397 |
|\ \ \ \ | | |_|/ | |/| | | | | | | | | | | Conflicts: repl/src/main/scala/spark/repl/SparkInterpreterLoop.scala | ||||
| * | | | Merge branch 'mos-bt' | Matei Zaharia | 2011-06-26 | 15 | -2274/+3285 |
| |\ \ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This merge keeps only the broadcast work in mos-bt because the structure of shuffle has changed with the new RDD design. We still need some kind of parallel shuffle but that will be added later. Conflicts: core/src/main/scala/spark/BitTorrentBroadcast.scala core/src/main/scala/spark/ChainedBroadcast.scala core/src/main/scala/spark/RDD.scala core/src/main/scala/spark/SparkContext.scala core/src/main/scala/spark/Utils.scala core/src/main/scala/spark/shuffle/BasicLocalFileShuffle.scala core/src/main/scala/spark/shuffle/DfsShuffle.scala | ||||
| | * | | | Issue #42 fixed. | Mosharaf Chowdhury | 2011-04-28 | 7 | -13/+19 |
| | | | | | |||||
| | * | | | Refactoring: daemonThreadFactories have all been moved to the Utils | Mosharaf Chowdhury | 2011-04-27 | 9 | -86/+54 |
| | | | | | | | | | | | | | | | | | | | | | | | | | object instead of having multiple copies in Broadcast and Shuffle objects. | ||||
| | * | | | Cleanup + refactoring... | Mosharaf Chowdhury | 2011-04-27 | 6 | -112/+41 |
| | | | | | |||||
| | * | | | Shuffle is also working from its own subpackage. | Mosharaf Chowdhury | 2011-04-27 | 10 | -12/+30 |
| | | | | | |||||
| | * | | | Removed some shuffle implementations. Remaining ones all use local files | Mosharaf Chowdhury | 2011-04-27 | 9 | -5723/+0 |
| | | | | | | | | | | | | | | | | | | | | to write map outputs. | ||||
| | * | | | Merge branch 'mos-shuffle-tracked' into mos-bt | Mosharaf Chowdhury | 2011-04-27 | 15 | -12/+7444 |
| | |\ \ \ | | | | | | | | | | | | | | | | | | | | | | | | | Conflicts: core/src/main/scala/spark/Broadcast.scala |