Commit message (Collapse) | Author | Age | Files | Lines | |
---|---|---|---|---|---|
* | Merge remote-tracking branch 'public/master' into dev | Matei Zaharia | 2012-10-24 | 193 | -2097/+5043 |
|\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Conflicts: core/src/main/scala/spark/BlockStoreShuffleFetcher.scala core/src/main/scala/spark/KryoSerializer.scala core/src/main/scala/spark/MapOutputTracker.scala core/src/main/scala/spark/RDD.scala core/src/main/scala/spark/SparkContext.scala core/src/main/scala/spark/executor/Executor.scala core/src/main/scala/spark/network/Connection.scala core/src/main/scala/spark/network/ConnectionManagerTest.scala core/src/main/scala/spark/rdd/BlockRDD.scala core/src/main/scala/spark/rdd/NewHadoopRDD.scala core/src/main/scala/spark/scheduler/ShuffleMapTask.scala core/src/main/scala/spark/scheduler/cluster/StandaloneSchedulerBackend.scala core/src/main/scala/spark/storage/BlockManager.scala core/src/main/scala/spark/storage/BlockMessage.scala core/src/main/scala/spark/storage/BlockStore.scala core/src/main/scala/spark/storage/StorageLevel.scala core/src/main/scala/spark/util/AkkaUtils.scala project/SparkBuild.scala run | ||||
| * | Strip leading mesos:// in URLs passed to Mesos | Matei Zaharia | 2012-10-24 | 1 | -2/+3 |
| | | |||||
| * | Merge pull request #281 from rxin/memreport | Matei Zaharia | 2012-10-23 | 3 | -71/+93 |
| |\ | | | | | | | Added a method to report slave memory status; force serialize accumulator update in local mode. | ||||
| | * | Serialize accumulator updates in TaskResult for local mode. | Reynold Xin | 2012-10-15 | 1 | -4/+5 |
| | | | |||||
| | * | Added a method to report slave memory status. | Reynold Xin | 2012-10-14 | 2 | -67/+88 |
| | | | |||||
| * | | Merge remote-tracking branch 'JoshRosen/shuffle_refactoring' into dev | Matei Zaharia | 2012-10-23 | 13 | -250/+113 |
| |\ \ | | | | | | | | | | | | | | | | | | | | | | | | | Conflicts: core/src/main/scala/spark/Dependency.scala core/src/main/scala/spark/rdd/CoGroupedRDD.scala core/src/main/scala/spark/rdd/ShuffledRDD.scala | ||||
| | * | | Remove map-side combining from ShuffleMapTask. | Josh Rosen | 2012-10-13 | 8 | -94/+37 |
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This separation of concerns simplifies the ShuffleDependency and ShuffledRDD interfaces. Map-side combining can be performed in a mapPartitions() call prior to shuffling the RDD. I don't anticipate this having much of a performance impact: in both approaches, each tuple is hashed twice: once in the bucket partitioning and once in the combiner's hashtable. The same steps are being performed, but in a different order and through one extra Iterator. | ||||
| | * | | Remove mapSideCombine field from Aggregator. | Josh Rosen | 2012-10-13 | 5 | -22/+15 |
| | | | | | | | | | | | | | | | | | | | | Instead, the presence or absense of a ShuffleDependency's aggregator will control whether map-side combining is performed. | ||||
| | * | | Change ShuffleFetcher to return an Iterator. | Josh Rosen | 2012-10-13 | 8 | -167/+63 |
| | | | | |||||
| | * | | Add helper methods to Aggregator. | Josh Rosen | 2012-10-13 | 1 | -1/+32 |
| | | | | |||||
| * | | | Support for Hadoop 2 distributions such as cdh4 | Thomas Dudziak | 2012-10-18 | 7 | -20/+45 |
| | |/ | |/| | |||||
| * | | Made ShuffleDependency automatically find a shuffle ID for itself | Matei Zaharia | 2012-10-14 | 3 | -5/+6 |
| | | | |||||
| * | | Take executor environment vars as an arguemnt to SparkContext | Matei Zaharia | 2012-10-13 | 7 | -79/+107 |
| |/ | |||||
| * | Protect from null env variables in mesos. | Denny | 2012-10-13 | 2 | -8/+16 |
| | | |||||
| * | Protect from setting null environment variables. | Denny | 2012-10-13 | 1 | -1/+5 |
| | | |||||
| * | Don't use system envs for Mesos. | Denny | 2012-10-13 | 2 | -2/+2 |
| | | |||||
| * | Let the user specify environment variables to be passed to the Executors. | Denny | 2012-10-13 | 5 | -48/+21 |
| | | | | | | | | Also removed unused variables in the ExecutorRunner. | ||||
| * | More doc updates, and moved Serializer to a subpackage. | Matei Zaharia | 2012-10-12 | 12 | -25/+51 |
| | | |||||
| * | Some doc and usability improvements: | Matei Zaharia | 2012-10-12 | 11 | -22/+83 |
| | | | | | | | | | | | | | | - Added a StorageLevels class for easy access to StorageLevel constants in Java - Added doc comments on Function classes in Java - Updated Accumulator and HadoopWriter docs slightly | ||||
| * | Added a test for when an RDD only partially fits in memory | Matei Zaharia | 2012-10-12 | 1 | -2/+18 |
| | | |||||
| * | Document cartesian() operation | Matei Zaharia | 2012-10-12 | 2 | -0/+8 |
| | | |||||
| * | Merge pull request #271 from shivaram/block-manager-npe-fix | Matei Zaharia | 2012-10-12 | 8 | -39/+58 |
| |\ | | | | | | | Change block manager to accept a ArrayBuffer | ||||
| | * | Add test to verify if RDD is computed even if block manager has insufficient | Shivaram Venkataraman | 2012-10-12 | 1 | -0/+10 |
| | | | | | | | | | | | | memory | ||||
| | * | Change block manager to accept a ArrayBuffer instead of an iterator to ensure | Shivaram Venkataraman | 2012-10-11 | 7 | -39/+48 |
| | | | | | | | | | | | | | | | that the computation can proceed even if we run out of memory to cache the block. Update CacheTracker to use this new interface | ||||
| * | | Adding Java documentation | Patrick Wendell | 2012-10-11 | 6 | -10/+454 |
| |/ | |||||
| * | Fixed bug when fetching Jar dependencies. | Denny | 2012-10-10 | 2 | -6/+6 |
| | | | | | | | | Instead of checking currentFiles check currentJars. | ||||
| * | Added documentation to all the *RDDFunction classes, and moved them into | Matei Zaharia | 2012-10-09 | 9 | -54/+273 |
| | | | | | | | | | | the spark package to make them more visible. Also documented various other miscellaneous things in the API. | ||||
| * | Updates to documentation: | Matei Zaharia | 2012-10-09 | 1 | -1/+1 |
| | | | | | | | | | | | | | | | | - Edited quick start and tuning guide to simplify them a little - Simplified top menu bar - Made private a SparkContext constructor parameter that was left as public - Various small fixes | ||||
| * | Fixes a typo, adds scaladoc comments to SparkContext constructors. | Andy Konwinski | 2012-10-08 | 2 | -5/+11 |
| | | |||||
| * | More docs in RDD class | Patrick Wendell | 2012-10-08 | 1 | -1/+45 |
| | | |||||
| * | A start on scaladoc for the public APIs. | Andy Konwinski | 2012-10-08 | 1 | -6/+29 |
| | | |||||
| * | Merge branch 'dev' into bc-fix-dev | Mosharaf Chowdhury | 2012-10-08 | 70 | -477/+1108 |
| |\ | |||||
| | * | Made compression configurable separately for shuffle, broadcast and RDDs | Matei Zaharia | 2012-10-07 | 4 | -38/+118 |
| | | | |||||
| | * | Merge pull request #251 from JoshRosen/docs/internals | Matei Zaharia | 2012-10-07 | 5 | -11/+40 |
| | |\ | | | | | | | | | Document Dependency classes and make minor interface improvements | ||||
| | | * | Make ShuffleDependency.aggregator explicitly optional. | Josh Rosen | 2012-10-07 | 4 | -7/+11 |
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | It was confusing to be using new Aggregator[K, V, V](null, null, null, false) to represent the absence of an aggregator. | ||||
| | | * | Document the Dependency classes. | Josh Rosen | 2012-10-07 | 2 | -1/+26 |
| | | | | |||||
| | | * | Remove unused isShuffle field from Dependency. | Josh Rosen | 2012-10-07 | 1 | -3/+3 |
| | | | | |||||
| | * | | Changed the println to logInfo in Utils.fetchFile. | Reynold Xin | 2012-10-07 | 1 | -1/+1 |
| | | | | |||||
| | * | | Merge pull request #250 from rxin/dev | Matei Zaharia | 2012-10-07 | 2 | -18/+40 |
| | |\ \ | | | | | | | | | | | Fixed a bug in addFile that if the file is specified as "file:///", the symlink is created incorrectly for local mode. | ||||
| | | * | | Fixed a bug in addFile that if the file is specified as "file:///", the | Reynold Xin | 2012-10-07 | 2 | -18/+40 |
| | | | | | | | | | | | | | | | | | | | | symlink is created wrong for local mode. | ||||
| | * | | | Improve error message | Matei Zaharia | 2012-10-07 | 1 | -1/+1 |
| | | | | | |||||
| | * | | | Don't crash on ask timeout exceptions in deploy.Client.stop() (fixes a crash ↵ | Matei Zaharia | 2012-10-07 | 1 | -3/+8 |
| | |/ / | | | | | | | | | | | | | in tests) | ||||
| | * / | Removed the need to sleep in tests due to waiting for Akka to shut down | Matei Zaharia | 2012-10-07 | 15 | -23/+40 |
| | |/ | |||||
| | * | Log message | Matei Zaharia | 2012-10-07 | 1 | -1/+1 |
| | | | |||||
| | * | More logging | Matei Zaharia | 2012-10-07 | 1 | -4/+7 |
| | | | |||||
| | * | Log more info in MapOutputTracker | root | 2012-10-07 | 1 | -4/+7 |
| | | | |||||
| | * | Made Akka thread pool and message batch sizes configurable | root | 2012-10-07 | 1 | -3/+5 |
| | | | |||||
| | * | Made run script add test-classes onto the classpath only if SPARK_TESTING is ↵ | root | 2012-10-07 | 3 | -3/+6 |
| | | | | | | | | | | | | set; fixes #216 | ||||
| | * | Avoid acquiring locks in BlockManager when fetching shuffle outputs | root | 2012-10-07 | 1 | -0/+24 |
| | | | |||||
| | * | Log initial number of fetches in reducer | root | 2012-10-07 | 1 | -1/+2 |
| | | |