aboutsummaryrefslogtreecommitdiff
path: root/core
Commit message (Collapse)AuthorAgeFilesLines
* Merge remote-tracking branch 'public/master' into devMatei Zaharia2012-10-24194-2097/+5043
|\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Conflicts: core/src/main/scala/spark/BlockStoreShuffleFetcher.scala core/src/main/scala/spark/KryoSerializer.scala core/src/main/scala/spark/MapOutputTracker.scala core/src/main/scala/spark/RDD.scala core/src/main/scala/spark/SparkContext.scala core/src/main/scala/spark/executor/Executor.scala core/src/main/scala/spark/network/Connection.scala core/src/main/scala/spark/network/ConnectionManagerTest.scala core/src/main/scala/spark/rdd/BlockRDD.scala core/src/main/scala/spark/rdd/NewHadoopRDD.scala core/src/main/scala/spark/scheduler/ShuffleMapTask.scala core/src/main/scala/spark/scheduler/cluster/StandaloneSchedulerBackend.scala core/src/main/scala/spark/storage/BlockManager.scala core/src/main/scala/spark/storage/BlockMessage.scala core/src/main/scala/spark/storage/BlockStore.scala core/src/main/scala/spark/storage/StorageLevel.scala core/src/main/scala/spark/util/AkkaUtils.scala project/SparkBuild.scala run
| * Strip leading mesos:// in URLs passed to MesosMatei Zaharia2012-10-241-2/+3
| |
| * Merge pull request #281 from rxin/memreportMatei Zaharia2012-10-233-71/+93
| |\ | | | | | | Added a method to report slave memory status; force serialize accumulator update in local mode.
| | * Serialize accumulator updates in TaskResult for local mode.Reynold Xin2012-10-151-4/+5
| | |
| | * Added a method to report slave memory status.Reynold Xin2012-10-142-67/+88
| | |
| * | Merge remote-tracking branch 'JoshRosen/shuffle_refactoring' into devMatei Zaharia2012-10-2313-250/+113
| |\ \ | | | | | | | | | | | | | | | | | | | | | | | | Conflicts: core/src/main/scala/spark/Dependency.scala core/src/main/scala/spark/rdd/CoGroupedRDD.scala core/src/main/scala/spark/rdd/ShuffledRDD.scala
| | * | Remove map-side combining from ShuffleMapTask.Josh Rosen2012-10-138-94/+37
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This separation of concerns simplifies the ShuffleDependency and ShuffledRDD interfaces. Map-side combining can be performed in a mapPartitions() call prior to shuffling the RDD. I don't anticipate this having much of a performance impact: in both approaches, each tuple is hashed twice: once in the bucket partitioning and once in the combiner's hashtable. The same steps are being performed, but in a different order and through one extra Iterator.
| | * | Remove mapSideCombine field from Aggregator.Josh Rosen2012-10-135-22/+15
| | | | | | | | | | | | | | | | | | | | Instead, the presence or absense of a ShuffleDependency's aggregator will control whether map-side combining is performed.
| | * | Change ShuffleFetcher to return an Iterator.Josh Rosen2012-10-138-167/+63
| | | |
| | * | Add helper methods to Aggregator.Josh Rosen2012-10-131-1/+32
| | | |
| * | | Support for Hadoop 2 distributions such as cdh4Thomas Dudziak2012-10-187-20/+45
| | |/ | |/|
| * | Made ShuffleDependency automatically find a shuffle ID for itselfMatei Zaharia2012-10-143-5/+6
| | |
| * | Take executor environment vars as an arguemnt to SparkContextMatei Zaharia2012-10-137-79/+107
| |/
| * Protect from null env variables in mesos.Denny2012-10-132-8/+16
| |
| * Protect from setting null environment variables.Denny2012-10-131-1/+5
| |
| * Don't use system envs for Mesos.Denny2012-10-132-2/+2
| |
| * Let the user specify environment variables to be passed to the Executors.Denny2012-10-135-48/+21
| | | | | | | | Also removed unused variables in the ExecutorRunner.
| * More doc updates, and moved Serializer to a subpackage.Matei Zaharia2012-10-1212-25/+51
| |
| * Some doc and usability improvements:Matei Zaharia2012-10-1211-22/+83
| | | | | | | | | | | | | | - Added a StorageLevels class for easy access to StorageLevel constants in Java - Added doc comments on Function classes in Java - Updated Accumulator and HadoopWriter docs slightly
| * Added a test for when an RDD only partially fits in memoryMatei Zaharia2012-10-121-2/+18
| |
| * Document cartesian() operationMatei Zaharia2012-10-122-0/+8
| |
| * Merge pull request #271 from shivaram/block-manager-npe-fixMatei Zaharia2012-10-128-39/+58
| |\ | | | | | | Change block manager to accept a ArrayBuffer
| | * Add test to verify if RDD is computed even if block manager has insufficientShivaram Venkataraman2012-10-121-0/+10
| | | | | | | | | | | | memory
| | * Change block manager to accept a ArrayBuffer instead of an iterator to ensureShivaram Venkataraman2012-10-117-39/+48
| | | | | | | | | | | | | | | that the computation can proceed even if we run out of memory to cache the block. Update CacheTracker to use this new interface
| * | Adding Java documentationPatrick Wendell2012-10-116-10/+454
| |/
| * Fixed bug when fetching Jar dependencies.Denny2012-10-102-6/+6
| | | | | | | | Instead of checking currentFiles check currentJars.
| * Added documentation to all the *RDDFunction classes, and moved them intoMatei Zaharia2012-10-099-54/+273
| | | | | | | | | | the spark package to make them more visible. Also documented various other miscellaneous things in the API.
| * Updates to documentation:Matei Zaharia2012-10-091-1/+1
| | | | | | | | | | | | | | | | - Edited quick start and tuning guide to simplify them a little - Simplified top menu bar - Made private a SparkContext constructor parameter that was left as public - Various small fixes
| * Fixes a typo, adds scaladoc comments to SparkContext constructors.Andy Konwinski2012-10-082-5/+11
| |
| * More docs in RDD classPatrick Wendell2012-10-081-1/+45
| |
| * A start on scaladoc for the public APIs.Andy Konwinski2012-10-081-6/+29
| |
| * Merge branch 'dev' into bc-fix-devMosharaf Chowdhury2012-10-0871-477/+1108
| |\
| | * Made compression configurable separately for shuffle, broadcast and RDDsMatei Zaharia2012-10-074-38/+118
| | |
| | * Merge pull request #251 from JoshRosen/docs/internalsMatei Zaharia2012-10-075-11/+40
| | |\ | | | | | | | | Document Dependency classes and make minor interface improvements
| | | * Make ShuffleDependency.aggregator explicitly optional.Josh Rosen2012-10-074-7/+11
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | It was confusing to be using new Aggregator[K, V, V](null, null, null, false) to represent the absence of an aggregator.
| | | * Document the Dependency classes.Josh Rosen2012-10-072-1/+26
| | | |
| | | * Remove unused isShuffle field from Dependency.Josh Rosen2012-10-071-3/+3
| | | |
| | * | Changed the println to logInfo in Utils.fetchFile.Reynold Xin2012-10-071-1/+1
| | | |
| | * | Merge pull request #250 from rxin/devMatei Zaharia2012-10-072-18/+40
| | |\ \ | | | | | | | | | | Fixed a bug in addFile that if the file is specified as "file:///", the symlink is created incorrectly for local mode.
| | | * | Fixed a bug in addFile that if the file is specified as "file:///", theReynold Xin2012-10-072-18/+40
| | | | | | | | | | | | | | | | | | | | symlink is created wrong for local mode.
| | * | | Improve error messageMatei Zaharia2012-10-071-1/+1
| | | | |
| | * | | Don't crash on ask timeout exceptions in deploy.Client.stop() (fixes a crash ↵Matei Zaharia2012-10-071-3/+8
| | |/ / | | | | | | | | | | | | in tests)
| | * / Removed the need to sleep in tests due to waiting for Akka to shut downMatei Zaharia2012-10-0715-23/+40
| | |/
| | * Log messageMatei Zaharia2012-10-071-1/+1
| | |
| | * More loggingMatei Zaharia2012-10-071-4/+7
| | |
| | * Log more info in MapOutputTrackerroot2012-10-071-4/+7
| | |
| | * Made Akka thread pool and message batch sizes configurableroot2012-10-071-3/+5
| | |
| | * Made run script add test-classes onto the classpath only if SPARK_TESTING is ↵root2012-10-073-3/+6
| | | | | | | | | | | | set; fixes #216
| | * Avoid acquiring locks in BlockManager when fetching shuffle outputsroot2012-10-071-0/+24
| | |
| | * Log initial number of fetches in reducerroot2012-10-071-1/+2
| | |
generated by cgit v1.2.3 (git 2.39.1) at 2024-09-06 17:18:52 +0000