Commit message (Collapse) | Author | Age | Files | Lines | |
---|---|---|---|---|---|
* | Made Serializer and JavaSerializer non private. | Reynold Xin | 2012-10-03 | 2 | -4/+4 |
| | |||||
* | Made StorageLevel public | Matei Zaharia | 2012-10-03 | 1 | -2/+2 |
| | |||||
* | Make more stuff private[spark] | Matei Zaharia | 2012-10-02 | 6 | -4/+11 |
| | |||||
* | Merge pull request #240 from dennybritz/private_classes | Matei Zaharia | 2012-10-02 | 117 | -186/+243 |
|\ | | | | | Package-Private Classes | ||||
| * | Make Java API abstract wrapped functions private | Denny | 2012-10-02 | 2 | -2/+2 |
| | | |||||
| * | accidentially removed trait | Denny | 2012-10-02 | 1 | -1/+1 |
| | | |||||
| * | Stylistic changes and Public Accumulable and Broadcast | Denny | 2012-10-02 | 30 | -59/+117 |
| | | |||||
| * | Make dependency classes public - used by spark | Denny | 2012-10-02 | 1 | -5/+5 |
| | | |||||
| * | Make classes package private | Denny | 2012-10-02 | 119 | -248/+247 |
| | | |||||
* | | Merge branch 'dev' of github.com:mesos/spark into dev | Matei Zaharia | 2012-10-02 | 1 | -36/+47 |
|\ \ | |||||
| * | | Added a check to make sure SPARK_MEM <= memoryPerSlave for local cluster | Reynold Xin | 2012-10-02 | 1 | -2/+13 |
| | | | | | | | | | | | | mode. | ||||
| * | | Merge branch 'dev' of https://github.com/mesos/spark into dev | Reynold Xin | 2012-10-02 | 4 | -225/+175 |
| |\| | |||||
| * | | Allow whitespaces in cluster URL configuration for local cluster. | Reynold Xin | 2012-10-02 | 1 | -34/+34 |
| | | | |||||
* | | | Fixed cache replacement behavior of BlockManager: | Matei Zaharia | 2012-10-02 | 4 | -190/+278 |
| |/ |/| | | | | | | | | | | | | | | | | | | | | | | | - Partitions that get dropped to disk will now be loaded back into RAM after they're accessed again - Same-RDD rule for cache replacement is now implemented (don't drop partitions from an RDD to make room for other partitions from itself) - Items stored as MEMORY_AND_DISK go into memory only first, instead of being eagerly written out to disk - MemoryStore.ensureFreeSpace is called within a lock on the writer thread to prevent race conditions (this can still be optimized to allow multiple concurrent calls to it but it's a start) - MemoryStore does not accept blocks larger than its limit | ||||
* | | Revert "Place Spray repo ahead of Cloudera in Maven search path" | Matei Zaharia | 2012-10-02 | 4 | -225/+175 |
|/ | | | | This reverts commit 42e0a68082327c78dbd0fd313145124d9b8a9d98. | ||||
* | Place Spray repo ahead of Cloudera in Maven search path | Matei Zaharia | 2012-10-02 | 4 | -175/+225 |
| | |||||
* | Include date in folder name for Spark local dir. | Matei Zaharia | 2012-10-01 | 1 | -5/+7 |
| | |||||
* | Merge branch 'dev' of github.com:mesos/spark into dev | Matei Zaharia | 2012-10-01 | 1 | -30/+32 |
|\ | |||||
| * | Fixed #232: DirectBuffer's cleaner was empty and Spark tried to invoke | Reynold Xin | 2012-10-01 | 1 | -30/+32 |
| | | | | | | | | clean on it. | ||||
* | | Some bug fixes and logging fixes for broadcast. | Matei Zaharia | 2012-10-01 | 9 | -107/+111 |
|/ | |||||
* | Improve log messages from BlockManager | Matei Zaharia | 2012-10-01 | 4 | -45/+41 |
| | |||||
* | Use underscores instead of colons in RDD IDs | Matei Zaharia | 2012-10-01 | 5 | -6/+6 |
| | |||||
* | Added a (failing) test for LRU with MEMORY_AND_DISK. | Matei Zaharia | 2012-09-30 | 1 | -2/+2 |
| | |||||
* | Simplified Class / ClassLoader test | Matei Zaharia | 2012-09-30 | 1 | -1/+1 |
| | |||||
* | Fixed several bugs that caused weird behavior with files in spark-shell: | Matei Zaharia | 2012-09-30 | 5 | -13/+19 |
| | | | | | | | | | - SizeEstimator was following through a ClassLoader field of Hadoop JobConfs, which referenced the whole interpreter, Scala compiler, etc. Chaos ensued, giving an estimated size in the tens of gigabytes. - Broadcast variables in local mode were only stored as MEMORY_ONLY and never made accessible over a server, so they fell out of the cache when they were deemed too large and couldn't be reloaded. | ||||
* | Comment | Matei Zaharia | 2012-09-29 | 1 | -1/+1 |
| | |||||
* | Removed Logging trait from CoalescedRDD since we don't log anything | Matei Zaharia | 2012-09-29 | 1 | -2/+1 |
| | |||||
* | Added a CoalescedRDD class for reducing the number of partitions in an RDD. | Matei Zaharia | 2012-09-29 | 1 | -0/+44 |
| | |||||
* | Merge branch 'dev' of github.com:mesos/spark into dev | Matei Zaharia | 2012-09-29 | 4 | -1/+8 |
|\ | |||||
| * | Merge pull request #227 from JoshRosen/fix/distinct_numsplits | Matei Zaharia | 2012-09-28 | 4 | -1/+8 |
| |\ | | | | | | | Allow controlling number of splits in distinct(). | ||||
| | * | Use null as dummy value in distinct(). | Josh Rosen | 2012-09-28 | 1 | -1/+1 |
| | | | |||||
| | * | Allow controlling number of splits in distinct(). | Josh Rosen | 2012-09-28 | 4 | -1/+8 |
| | | | |||||
* | | | Made BlockManager unmap memory-mapped files when necessary to reduce the | Matei Zaharia | 2012-09-29 | 9 | -116/+221 |
|/ / | | | | | | | number of open files. Also optimized sending of disk-based blocks. | ||||
* | | Don't create a Cache in SparkEnv because we don't use it | Matei Zaharia | 2012-09-28 | 1 | -5/+1 |
| | | |||||
* | | Logging tweaks | Matei Zaharia | 2012-09-28 | 5 | -17/+23 |
| | | |||||
* | | Renamed subdirs option | Matei Zaharia | 2012-09-28 | 1 | -1/+1 |
| | | |||||
* | | Made subdirs per local dir configurable, and reduced lock usage a bit | Matei Zaharia | 2012-09-28 | 1 | -12/+15 |
| | | |||||
* | | Made disk store use multiple directories, deleted ShuffleManager | Matei Zaharia | 2012-09-28 | 6 | -417/+345 |
| | | |||||
* | | Print and track user call sites in more places in Spark | Matei Zaharia | 2012-09-28 | 6 | -54/+63 |
|/ | |||||
* | Merge pull request #225 from pwendell/dev | Matei Zaharia | 2012-09-28 | 2 | -1/+39 |
|\ | | | | | Log message which records RDD origin | ||||
| * | Fixing some whitespace issues | Patrick Wendell | 2012-09-28 | 1 | -9/+9 |
| | | |||||
| * | Changes based on Matei's comments | Patrick Wendell | 2012-09-28 | 1 | -14/+14 |
| | | |||||
| * | Log message which records RDD origin | Patrick Wendell | 2012-09-28 | 2 | -1/+39 |
| | | | | | | | | | | | | | | | | | | | | | | | | | | This adds tracking to determine the "origin" of an RDD. Origin is defined by the boundary between the user's code and the spark code, during an RDD's instantiation. It is meant to help users understand where a Spark RDD is coming from in their code. This patch also logs origin data when stages are submitted to the scheduler. Finally, it adds a new log message to fix an inconsitency in the way that dependent stages (those missing parents) and independent stages (those without) are logged during submission. | ||||
* | | Changed the way tasks' dependency files are sent to workers so that | Matei Zaharia | 2012-09-28 | 13 | -155/+202 |
| | | | | | | | | custom serializers or Kryo registrators can be loaded. | ||||
* | | Fixed a bug where isLocal was set to false when using local[K] | Matei Zaharia | 2012-09-28 | 1 | -1/+1 |
| | | |||||
* | | Fix a bug in JAR fetcher that made it always fetch the JAR | Matei Zaharia | 2012-09-27 | 3 | -13/+9 |
| | | |||||
* | | Added an option to compress blocks in the block store | Matei Zaharia | 2012-09-27 | 3 | -8/+23 |
| | | |||||
* | | Renamed storage levels to something cleaner; fixes #223. | Matei Zaharia | 2012-09-27 | 8 | -20/+20 |
|/ | |||||
* | Merge pull request #222 from rxin/dev | Matei Zaharia | 2012-09-26 | 1 | -0/+18 |
|\ | | | | | Added MapPartitionsWithSplitRDD. | ||||
| * | Added MapPartitionsWithSplitRDD. | Reynold Xin | 2012-09-26 | 1 | -0/+18 |
| | |