aboutsummaryrefslogtreecommitdiff
path: root/core/src/main
Commit message (Collapse)AuthorAgeFilesLines
* Made Serializer and JavaSerializer non private.Reynold Xin2012-10-032-4/+4
|
* Made StorageLevel publicMatei Zaharia2012-10-031-2/+2
|
* Make more stuff private[spark]Matei Zaharia2012-10-026-4/+11
|
* Merge pull request #240 from dennybritz/private_classesMatei Zaharia2012-10-02117-186/+243
|\ | | | | Package-Private Classes
| * Make Java API abstract wrapped functions privateDenny2012-10-022-2/+2
| |
| * accidentially removed traitDenny2012-10-021-1/+1
| |
| * Stylistic changes and Public Accumulable and BroadcastDenny2012-10-0230-59/+117
| |
| * Make dependency classes public - used by sparkDenny2012-10-021-5/+5
| |
| * Make classes package privateDenny2012-10-02119-248/+247
| |
* | Merge branch 'dev' of github.com:mesos/spark into devMatei Zaharia2012-10-021-36/+47
|\ \
| * | Added a check to make sure SPARK_MEM <= memoryPerSlave for local clusterReynold Xin2012-10-021-2/+13
| | | | | | | | | | | | mode.
| * | Merge branch 'dev' of https://github.com/mesos/spark into devReynold Xin2012-10-024-225/+175
| |\|
| * | Allow whitespaces in cluster URL configuration for local cluster.Reynold Xin2012-10-021-34/+34
| | |
* | | Fixed cache replacement behavior of BlockManager:Matei Zaharia2012-10-024-190/+278
| |/ |/| | | | | | | | | | | | | | | | | | | | | | | - Partitions that get dropped to disk will now be loaded back into RAM after they're accessed again - Same-RDD rule for cache replacement is now implemented (don't drop partitions from an RDD to make room for other partitions from itself) - Items stored as MEMORY_AND_DISK go into memory only first, instead of being eagerly written out to disk - MemoryStore.ensureFreeSpace is called within a lock on the writer thread to prevent race conditions (this can still be optimized to allow multiple concurrent calls to it but it's a start) - MemoryStore does not accept blocks larger than its limit
* | Revert "Place Spray repo ahead of Cloudera in Maven search path"Matei Zaharia2012-10-024-225/+175
|/ | | | This reverts commit 42e0a68082327c78dbd0fd313145124d9b8a9d98.
* Place Spray repo ahead of Cloudera in Maven search pathMatei Zaharia2012-10-024-175/+225
|
* Include date in folder name for Spark local dir.Matei Zaharia2012-10-011-5/+7
|
* Merge branch 'dev' of github.com:mesos/spark into devMatei Zaharia2012-10-011-30/+32
|\
| * Fixed #232: DirectBuffer's cleaner was empty and Spark tried to invokeReynold Xin2012-10-011-30/+32
| | | | | | | | clean on it.
* | Some bug fixes and logging fixes for broadcast.Matei Zaharia2012-10-019-107/+111
|/
* Improve log messages from BlockManagerMatei Zaharia2012-10-014-45/+41
|
* Use underscores instead of colons in RDD IDsMatei Zaharia2012-10-015-6/+6
|
* Added a (failing) test for LRU with MEMORY_AND_DISK.Matei Zaharia2012-09-301-2/+2
|
* Simplified Class / ClassLoader testMatei Zaharia2012-09-301-1/+1
|
* Fixed several bugs that caused weird behavior with files in spark-shell:Matei Zaharia2012-09-305-13/+19
| | | | | | | | | - SizeEstimator was following through a ClassLoader field of Hadoop JobConfs, which referenced the whole interpreter, Scala compiler, etc. Chaos ensued, giving an estimated size in the tens of gigabytes. - Broadcast variables in local mode were only stored as MEMORY_ONLY and never made accessible over a server, so they fell out of the cache when they were deemed too large and couldn't be reloaded.
* CommentMatei Zaharia2012-09-291-1/+1
|
* Removed Logging trait from CoalescedRDD since we don't log anythingMatei Zaharia2012-09-291-2/+1
|
* Added a CoalescedRDD class for reducing the number of partitions in an RDD.Matei Zaharia2012-09-291-0/+44
|
* Merge branch 'dev' of github.com:mesos/spark into devMatei Zaharia2012-09-294-1/+8
|\
| * Merge pull request #227 from JoshRosen/fix/distinct_numsplitsMatei Zaharia2012-09-284-1/+8
| |\ | | | | | | Allow controlling number of splits in distinct().
| | * Use null as dummy value in distinct().Josh Rosen2012-09-281-1/+1
| | |
| | * Allow controlling number of splits in distinct().Josh Rosen2012-09-284-1/+8
| | |
* | | Made BlockManager unmap memory-mapped files when necessary to reduce theMatei Zaharia2012-09-299-116/+221
|/ / | | | | | | number of open files. Also optimized sending of disk-based blocks.
* | Don't create a Cache in SparkEnv because we don't use itMatei Zaharia2012-09-281-5/+1
| |
* | Logging tweaksMatei Zaharia2012-09-285-17/+23
| |
* | Renamed subdirs optionMatei Zaharia2012-09-281-1/+1
| |
* | Made subdirs per local dir configurable, and reduced lock usage a bitMatei Zaharia2012-09-281-12/+15
| |
* | Made disk store use multiple directories, deleted ShuffleManagerMatei Zaharia2012-09-286-417/+345
| |
* | Print and track user call sites in more places in SparkMatei Zaharia2012-09-286-54/+63
|/
* Merge pull request #225 from pwendell/devMatei Zaharia2012-09-282-1/+39
|\ | | | | Log message which records RDD origin
| * Fixing some whitespace issuesPatrick Wendell2012-09-281-9/+9
| |
| * Changes based on Matei's commentsPatrick Wendell2012-09-281-14/+14
| |
| * Log message which records RDD originPatrick Wendell2012-09-282-1/+39
| | | | | | | | | | | | | | | | | | | | | | | | | | This adds tracking to determine the "origin" of an RDD. Origin is defined by the boundary between the user's code and the spark code, during an RDD's instantiation. It is meant to help users understand where a Spark RDD is coming from in their code. This patch also logs origin data when stages are submitted to the scheduler. Finally, it adds a new log message to fix an inconsitency in the way that dependent stages (those missing parents) and independent stages (those without) are logged during submission.
* | Changed the way tasks' dependency files are sent to workers so thatMatei Zaharia2012-09-2813-155/+202
| | | | | | | | custom serializers or Kryo registrators can be loaded.
* | Fixed a bug where isLocal was set to false when using local[K]Matei Zaharia2012-09-281-1/+1
| |
* | Fix a bug in JAR fetcher that made it always fetch the JARMatei Zaharia2012-09-273-13/+9
| |
* | Added an option to compress blocks in the block storeMatei Zaharia2012-09-273-8/+23
| |
* | Renamed storage levels to something cleaner; fixes #223.Matei Zaharia2012-09-278-20/+20
|/
* Merge pull request #222 from rxin/devMatei Zaharia2012-09-261-0/+18
|\ | | | | Added MapPartitionsWithSplitRDD.
| * Added MapPartitionsWithSplitRDD.Reynold Xin2012-09-261-0/+18
| |
generated by cgit v1.2.3 (git 2.39.1) at 2024-08-23 07:47:40 +0000