aboutsummaryrefslogtreecommitdiff
path: root/core/src/test/scala
Commit message (Collapse)AuthorAgeFilesLines
* Merge branch 'dev' of github.com:mesos/spark into devMatei Zaharia2012-10-021-3/+19
|\
| * Merge branch 'dev' of https://github.com/mesos/spark into devReynold Xin2012-10-021-3/+3
| |\
| * | Allow whitespaces in cluster URL configuration for local cluster.Reynold Xin2012-10-021-3/+19
| | |
* | | Added a test for overly large blocks in memory storeMatei Zaharia2012-10-021-0/+9
| | |
* | | Fixed cache replacement behavior of BlockManager:Matei Zaharia2012-10-021-6/+83
| |/ |/| | | | | | | | | | | | | | | | | | | | | | | - Partitions that get dropped to disk will now be loaded back into RAM after they're accessed again - Same-RDD rule for cache replacement is now implemented (don't drop partitions from an RDD to make room for other partitions from itself) - Items stored as MEMORY_AND_DISK go into memory only first, instead of being eagerly written out to disk - MemoryStore.ensureFreeSpace is called within a lock on the writer thread to prevent race conditions (this can still be optimized to allow multiple concurrent calls to it but it's a start) - MemoryStore does not accept blocks larger than its limit
* | Revert "Place Spray repo ahead of Cloudera in Maven search path"Matei Zaharia2012-10-021-3/+3
|/ | | | This reverts commit 42e0a68082327c78dbd0fd313145124d9b8a9d98.
* Place Spray repo ahead of Cloudera in Maven search pathMatei Zaharia2012-10-021-3/+3
|
* Remove some printlns in testsMatei Zaharia2012-10-012-2/+5
|
* Added a (failing) test for LRU with MEMORY_AND_DISK.Matei Zaharia2012-09-301-3/+7
|
* Fixed several bugs that caused weird behavior with files in spark-shell:Matei Zaharia2012-09-302-1/+23
| | | | | | | | | - SizeEstimator was following through a ClassLoader field of Hadoop JobConfs, which referenced the whole interpreter, Scala compiler, etc. Chaos ensued, giving an estimated size in the tens of gigabytes. - Broadcast variables in local mode were only stored as MEMORY_ONLY and never made accessible over a server, so they fell out of the cache when they were deemed too large and couldn't be reloaded.
* CommentMatei Zaharia2012-09-291-1/+1
|
* Added a CoalescedRDD class for reducing the number of partitions in an RDD.Matei Zaharia2012-09-291-0/+31
|
* CommentMatei Zaharia2012-09-291-0/+1
|
* Made BlockManager unmap memory-mapped files when necessary to reduce theMatei Zaharia2012-09-291-2/+58
| | | | number of open files. Also optimized sending of disk-based blocks.
* Added an option to compress blocks in the block storeMatei Zaharia2012-09-271-0/+16
|
* Renamed storage levels to something cleaner; fixes #223.Matei Zaharia2012-09-271-27/+27
|
* Merge pull request #222 from rxin/devMatei Zaharia2012-09-261-0/+5
|\ | | | | Added MapPartitionsWithSplitRDD.
| * Added MapPartitionsWithSplitRDD.Reynold Xin2012-09-261-0/+5
| |
* | Allow controlling number of splits in sortByKey.Matei Zaharia2012-09-261-5/+43
|/
* Fixed a test that was getting extremely lucky before, and increased theMatei Zaharia2012-09-261-9/+9
| | | | number of samples used for sorting
* Fix some test issuesMatei Zaharia2012-09-242-12/+14
|
* Separated ShuffledRDD into multiple classes: RepartitionShuffledRDD,Reynold Xin2012-09-191-9/+9
| | | | ShuffledSortedRDD, and ShuffledAggregatedRDD.
* Merge branch 'dev' into feature/fileserverDenny2012-09-111-2/+25
|\ | | | | | | | | Conflicts: core/src/main/scala/spark/SparkContext.scala
| * Manually merge pull request #175 by Imran RashidMatei Zaharia2012-09-111-2/+25
| |
| * Added a unit test for local-cluster mode and simplified some of the code ↵Matei Zaharia2012-09-071-0/+68
| | | | | | | | involved in that
* | Fix serialization bugs and added local cluster testsDenny2012-09-101-2/+31
| |
* | Dynamically adding jar files and caching fileSets.Denny2012-09-101-9/+30
| |
* | General FileServerDenny2012-09-101-0/+43
| | | | | | | | A general fileserver for both JARs and regular files.
* | Added a unit test for local-cluster mode and simplified some of the code ↵Matei Zaharia2012-09-101-0/+68
|/ | | | involved in that
* Removed cache add/remove log messages from CacheTracker.Reynold Xin2012-09-051-2/+2
| | | | | Added log messages on BlockManagerMaster to reflect block add/remove. Also did some minor cleanup of storage package code.
* Added a test for testing map-side combine on/off switch.Reynold Xin2012-08-301-1/+44
|
* Fix further issues with tests and broadcast.Matei Zaharia2012-08-239-8/+15
| | | | | The broadcast fix is to store values as MEMORY_ONLY_DESER instead of MEMORY_ONLY, which will save substantial time on serialization.
* Change BlockManagerSuite test cases to use a deterministic size estimator andShivaram Venkataraman2012-08-131-9/+31
| | | | update the results to match the new estimates
* Add test-cases for 32-bit and no-compressed oops scenarios.Shivaram Venkataraman2012-08-132-14/+46
|
* Update test cases to match the new size estimates. Uses 64-bit and compressedShivaram Venkataraman2012-08-132-35/+77
| | | | oops setting to get deterministic results
* Renamed apply() to call() in Java API and allowed it to throw ExceptionsMatei Zaharia2012-08-121-21/+21
|
* Merge pull request #167 from JoshRosen/piped-rdd-fixesMatei Zaharia2012-08-101-0/+9
|\ | | | | Detect non-zero exit status from PipedRDD process
| * Detect non-zero exit status from PipedRDD process.Josh Rosen2012-07-261-0/+9
| |
* | Fix testcheckpoint to reuse spark context defined in the classShivaram Venkataraman2012-08-031-2/+1
| |
* | Made range partition balance tests more aggressive.Matei Zaharia2012-08-031-8/+8
| | | | | | | | | | | | This is because we pull out such a large sample (10x the number of partitions) that we should expect pretty good balance. The tests are also deterministic so there's no worry about them failing irreproducibly.
* | Added a unit test for cross-partition balancing in sort, and changes toMatei Zaharia2012-08-031-29/+61
| | | | | | | | | | RangePartitioner to make it pass. It turns out that the first partition was always kind of small due to how we picked partition boundaries.
* | Merge branch 'block-manager-fix' into devMatei Zaharia2012-07-301-19/+21
|\ \
| * | Made BlockManagerMaster no longer be a singleton.Matei Zaharia2012-07-301-19/+21
| | | | | | | | | | | | Also cleaned up a few formatting things throughout block manager code.
* | | Merge branch 'dev' of git://github.com/tdas/spark into devMatei Zaharia2012-07-301-0/+7
|\ \ \ | |/ / |/| |
| * | Added BlockRDD and a first-cut version of checkpoint() to RDD class.Tathagata Das2012-07-271-0/+7
| | |
| * | Modified the block dropping code for better performance.Tathagata Das2012-07-261-0/+1
| | |
* | | Fixed AccumulatorSuite to clean up SparkContext with BeforeAndAfterMatei Zaharia2012-07-281-10/+20
| | |
* | | tasks cannot access value of accumulatorImran Rashid2012-07-281-53/+12
| | |
* | | rename addToAccum to addAccumulatorImran Rashid2012-07-281-1/+1
| | |
* | | add some functionality to Vector, delete copy in AccumulatorSuiteImran Rashid2012-07-281-112/+2
| | |
generated by cgit v1.2.3 (git 2.39.1) at 2024-09-12 20:45:15 +0000