aboutsummaryrefslogtreecommitdiff
path: root/core
Commit message (Collapse)AuthorAgeFilesLines
* Rename top-level 'pyspark' directory to 'python'Josh Rosen2013-01-011-1/+1
|
* Minor documentation and style fixes for PySpark.Josh Rosen2013-01-012-15/+32
|
* Update PySpark for compatibility with TaskContext.Josh Rosen2012-12-291-8/+5
|
* Merge remote-tracking branch 'origin/master' into python-apiJosh Rosen2012-12-2997-1531/+2892
|\ | | | | | | | | Conflicts: docs/quick-start.md
| * Change Utils.fetchFile() warning to SparkException.Josh Rosen2012-12-282-7/+10
| |
| * Add addFile() and addJar() to JavaSparkContext.Josh Rosen2012-12-281-0/+34
| |
| * Add synchronization to LocalScheduler.updateDependencies().Josh Rosen2012-12-281-16/+18
| |
| * Skip deletion of files in clearFiles().Josh Rosen2012-12-282-20/+46
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This fixes an issue where Spark could delete original files in the current working directory that were added to the job using addFile(). There was also the potential for addFile() to overwrite local files, which is addressed by changing Utils.fetchFile() to log a warning instead of overwriting a file with new contents. This is a short-term fix; a better long-term solution would be to remove the dependence on storing files in the current working directory, since we can't change the cwd from Java.
| * fall back to filter-map-collect when calling lookup() on an RDD without a ↵Mark Hamstra2012-12-242-1/+12
| | | | | | | | partitioner
| * Allow distinct() to be called without parentheses when using the default ↵Mark Hamstra2012-12-242-5/+11
| | | | | | | | number of splits.
| * Remove the call to close input stream in Kryo serializer.Reynold Xin2012-12-211-1/+1
| |
| * Merge pull request #341 from rxin/4a3fb06ac2d11125feb08acbbd4df76d1e91b677Matei Zaharia2012-12-211-137/+68
| |\ | | | | | | Kryo2 update against Spark master
| | * Updated Kryo to 2.20.Reynold Xin2012-10-161-137/+68
| | |
| * | Merge pull request #327 from rxin/spark-633Matei Zaharia2012-12-2016-769/+1096
| |\ \ | | | | | | | | Added the ability in block manager to remove blocks.
| | * | Let the slave notify the master block removal.Reynold Xin2012-12-204-92/+83
| | | |
| | * | Moved BlockManager's IdGenerator into BlockManager object. Removed someReynold Xin2012-12-195-36/+22
| | | | | | | | | | | | | | | | excessive debug messages.
| | * | Merge branch 'spark-633' of github.com:rxin/spark into spark-633Reynold Xin2012-12-144-7/+23
| | |\ \
| | | * | Return success or failure in BlockStore.remove().Josh Rosen2012-12-134-7/+23
| | | | |
| | * | | Fixed conflicts from merging Charles' and TD's block manager changes.Reynold Xin2012-12-143-173/+158
| | | | |
| | * | | Merge branch 'master' of github.com:mesos/spark into spark-633Reynold Xin2012-12-131-7/+9
| | |\ \ \
| | * | | | Merged TD's block manager refactoring.Reynold Xin2012-12-139-647/+805
| | | | | |
| | * | | | Merge branch 'master' of github.com:mesos/spark into spark-633Reynold Xin2012-12-138-15/+118
| | |\ \ \ \
| | * \ \ \ \ Merge branch 'master' of github.com:mesos/spark into spark-633Reynold Xin2012-12-1326-208/+207
| | |\ \ \ \ \ | | | |_|_|/ / | | |/| | | |
| | * | | | | Added the ability in block manager to remove blocks.Reynold Xin2012-12-139-170/+361
| | | | | | |
| * | | | | | SPARK-616: Logging dead workers in Web UI.Patrick Wendell2012-12-175-3/+19
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch keeps track of which workers have died and marks them as such in the master web UI. It also handles workers which die and re-register using different actor ID's.
| * | | | | | Merge pull request #332 from JoshRosen/spark-607Matei Zaharia2012-12-141-12/+17
| |\ \ \ \ \ \ | | | | | | | | | | | | | | | | Add try-finally to handle MapOutputTracker timeouts
| | * | | | | | Add try-finally to handle MapOutputTracker timeouts.Josh Rosen2012-12-131-12/+17
| | |/ / / / /
| * | | | | | Code review cleanup.Charles Reiss2012-12-131-1/+4
| | | | | | |
| * | | | | | Have standalone cluster report exit codes to clients. Addresses SPARK-639.Charles Reiss2012-12-138-24/+20
| | |_|_|/ / | |/| | | |
| * | | | | Fixed the broken Java unit test from SPARK-635.Reynold Xin2012-12-131-7/+9
| | |_|/ / | |/| | |
| * | | | Merge pull request #329 from woggling/executor-status-codesMatei Zaharia2012-12-138-15/+118
| |\ \ \ \ | | |_|/ / | |/| | | Executor exit status codes
| | * | | ExecutorLostReason -> ExecutorLossReasonCharles Reiss2012-12-134-6/+6
| | | | |
| | * | | Extra whitespace in ExecutorExitCodeCharles Reiss2012-12-131-0/+3
| | | | |
| | * | | Explain slaveLost calls made by StandaloneSchedulerBackendCharles Reiss2012-12-131-5/+5
| | | | |
| | * | | Log duplicate slaveLost() calls in ClusterScheduler.Charles Reiss2012-12-131-1/+7
| | | | |
| | * | | Normalize executor exit statuses and report them to the user.Charles Reiss2012-12-138-11/+94
| | | | |
| | * | | Call slaveLost on executor death for standalone clusters.Charles Reiss2012-12-121-1/+12
| | |/ /
| * | | SPARK-635: Pass a TaskContext object to compute() interface and useReynold Xin2012-12-1320-49/+46
| | | | | | | | | | | | | | | | that to close Hadoop input stream. Incorporated Matei's command.
| * | | SPARK-635: Pass a TaskContext object to compute() interface and use thatReynold Xin2012-12-1326-205/+207
| |/ / | | | | | | | | | to close Hadoop input stream.
| * | Suppress shuffle block updates when a slave node comes back.Reynold Xin2012-12-102-12/+17
| | |
| * | Merge pull request #317 from woggling/block-manager-heartbeatMatei Zaharia2012-12-106-62/+318
| |\ \ | | | | | | | | Implement block manager heartbeat
| | * | Decrease BlockManagerMaster logging verbosityCharles Reiss2012-12-101-1/+2
| | | |
| | * | Use Akka scheduler for BlockManager heart beats.Charles Reiss2012-12-104-49/+41
| | | | | | | | | | | | | | | | Adds required ActorSystem argument to BlockManager constructors.
| | * | Changed default block manager heartbeat interval to 5 sCharles Reiss2012-12-091-1/+1
| | | |
| | * | Don't divide by milliseconds by 1000 more.Charles Reiss2012-12-061-1/+1
| | | |
| | * | map -> foreachCharles Reiss2012-12-061-1/+1
| | | |
| | * | Make LocalSparkCluster use distinct IPsCharles Reiss2012-12-061-2/+6
| | | |
| | * | Tests for block manager heartbeats.Charles Reiss2012-12-051-0/+68
| | | |
| | * | Add block manager heart beats.Charles Reiss2012-12-052-25/+193
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Renames old message called 'HeartBeat' to 'BlockUpdate'. The BlockManager periodically sends a heart beat message to the master. If the manager is currently not registered. The master responds to the heart beat by indicating whether the BlockManager is currently registered with the master. Additionally, the master now also responds to block updates by indicating whether the BlockManager in question is registered. When the BlockManager detects (by heart beat or failed block update) that it stopped being registered, it reregisters and sends block updates for all its blocks.
| | * | Track block managers by hostname; handle manager removal.Charles Reiss2012-12-051-3/+27
| | | |
generated by cgit v1.2.3 (git 2.39.1) at 2024-07-26 22:06:54 +0000