aboutsummaryrefslogtreecommitdiff
Commit message (Collapse)AuthorAgeFilesLines
* Further simplify checking for Nil.Stephen Haberman2013-02-021-3/+1
|
* Once we find a split with no block, we don't have to look for more.Stephen Haberman2013-01-311-12/+11
|
* Merge pull request #430 from pwendell/pyspark-guideMatei Zaharia2013-01-302-2/+10
|\ | | | | Minor improvements to PySpark docs
| * Make module help available in python shell.Patrick Wendell2013-01-302-0/+2
| | | | | | | | Also, adds a line in doc explaining how to use.
| * Inclue packaging and launching pyspark in guide.Patrick Wendell2013-01-301-2/+8
| | | | | | | | It's nicer if all the commands you need are made explicit.
* | Merge pull request #426 from woggling/conn-manager-ipsMatei Zaharia2013-01-302-5/+13
|\ \ | | | | | | Remember ConnectionManagerId used to initiate SendingConnections
| * | Remember ConnectionManagerId used to initiate SendingConnections.Charles Reiss2013-01-292-5/+13
| | | | | | | | | | | | | | | | | | This prevents ConnectionManager from getting confused if a machine has multiple host names and the one getHostName() finds happens not to be the one that was passed from, e.g., the BlockManagerMaster.
* | | Merge pull request #428 from woggling/mesos-exec-idMatei Zaharia2013-01-302-15/+21
|\ \ \ | | | | | | | | Make ExecutorIDs include SlaveIDs when running Mesos
| * | | Remove remants of attempt to use slaveId-executorId in MesosExecutorBackendCharles Reiss2013-01-301-1/+1
| | | |
| * | | Use Mesos ExecutorIDs to hold SlaveIDs. Then we can safely useCharles Reiss2013-01-302-15/+21
| |/ / | | | | | | | | | the Mesos ExecutorID as a Spark ExecutorID.
* | | Merge pull request #429 from stephenh/includemessageMatei Zaharia2013-01-301-1/+3
|\ \ \ | | | | | | | | Include message and exitStatus if availalbe.
| * | | Include message and exitStatus if availalbe.Stephen Haberman2013-01-301-1/+3
|/ / /
* | | Merge remote-tracking branch 'stephenh/removefailedjob'Matei Zaharia2013-01-292-7/+9
|\ \ \ | | | | | | | | | | | | | | | | Conflicts: core/src/main/scala/spark/deploy/master/Master.scala
| * | | Fix Worker logInfo about unknown executor.Stephen Haberman2013-01-221-1/+1
| | | |
| * | | Don't bother creating an exception.Stephen Haberman2013-01-221-2/+1
| | | |
| * | | Handle Master telling the Worker to kill an already-dead executor.Stephen Haberman2013-01-221-3/+7
| | | |
| * | | Call removeJob instead of killing the cluster.Stephen Haberman2013-01-221-2/+1
| | | |
* | | | Merge pull request #425 from stephenh/toDebugStringMatei Zaharia2013-01-292-0/+21
|\ \ \ \ | | | | | | | | | | Add RDD.toDebugString.
| * | | | Include name, if set, in RDD.toString().Stephen Haberman2013-01-291-1/+6
| | | | |
| * | | | Add number of splits.Stephen Haberman2013-01-291-1/+2
| | | | |
| * | | | Add JavaRDDLike.toDebugString().Stephen Haberman2013-01-291-0/+5
| | | | |
| * | | | Add RDD.toDebugString.Stephen Haberman2013-01-281-0/+10
| | | | | | | | | | | | | | | | | | | | Original idea by Nathan Kronenfeld.
* | | | | Merge pull request #415 from stephenh/driverMatei Zaharia2013-01-2934-216/+214
|\ \ \ \ \ | |_|_|/ / |/| | | | Replace old 'master' term with 'driver'.
| * | | | Merge branch 'master' into driverStephen Haberman2013-01-2857-390/+938
| |\| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Conflicts: core/src/main/scala/spark/SparkContext.scala core/src/main/scala/spark/SparkEnv.scala core/src/main/scala/spark/deploy/LocalSparkCluster.scala core/src/main/scala/spark/executor/StandaloneExecutorBackend.scala core/src/main/scala/spark/scheduler/cluster/SparkDeploySchedulerBackend.scala core/src/main/scala/spark/scheduler/cluster/StandaloneClusterMessage.scala core/src/main/scala/spark/scheduler/cluster/StandaloneSchedulerBackend.scala core/src/main/scala/spark/storage/BlockManagerMaster.scala core/src/main/scala/spark/storage/ThreadingTest.scala core/src/test/scala/spark/MapOutputTrackerSuite.scala
| * | | | Replace old 'master' term with 'driver'.Stephen Haberman2013-01-2534-251/+248
| | | | |
* | | | | Simplify checkpointing code and RDD class a little:Matei Zaharia2013-01-2815-168/+153
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - RDD's getDependencies and getSplits methods are now guaranteed to be called only once, so subclasses can safely do computation in there without worrying about caching the results. - The management of a "splits_" variable that is cleared out when we checkpoint an RDD is now done in the RDD class. - A few of the RDD subclasses are simpler. - CheckpointRDD's compute() method no longer assumes that it is given a CheckpointRDDSplit -- it can work just as well on a split from the original RDD, because it only looks at its index. This is important because things like UnionRDD and ZippedRDD remember the parent's splits as part of their own and wouldn't work on checkpointed parents. - RDD.iterator can now reuse cached data if an RDD is computed before it is checkpointed. It seems like it wouldn't do this before (it always called iterator() on the CheckpointRDD, which read from HDFS).
* | | | | Fix code that depended on metadata cleaner interval being in minutesMatei Zaharia2013-01-282-5/+5
| | | | |
* | | | | Merge branch 'master' of github.com:mesos/sparkMatei Zaharia2013-01-286-16/+47
|\ \ \ \ \
| * \ \ \ \ Merge pull request #413 from pwendell/stage-loggingMatei Zaharia2013-01-282-4/+19
| |\ \ \ \ \ | | | | | | | | | | | | | | SPARK-658: Adding logging of stage duration
| | * | | | | Units from ms -> sPatrick Wendell2013-01-281-2/+2
| | | | | | |
| | * | | | | Making submission time a fieldPatrick Wendell2013-01-282-4/+6
| | | | | | |
| | * | | | | Renaming stage finished functionPatrick Wendell2013-01-281-3/+3
| | | | | | |
| | * | | | | SPARK-658: Adding logging of stage durationPatrick Wendell2013-01-281-4/+17
| | | |/ / / | | |/| | |
| * | | | | Merge pull request #424 from pwendell/logging-cleanupMatei Zaharia2013-01-283-12/+12
| |\ \ \ \ \ | | | | | | | | | | | | | | Some DEBUG-level log cleanup.
| | * | | | | Some DEBUG-level log cleanup.Patrick Wendell2013-01-283-12/+12
| | |/ / / / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | A few changes to make the DEBUG-level logs less noisy and more readable. - Moved a few very frequent messages to Trace - Changed some BlockManger log messages to make them more understandable SPARK-666 #resolve
| * | | | | Merge pull request #423 from squito/long_float_accumsMatei Zaharia2013-01-282-0/+16
| |\ \ \ \ \ | | |/ / / / | |/| | | | add long and float accumulatorparams
| | * | | | add long and float accumulatorparamsImran Rashid2013-01-282-0/+16
| |/ / / /
* / / / / Change time unit in MetadataCleaner to secondsMatei Zaharia2013-01-281-3/+2
|/ / / /
* | | | Clean up BlockManagerUI a little (make it not be an object, merge withMatei Zaharia2013-01-277-74/+91
| | | | | | | | | | | | | | | | Directives, and bind to a random port)
* | | | Rename more things from slave to executorMatei Zaharia2013-01-278-60/+50
| | | |
* | | | Track workers by executor ID instead of hostname to allow multipleMatei Zaharia2013-01-2735-314/+343
| | | | | | | | | | | | | | | | | | | | executors per machine and remove the need for multiple IP addresses in unit tests.
* | | | Merge pull request #419 from shivaram/ec2-ip-changeMatei Zaharia2013-01-271-1/+2
|\ \ \ \ | | | | | | | | | | Detect whether we run on EC2 using ec2-metadata as well
| * | | | Detect whether we run on EC2 using ec2-metadata as wellShivaram Venkataraman2013-01-261-1/+2
| | | | |
* | | | | Merge pull request #401 from squito/blockmanager_uiMatei Zaharia2013-01-2717-8/+401
|\ \ \ \ \ | | | | | | | | | | | | Blockmanager ui
| * | | | | add metadatacleaner for persisentRdd mapImran Rashid2013-01-251-1/+11
| | | | | |
| * | | | | fixup 1cadaa1, changed api of mapImran Rashid2013-01-251-2/+2
| | | | | |
| * | | | | switch to TimeStampedHashMap for storing persistent RddsImran Rashid2013-01-251-1/+2
| | | | | |
| * | | | | code reformattingImran Rashid2013-01-252-5/+7
| | | | | |
| * | | | | Merge branch 'master' into blockmanager_uiImran Rashid2013-01-22159-469/+11630
| |\ \ \ \ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | Conflicts: core/src/main/scala/spark/RDD.scala
| * | | | | | Fix up some problems from the mergeImran Rashid2013-01-223-4/+18
| | | | | | |
generated by cgit v1.2.3 (git 2.39.1) at 2024-09-20 23:59:57 +0000