| Commit message (Collapse) | Author | Age | Files | Lines |
| |
|
|\
| |
| | |
Scheduler fixes and improvements
|
| |
| |
| |
| | |
resetting locality level after a non-local launch
|
| | |
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
- When a resourceOffers() call has multiple offers, force the TaskSets
to consider them in increasing order of locality levels so that they
get a chance to launch stuff locally across all offers
- Simplify ClusterScheduler.prioritizeContainers
- Add docs on the new configuration options
|
| | |
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
- Added periodic revival of offers in StandaloneSchedulerBackend
- Replaced task scheduling aggression with multi-level delay scheduling
in ClusterTaskSetManager
- Fixed ZippedRDD preferred locations because they can't currently be
process-local
- Fixed some uses of hostPort
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
- Replace use of hostPort vs host in Task.preferredLocations with a
TaskLocation class that contains either an executorId and a host or
just a host. This is part of a bigger effort to eliminate hostPort
based data structures and just use executorID, since the hostPort vs
host stuff is confusing (and not checkable with static typing, leading
to ugly debug code), and hostPorts are not provided by Mesos.
- Replaced most hostPort-based data structures and fields as above.
- Simplified ClusterTaskSetManager to deal with preferred locations in a
more concise way and generally be more concise.
- Updated the way ClusterTaskSetManager handles racks: instead of
enqueueing a task to a separate queue for all the hosts in the rack,
which would create lots of large queues, have one queue per rack name.
- Removed non-local fallback stuff in ClusterScheduler that tried to
launch less-local tasks on a node once the local ones were all
assigned. This change didn't work because many cluster schedulers send
offers for just one node at a time (even the standalone and YARN ones
do so as nodes join the cluster one by one). Thus, lots of non-local
tasks would be assigned even though a node with locality for them
would be able to receive tasks just a short time later.
- Renamed MapOutputTracker "generations" to "epochs".
|
|\ \
| | |
| | | |
Small fixes to web UI
|
| | | |
|
| |/
| |
| |
| |
| |
| | |
- Use SPARK_PUBLIC_DNS environment variable if set (for EC2)
- Use a non-ephemeral port (3030 instead of 33000) by default
- Updated test to use non-ephemeral port too
|
|\ \
| |/
|/| |
Allow subclasses of Product2 in all key-value related classes
|
| |
| |
| |
| | |
documentation on MutablePair)
|
| | |
|
| |
| |
| |
| | |
code path working with general Product2.
|
| | |
|
| |
| |
| |
| | |
(ShuffleDependency, PairRDDFunctions, etc).
|
|\ \
| |/
|/| |
Implementing SPARK-878 for PySpark: adding zip and egg files to context ...
|
| |
| |
| |
| | |
passing it down to workers which add these to their sys.path
|
| |
| |
| |
| | |
setSerializer method in various RDDs that involve shuffle operations.
|
| | |
|
| | |
|
| | |
|
|\ \
| | |
| | | |
Currying RDD.zipPartitions
|
| | | |
|
| | |
| | |
| | |
| | | |
It made the JSON creation slightly more complicated, but reduces one external dependency. The scala library also properly escape "/" (which lift-json doesn't).
|
| | |
| | |
| | |
| | |
| | | |
This reverts commit 230ab2722ebd399afcf64c1a131f4929f602177d, reversing
changes made to 659553b21ddd7504889ce113a816c1db4a73f167.
|
|\ \ \
| |_|/
|/| | |
Updated json output to allow for display of worker state
|
| | |
| | |
| | | |
Ops teams need to ensure that the cluster is functional and performant. Having to scrape the html source for worker state won't work reliably, and will be slow. By exposing the state in the json output, ops teams are able to ensure a fully functional environment by querying for the json output and parsing for dead nodes.
|
|\ \ \
| |_|/
|/| | |
Rename `memoryBytesToString` and `memoryMegabytesToString`
|
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
These are used all over the place now and they are not specific to memory at all.
memoryBytesToString --> bytesToString
memoryMegabytesToString --> megabytesToString
|
| | | |
|
| |/
|/| |
|
| | |
|
|/
|
|
|
|
|
|
| |
1. Renamed SparkContext.addLocalProperty to setLocalProperty. And allow this function to unset a property.
2. Renamed SparkContext.setDescription to setCurrentJobDescription.
3. Throw an exception if the fair scheduler allocation file is invalid.
|
|\
| |
| | |
Adding GC Stats to TaskMetrics (and three small fixes)
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
| |
| |
| |
| | |
This fixes an error where empty cells are missing if a given task
has no shuffle read/write.
|
| |
| |
| |
| |
| |
| | |
1) UI crashed if the executor UI was loaded before any tasks started.
2) The total tasks was incorrectly reported due to using string (rather
than int) arithmetic.
|
|\ \
| | |
| | | |
Print run command to stderr rather than stdout
|
| | | |
|
| | | |
|
| |/
|/|
| |
| |
| |
| |
| | |
The TaskState class's isFinished() method didn't return true for
KILLED tasks, which means some resources are never reclaimed
for tasks that are killed. This also made it inconsistent with the
isFinished() method used by CoarseMesosSchedulerBackend.
|
| | |
|
|\ \
| | |
| | | |
Add MetricsServlet for Spark metrics system
|
| | | |
|