diff options
author | Patrick Wendell <pwendell@gmail.com> | 2013-10-14 22:25:47 -0700 |
---|---|---|
committer | Patrick Wendell <pwendell@gmail.com> | 2013-10-14 22:25:47 -0700 |
commit | e33b1839e27249e232a2126cec67a38109e03243 (patch) | |
tree | c25eb6810167f7a0a19fd210712ca4c209bfb48a /pom.xml | |
parent | 3b11f43e36e2aca2346db7542c52fcbbeee70da2 (diff) | |
parent | 9cd8786e4a6aebe29dabe802cc177e4338e140e6 (diff) | |
download | spark-e33b1839e27249e232a2126cec67a38109e03243.tar.gz spark-e33b1839e27249e232a2126cec67a38109e03243.tar.bz2 spark-e33b1839e27249e232a2126cec67a38109e03243.zip |
Merge pull request #29 from rxin/kill
Job killing
Moving https://github.com/mesos/spark/pull/935 here
The high level idea is to have an "interrupted" field in TaskContext, and a task should check that flag to determine if its execution should continue. For convenience, I provide an InterruptibleIterator which wraps around a normal iterator but checks for the interrupted flag. I also provide an InterruptibleRDD that wraps around an existing RDD.
As part of this pull request, I added an AsyncRDDActions class that provides a number of RDD actions that return a FutureJob (extending scala.concurrent.Future). The FutureJob can be used to kill the job execution, or waits until the job finishes.
This is NOT ready for merging yet. Remaining TODOs:
1. Add unit tests
2. Add job killing functionality for local scheduler (current job killing functionality only works in cluster scheduler)
-------------
Update on Oct 10, 2013:
This is ready!
Related future work:
- Figure out how to handle the job triggered by RangePartitioner (this one is tough; might become future work)
- Java API
- Python API
Diffstat (limited to 'pom.xml')
0 files changed, 0 insertions, 0 deletions