diff options
| author | Andy Sloane <asloane@tetrationanalytics.com> | 2016-03-09 10:25:47 +0000 |
|---|---|---|
| committer | Sean Owen <sowen@cloudera.com> | 2016-03-09 10:25:47 +0000 |
| commit | cbff2803ef117d7cffe6f05fc1bbd395a1e9c587 (patch) | |
| tree | ea2c9e175d8702754929c4e93810d71a971e828b /launcher/src/main/java/org/apache | |
| parent | 2c5af7d4d939e18a749d33b5de2e5113aa3eff08 (diff) | |
| download | spark-cbff2803ef117d7cffe6f05fc1bbd395a1e9c587.tar.gz spark-cbff2803ef117d7cffe6f05fc1bbd395a1e9c587.tar.bz2 spark-cbff2803ef117d7cffe6f05fc1bbd395a1e9c587.zip | |
[SPARK-13631][CORE] Thread-safe getLocationsWithLargestOutputs
## What changes were proposed in this pull request?
If a job is being scheduled in one thread which has a dependency on an
RDD currently executing a shuffle in another thread, Spark would throw a
NullPointerException. This patch synchronizes access to `mapStatuses` and
skips null status entries (which are in-progress shuffle tasks).
## How was this patch tested?
Our client code unit test suite, which was reliably reproducing the race
condition with 10 threads, shows that this fixes it. I have not found a minimal
test case to add to Spark, but I will attempt to do so if desired.
The same test case was tripping up on SPARK-4454, which was fixed by
making other DAGScheduler code thread-safe.
shivaram srowen
Author: Andy Sloane <asloane@tetrationanalytics.com>
Closes #11505 from a1k0n/SPARK-13631.
Diffstat (limited to 'launcher/src/main/java/org/apache')
0 files changed, 0 insertions, 0 deletions