diff options
| author | Kishor Patil <kpatil@yahoo-inc.com> | 2016-09-14 14:19:35 -0500 |
|---|---|---|
| committer | Tom Graves <tgraves@yahoo-inc.com> | 2016-09-14 14:19:35 -0500 |
| commit | ff6e4cbdc80e2ad84c5d70ee07f323fad9374e3e (patch) | |
| tree | 043178e11f33178894f9bf287d61a381c15447e6 /core/src/test/scala/org/apache | |
| parent | 040e46979d5f90edc7f9be3cbedd87e8986e8053 (diff) | |
| download | spark-ff6e4cbdc80e2ad84c5d70ee07f323fad9374e3e.tar.gz spark-ff6e4cbdc80e2ad84c5d70ee07f323fad9374e3e.tar.bz2 spark-ff6e4cbdc80e2ad84c5d70ee07f323fad9374e3e.zip | |
[SPARK-17511] Yarn Dynamic Allocation: Avoid marking released container as Failed
## What changes were proposed in this pull request?
Due to race conditions, the ` assert(numExecutorsRunning <= targetNumExecutors)` can fail causing `AssertionError`. So removed the assertion, instead moved the conditional check before launching new container:
```
java.lang.AssertionError: assertion failed
at scala.Predef$.assert(Predef.scala:156)
at org.apache.spark.deploy.yarn.YarnAllocator$$anonfun$runAllocatedContainers$1.org$apache$spark$deploy$yarn$YarnAllocator$$anonfun$$updateInternalState$1(YarnAllocator.scala:489)
at org.apache.spark.deploy.yarn.YarnAllocator$$anonfun$runAllocatedContainers$1$$anon$1.run(YarnAllocator.scala:519)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
```
## How was this patch tested?
This was manually tested using a large ForkAndJoin job with Dynamic Allocation enabled to validate the failing job succeeds, without any such exception.
Author: Kishor Patil <kpatil@yahoo-inc.com>
Closes #15069 from kishorvpatil/SPARK-17511.
Diffstat (limited to 'core/src/test/scala/org/apache')
0 files changed, 0 insertions, 0 deletions