diff options
| author | Tejas Patil <tejasp@fb.com> | 2016-03-24 00:31:13 -0700 |
|---|---|---|
| committer | Reynold Xin <rxin@databricks.com> | 2016-03-24 00:31:13 -0700 |
| commit | 01849da080439d1f2dbb90a8985c661522ed3d7a (patch) | |
| tree | 9591413e38ade17e65658599a9bbad8b27c98829 /mllib | |
| parent | c44d140cae99d0b880e6d25f158125ad3adc6a05 (diff) | |
| download | spark-01849da080439d1f2dbb90a8985c661522ed3d7a.tar.gz spark-01849da080439d1f2dbb90a8985c661522ed3d7a.tar.bz2 spark-01849da080439d1f2dbb90a8985c661522ed3d7a.zip | |
[SPARK-14110][CORE] PipedRDD to print the command ran on non zero exit
## What changes were proposed in this pull request?
In case of failure in subprocess launched in PipedRDD, the failure exception reads “Subprocess exited with status XXX”. Debugging this is not easy for users especially if there are multiple pipe() operations in the Spark application.
Changes done:
- Changed the exception message when non-zero exit code is seen
- If the reader and writer threads see exception, simply logging the command ran. The current model is to propagate the exception "as is" so that upstream Spark logic will take the right action based on what the exception was (eg. for fetch failure, it needs to retry; but for some fatal exception, it will decide to fail the stage / job). So wrapping the exception with a generic exception will not work. Altering the exception message will keep that guarantee but that is ugly (plus not all exceptions might have a constructor for a string message)
## How was this patch tested?
- Added a new test case
- Ran all existing tests for PipedRDD
Author: Tejas Patil <tejasp@fb.com>
Closes #11927 from tejasapatil/SPARK-14110-piperdd-failure.
Diffstat (limited to 'mllib')
0 files changed, 0 insertions, 0 deletions