diff options
| author | Shixiong Zhu <shixiong@databricks.com> | 2016-03-25 13:28:26 -0700 |
|---|---|---|
| committer | Shixiong Zhu <shixiong@databricks.com> | 2016-03-25 13:28:26 -0700 |
| commit | b554b3c46b0019a6caf0f9a975b460dc2570c3b2 (patch) | |
| tree | 7de1a82630527694e73a9967b0f9b1def389ec0e /mllib/src/test/scala/org | |
| parent | afd0debe075e9ea8466e384932a513ef0188273c (diff) | |
| download | spark-b554b3c46b0019a6caf0f9a975b460dc2570c3b2.tar.gz spark-b554b3c46b0019a6caf0f9a975b460dc2570c3b2.tar.bz2 spark-b554b3c46b0019a6caf0f9a975b460dc2570c3b2.zip | |
[SPARK-14131][SQL] Add a workaround for HADOOP-10622 to fix DataFrameReaderWriterSuite
## What changes were proposed in this pull request?
There is a potential dead-lock in Hadoop Shell.runCommand before 2.5.0 ([HADOOP-10622](https://issues.apache.org/jira/browse/HADOOP-10622)). If we interrupt some thread running Shell.runCommand, we may hit this issue.
This PR adds some protecion to prevent from interrupting the microBatchThread when we may run into Shell.runCommand. There are two places will call Shell.runCommand now:
- offsetLog.add
- FileStreamSource.getOffset
They will create a file using HDFS API and call Shell.runCommand to set the file permission.
## How was this patch tested?
Existing unit tests.
Author: Shixiong Zhu <shixiong@databricks.com>
Closes #11940 from zsxwing/workaround-for-HADOOP-10622.
Diffstat (limited to 'mllib/src/test/scala/org')
0 files changed, 0 insertions, 0 deletions