diff options
| author | Cheng Lian <lian@databricks.com> | 2015-08-06 09:53:53 -0700 |
|---|---|---|
| committer | Yin Huai <yhuai@databricks.com> | 2015-08-06 09:53:53 -0700 |
| commit | 9f94c85ff35df6289371f80edde51c2aa6c4bcdc (patch) | |
| tree | 9858091ca33d98022573ed721ab165ee3dbd2ba9 /python/pyspark/ml/feature.py | |
| parent | 93085c992e40dbc06714cb1a64c838e25e683a6f (diff) | |
| download | spark-9f94c85ff35df6289371f80edde51c2aa6c4bcdc.tar.gz spark-9f94c85ff35df6289371f80edde51c2aa6c4bcdc.tar.bz2 spark-9f94c85ff35df6289371f80edde51c2aa6c4bcdc.zip | |
[SPARK-9593] [SQL] [HOTFIX] Makes the Hadoop shims loading fix more robust
This is a follow-up of #7929.
We found that Jenkins SBT master build still fails because of the Hadoop shims loading issue. But the failure doesn't appear to be deterministic. My suspect is that Hadoop `VersionInfo` class may fail to inspect Hadoop version, and the shims loading branch is skipped.
This PR tries to make the fix more robust:
1. When Hadoop version is available, we load `Hadoop20SShims` for versions <= 2.0.x as srowen suggested in PR #7929.
2. Otherwise, we use `Path.getPathWithoutSchemeAndAuthority` as a probe method, which doesn't exist in Hadoop 1.x or 2.0.x. If this method is not found, `Hadoop20SShims` is also loaded.
Author: Cheng Lian <lian@databricks.com>
Closes #7994 from liancheng/spark-9593/fix-hadoop-shims and squashes the following commits:
e1d3d70 [Cheng Lian] Fixes typo in comments
8d971da [Cheng Lian] Makes the Hadoop shims loading fix more robust
Diffstat (limited to 'python/pyspark/ml/feature.py')
0 files changed, 0 insertions, 0 deletions