diff options
| author | Yin Huai <yhuai@databricks.com> | 2015-11-26 16:20:08 -0800 |
|---|---|---|
| committer | Yin Huai <yhuai@databricks.com> | 2015-11-26 16:20:08 -0800 |
| commit | ad76562390b81207f8f32491c0bd8ad0e020141f (patch) | |
| tree | 65058912455c4195ac12e389cf6e619f9581ee16 /docs/mllib-linear-methods.md | |
| parent | bc16a67562560c732833260cbc34825f7e9dcb8f (diff) | |
| download | spark-ad76562390b81207f8f32491c0bd8ad0e020141f.tar.gz spark-ad76562390b81207f8f32491c0bd8ad0e020141f.tar.bz2 spark-ad76562390b81207f8f32491c0bd8ad0e020141f.zip | |
[SPARK-11998][SQL][TEST-HADOOP2.0] When downloading Hadoop artifacts from maven, we need to try to download the version that is used by Spark
If we need to download Hive/Hadoop artifacts, try to download a Hadoop that matches the Hadoop used by Spark. If the Hadoop artifact cannot be resolved (e.g. Hadoop version is a vendor specific version like 2.0.0-cdh4.1.1), we will use Hadoop 2.4.0 (we used to hard code this version as the hadoop that we will download from maven) and we will not share Hadoop classes.
I tested this match in my laptop with the following confs (these confs are used by our builds). All tests are good.
```
build/sbt -Phadoop-1 -Dhadoop.version=1.2.1 -Pkinesis-asl -Phive-thriftserver -Phive
build/sbt -Phadoop-1 -Dhadoop.version=2.0.0-mr1-cdh4.1.1 -Pkinesis-asl -Phive-thriftserver -Phive
build/sbt -Pyarn -Phadoop-2.2 -Pkinesis-asl -Phive-thriftserver -Phive
build/sbt -Pyarn -Phadoop-2.3 -Dhadoop.version=2.3.0 -Pkinesis-asl -Phive-thriftserver -Phive
```
Author: Yin Huai <yhuai@databricks.com>
Closes #9979 from yhuai/versionsSuite.
Diffstat (limited to 'docs/mllib-linear-methods.md')
0 files changed, 0 insertions, 0 deletions