| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
- Move external/java8-tests tests into core, streaming, sql and remove
- Remove MaxPermGen and related options
- Fix some reflection / TODOs around Java 8+ methods
- Update doc references to 1.7/1.8 differences
- Remove Java 7/8 related build profiles
- Update some plugins for better Java 8 compatibility
- Fix a few Java-related warnings
For the future:
- Update Java 8 examples to fully use Java 8
- Update Java tests to use lambdas for simplicity
- Update Java internal implementations to use lambdas
## How was this patch tested?
Existing tests
Author: Sean Owen <sowen@cloudera.com>
Closes #16871 from srowen/SPARK-19493.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
## What changes were proposed in this pull request?
It seems wintuils for Hadoop 2.6.5 not exiting for now in https://github.com/steveloughran/winutils
This breaks the tests in SparkR on Windows so this PR proposes to use winutils built by Hadoop 2.6.4 for now.
## How was this patch tested?
Manually via AppVeyor
**Before**
https://ci.appveyor.com/project/spark-test/spark/build/627-r-test-break
**After**
https://ci.appveyor.com/project/spark-test/spark/build/629-r-test-break
Author: hyukjinkwon <gurwls223@gmail.com>
Closes #16927 from HyukjinKwon/spark-r-windows-break.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
earlier
## What changes were proposed in this pull request?
- Remove support for Hadoop 2.5 and earlier
- Remove reflection and code constructs only needed to support multiple versions at once
- Update docs to reflect newer versions
- Remove older versions' builds and profiles.
## How was this patch tested?
Existing tests
Author: Sean Owen <sowen@cloudera.com>
Closes #16810 from srowen/SPARK-19464.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
AppVeyor tests for Hadoop libraries to call native codes properly
## What changes were proposed in this pull request?
It seems Hadoop libraries need winutils binaries for native libraries in the path.
It is not a problem in tests for now because we are only testing SparkR on Windows via AppVeyor but it can be a problem if we run Scala tests via AppVeyor as below:
```
- SPARK-18220: read Hive orc table with varchar column *** FAILED *** (3 seconds, 937 milliseconds)
org.apache.spark.sql.execution.QueryExecutionException: FAILED: Execution Error, return code -101 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask. org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z
at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$runHive$1.apply(HiveClientImpl.scala:625)
at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$runHive$1.apply(HiveClientImpl.scala:609)
at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$withHiveState$1.apply(HiveClientImpl.scala:283)
...
```
This PR proposes to add it to the `Path` for AppVeyor tests.
## How was this patch tested?
Manually via AppVeyor.
**Before**
https://ci.appveyor.com/project/spark-test/spark/build/549-windows-complete/job/gc8a1pjua2bc4i8m
**After**
https://ci.appveyor.com/project/spark-test/spark/build/572-windows-complete/job/c4vrysr5uvj2hgu7
Author: hyukjinkwon <gurwls223@gmail.com>
Closes #16584 from HyukjinKwon/set-path-appveyor.
|
|
|
Windows (currently SparkR only)
## What changes were proposed in this pull request?
This PR adds the build automation on Windows with [AppVeyor](https://www.appveyor.com/) CI tool.
Currently, this only runs the tests for SparkR as we have been having some issues with testing Windows-specific PRs (e.g. https://github.com/apache/spark/pull/14743 and https://github.com/apache/spark/pull/13165) and hard time to verify this.
One concern is, this build is dependent on [steveloughran/winutils](https://github.com/steveloughran/winutils) for pre-built Hadoop bin package (who is a Hadoop PMC member).
## How was this patch tested?
Manually, https://ci.appveyor.com/project/HyukjinKwon/spark/build/88-SPARK-17200-build-profile
This takes roughly 40 mins.
Some tests are already being failed and this was found in https://github.com/apache/spark/pull/14743#issuecomment-241405287.
Author: hyukjinkwon <gurwls223@gmail.com>
Closes #14859 from HyukjinKwon/SPARK-17200-build.
|
