diff options
| author | hyukjinkwon <gurwls223@gmail.com> | 2017-01-14 08:31:07 -0800 |
|---|---|---|
| committer | Shivaram Venkataraman <shivaram@cs.berkeley.edu> | 2017-01-14 08:31:07 -0800 |
| commit | b6a7aa4f770634e6db7244e88f8b6273fb9b6d1e (patch) | |
| tree | 6f344c1df92d439a5d611d3c077a4cc7c262bd86 /dev/appveyor-install-dependencies.ps1 | |
| parent | ad0dadaa251b031a480fc2080f792a54ed7dfc5f (diff) | |
| download | spark-b6a7aa4f770634e6db7244e88f8b6273fb9b6d1e.tar.gz spark-b6a7aa4f770634e6db7244e88f8b6273fb9b6d1e.tar.bz2 spark-b6a7aa4f770634e6db7244e88f8b6273fb9b6d1e.zip | |
[SPARK-19221][PROJECT INFRA][R] Add winutils binaries to the path in AppVeyor tests for Hadoop libraries to call native codes properly
## What changes were proposed in this pull request?
It seems Hadoop libraries need winutils binaries for native libraries in the path.
It is not a problem in tests for now because we are only testing SparkR on Windows via AppVeyor but it can be a problem if we run Scala tests via AppVeyor as below:
```
- SPARK-18220: read Hive orc table with varchar column *** FAILED *** (3 seconds, 937 milliseconds)
org.apache.spark.sql.execution.QueryExecutionException: FAILED: Execution Error, return code -101 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask. org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z
at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$runHive$1.apply(HiveClientImpl.scala:625)
at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$runHive$1.apply(HiveClientImpl.scala:609)
at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$withHiveState$1.apply(HiveClientImpl.scala:283)
...
```
This PR proposes to add it to the `Path` for AppVeyor tests.
## How was this patch tested?
Manually via AppVeyor.
**Before**
https://ci.appveyor.com/project/spark-test/spark/build/549-windows-complete/job/gc8a1pjua2bc4i8m
**After**
https://ci.appveyor.com/project/spark-test/spark/build/572-windows-complete/job/c4vrysr5uvj2hgu7
Author: hyukjinkwon <gurwls223@gmail.com>
Closes #16584 from HyukjinKwon/set-path-appveyor.
Diffstat (limited to 'dev/appveyor-install-dependencies.ps1')
| -rw-r--r-- | dev/appveyor-install-dependencies.ps1 | 1 |
1 files changed, 1 insertions, 0 deletions
diff --git a/dev/appveyor-install-dependencies.ps1 b/dev/appveyor-install-dependencies.ps1 index 087b8666cc..b72d6b53bc 100644 --- a/dev/appveyor-install-dependencies.ps1 +++ b/dev/appveyor-install-dependencies.ps1 @@ -109,6 +109,7 @@ Invoke-Expression "7z.exe x winutils-master.zip" # add hadoop bin to environment variables $env:HADOOP_HOME = "$hadoopPath/winutils-master/hadoop-$hadoopVer" +$env:Path += ";$env:HADOOP_HOME\bin" Pop-Location |