diff options
| author | hyukjinkwon <gurwls223@gmail.com> | 2016-05-26 20:55:06 -0700 |
|---|---|---|
| committer | Shivaram Venkataraman <shivaram@cs.berkeley.edu> | 2016-05-26 20:55:06 -0700 |
| commit | 1c403733b89258e57daf7b8b0a2011981ad7ed8a (patch) | |
| tree | 4b35e45f6fe8615a7bbc9889824e03333d529c85 /R/WINDOWS.md | |
| parent | 3fca635b4ed322208debcd89a539e42cdde6bbd4 (diff) | |
| download | spark-1c403733b89258e57daf7b8b0a2011981ad7ed8a.tar.gz spark-1c403733b89258e57daf7b8b0a2011981ad7ed8a.tar.bz2 spark-1c403733b89258e57daf7b8b0a2011981ad7ed8a.zip | |
[SPARK-8603][SPARKR] Use shell() instead of system2() for SparkR on Windows
## What changes were proposed in this pull request?
This PR corrects SparkR to use `shell()` instead of `system2()` on Windows.
Using `system2(...)` on Windows does not process windows file separator `\`. `shell(tralsate = TRUE, ...)` can treat this problem. So, this was changed to be chosen according to OS.
Existing tests were failed on Windows due to this problem. For example, those were failed.
```
8. Failure: sparkJars tag in SparkContext (test_includeJAR.R#34)
9. Failure: sparkJars tag in SparkContext (test_includeJAR.R#36)
```
The cases above were due to using of `system2`.
In addition, this PR also fixes some tests failed on Windows.
```
5. Failure: sparkJars sparkPackages as comma-separated strings (test_context.R#128)
6. Failure: sparkJars sparkPackages as comma-separated strings (test_context.R#131)
7. Failure: sparkJars sparkPackages as comma-separated strings (test_context.R#134)
```
The cases above were due to a weird behaviour of `normalizePath()`. On Linux, if the path does not exist, it just prints out the input but it prints out including the current path on Windows.
```r
# On Linus
path <- normalizePath("aa")
print(path)
[1] "aa"
# On Windows
path <- normalizePath("aa")
print(path)
[1] "C:\\Users\\aa"
```
## How was this patch tested?
Jenkins tests and manually tested in a Window machine as below:
Here is the [stdout](https://gist.github.com/HyukjinKwon/4bf35184f3a30f3bce987a58ec2bbbab) of testing.
Closes #7025
Author: hyukjinkwon <gurwls223@gmail.com>
Author: Hyukjin Kwon <gurwls223@gmail.com>
Author: Prakash PC <prakash.chinnu@gmail.com>
Closes #13165 from HyukjinKwon/pr/7025.
Diffstat (limited to 'R/WINDOWS.md')
| -rw-r--r-- | R/WINDOWS.md | 2 |
1 files changed, 1 insertions, 1 deletions
diff --git a/R/WINDOWS.md b/R/WINDOWS.md index f948ed3974..f67a1c51d1 100644 --- a/R/WINDOWS.md +++ b/R/WINDOWS.md @@ -28,6 +28,6 @@ To run the SparkR unit tests on Windows, the following steps are required —ass ``` R -e "install.packages('testthat', repos='http://cran.us.r-project.org')" - .\bin\spark-submit2.cmd --conf spark.hadoop.fs.defualt.name="file:///" R\pkg\tests\run-all.R + .\bin\spark-submit2.cmd --conf spark.hadoop.fs.default.name="file:///" R\pkg\tests\run-all.R ``` |