diff options
| author | Sun Rui <rui.sun@intel.com> | 2015-07-13 08:21:47 -0700 |
|---|---|---|
| committer | Shivaram Venkataraman <shivaram@cs.berkeley.edu> | 2015-07-13 08:21:47 -0700 |
| commit | 7f487c8bde14dbdd244a3493ad11a129ef2bb327 (patch) | |
| tree | e127a2d84beb0010eadc5c0bac664d15ebc22667 /project | |
| parent | a5bc803b7271028e34de1548d55b80ecfb812a7b (diff) | |
| download | spark-7f487c8bde14dbdd244a3493ad11a129ef2bb327.tar.gz spark-7f487c8bde14dbdd244a3493ad11a129ef2bb327.tar.bz2 spark-7f487c8bde14dbdd244a3493ad11a129ef2bb327.zip | |
[SPARK-6797] [SPARKR] Add support for YARN cluster mode.
This PR enables SparkR to dynamically ship the SparkR binary package to the AM node in YARN cluster mode, thus it is no longer required that the SparkR package be installed on each worker node.
This PR uses the JDK jar tool to package the SparkR package, because jar is thought to be available on both Linux/Windows platforms where JDK has been installed.
This PR does not address the R worker involved in RDD API. Will address it in a separate JIRA issue.
This PR does not address SBT build. SparkR installation and packaging by SBT will be addressed in a separate JIRA issue.
R/install-dev.bat is not tested. shivaram , Could you help to test it?
Author: Sun Rui <rui.sun@intel.com>
Closes #6743 from sun-rui/SPARK-6797 and squashes the following commits:
ca63c86 [Sun Rui] Adjust MimaExcludes after rebase.
7313374 [Sun Rui] Fix unit test errors.
72695fb [Sun Rui] Fix unit test failures.
193882f [Sun Rui] Fix Mima test error.
fe25a33 [Sun Rui] Fix Mima test error.
35ecfa3 [Sun Rui] Fix comments.
c38a005 [Sun Rui] Unzipped SparkR binary package is still required for standalone and Mesos modes.
b05340c [Sun Rui] Fix scala style.
2ca5048 [Sun Rui] Fix comments.
1acefd1 [Sun Rui] Fix scala style.
0aa1e97 [Sun Rui] Fix scala style.
41d4f17 [Sun Rui] Add support for locating SparkR package for R workers required by RDD APIs.
49ff948 [Sun Rui] Invoke jar.exe with full path in install-dev.bat.
7b916c5 [Sun Rui] Use 'rem' consistently.
3bed438 [Sun Rui] Add a comment.
681afb0 [Sun Rui] Fix a bug that RRunner does not handle client deployment modes.
cedfbe2 [Sun Rui] [SPARK-6797][SPARKR] Add support for YARN cluster mode.
Diffstat (limited to 'project')
| -rw-r--r-- | project/MimaExcludes.scala | 12 |
1 files changed, 12 insertions, 0 deletions
diff --git a/project/MimaExcludes.scala b/project/MimaExcludes.scala index 79089aae2a..4e4e810ec3 100644 --- a/project/MimaExcludes.scala +++ b/project/MimaExcludes.scala @@ -83,6 +83,18 @@ object MimaExcludes { "org.apache.spark.streaming.scheduler.InputInfo$"), ProblemFilters.exclude[MissingClassProblem]( "org.apache.spark.streaming.scheduler.InputInfo") + ) ++ Seq( + // SPARK-6797 Support YARN modes for SparkR + ProblemFilters.exclude[MissingMethodProblem]( + "org.apache.spark.api.r.PairwiseRRDD.this"), + ProblemFilters.exclude[MissingMethodProblem]( + "org.apache.spark.api.r.RRDD.createRWorker"), + ProblemFilters.exclude[MissingMethodProblem]( + "org.apache.spark.api.r.RRDD.this"), + ProblemFilters.exclude[MissingMethodProblem]( + "org.apache.spark.api.r.StringRRDD.this"), + ProblemFilters.exclude[MissingMethodProblem]( + "org.apache.spark.api.r.BaseRRDD.this") ) case v if v.startsWith("1.4") => |