aboutsummaryrefslogtreecommitdiff
path: root/yarn
Commit message (Collapse)AuthorAgeFilesLines
* Preparing Spark release v1.2.2-rc1v1.2.2Patrick Wendell2015-04-053-3/+3
|
* [SPARK-6018] [YARN] NoSuchMethodError in Spark app is swallowed by YARN AMCheolsoo Park2015-02-261-3/+3
| | | | | | | | | | | | | Author: Cheolsoo Park <cheolsoop@netflix.com> Closes #4773 from piaozhexiu/SPARK-6018 and squashes the following commits: 2a919d5 [Cheolsoo Park] Rename e with cause to avoid duplicate names 1e71d2d [Cheolsoo Park] Replace placeholder with throwable eb5750d [Cheolsoo Park] NoSuchMethodError in Spark app is swallowed by YARN AM (cherry picked from commit 5f3238b3b0157091d28803aa3b1d248dfa6cdc59) Signed-off-by: Andrew Or <andrew@databricks.com>
* SPARK-4267 [YARN] Backport: Failing to launch jobs on Spark on YARN with ↵Sean Owen2015-02-133-12/+17
| | | | | | | | | | | | Hadoop 2.5.0 or later Backport of SPARK-4267 (https://github.com/apache/spark/commit/de7806048ac49a8bfdf44d8f87bc11cea1dfb242) for branch-1.2: Before passing to YARN, escape arguments in "extraJavaOptions" args, in order to correctly handle cases like -Dfoo="one two three". Also standardize how these args are handled and ensure that individual args are treated as stand-alone args, not one string. Author: Sean Owen <sowen@cloudera.com> Closes #4575 from srowen/SPARK-4267-1.2 and squashes the following commits: 7bfc75b [Sean Owen] Backport of SPARK-4267 for branch-1.2: Before passing to YARN, escape arguments in "extraJavaOptions" args, in order to correctly handle cases like -Dfoo="one two three". Also standardize how these args are handled and ensure that individual args are treated as stand-alone args, not one string.
* SPARK-5613: Catch the ApplicationNotFoundException exception to avoid thread ↵Kashish Jain2015-02-061-2/+9
| | | | | | | | | | | | | from getting killed on yarn restart. [SPARK-5613] Added a catch block to catch the ApplicationNotFoundException. Without this catch block the thread gets killed on occurrence of this exception. This Exception occurs when yarn restarts and tries to find an application id for a spark job which got interrupted due to yarn getting stopped. See the stacktrace in the bug for more details. Author: Kashish Jain <kashish.jain@guavus.com> Closes #4392 from kasjain/branch-1.2 and squashes the following commits: 4831000 [Kashish Jain] SPARK-5613: Catch the ApplicationNotFoundException exception to avoid thread from getting killed on yarn restart.
* Preparing development version 1.2.2-SNAPSHOTPatrick Wendell2015-02-033-3/+3
|
* Preparing Spark release v1.2.1-rc3v1.2.1Patrick Wendell2015-02-033-3/+3
|
* Revert "Preparing Spark release v1.2.1-rc2"Patrick Wendell2015-02-023-3/+3
| | | | This reverts commit b77f87673d1f9f03d4c83cf583158227c551359b.
* Revert "Preparing development version 1.2.2-SNAPSHOT"Patrick Wendell2015-02-023-3/+3
| | | | This reverts commit 0a16abadc59082b7d3a24d7f3625236658632813.
* Preparing development version 1.2.2-SNAPSHOTPatrick Wendell2015-01-283-3/+3
|
* Preparing Spark release v1.2.1-rc2Patrick Wendell2015-01-283-3/+3
|
* Revert "Preparing Spark release v1.2.1-rc1"Patrick Wendell2015-01-273-3/+3
| | | | This reverts commit 3e2d7d310b76c293b9ac787f204e6880f508f6ec.
* Revert "Preparing development version 1.2.2-SNAPSHOT"Patrick Wendell2015-01-273-3/+3
| | | | This reverts commit f53a4319ba5f0843c077e64ae5a41e2fac835a5b.
* Preparing development version 1.2.2-SNAPSHOTPatrick Wendell2015-01-273-3/+3
|
* Preparing Spark release v1.2.1-rc1Patrick Wendell2015-01-273-3/+3
|
* Revert "Preparing Spark release v1.2.1-rc1"Patrick Wendell2015-01-263-3/+3
| | | | This reverts commit e87eb2b42f137c22194cfbca2abf06fecdf943da.
* Revert "Preparing development version 1.2.2-SNAPSHOT"Patrick Wendell2015-01-263-3/+3
| | | | This reverts commit adfed7086f10fa8db4eeac7996c84cf98f625e9a.
* Preparing development version 1.2.2-SNAPSHOTUbuntu2015-01-273-3/+3
|
* Preparing Spark release v1.2.1-rc1Ubuntu2015-01-273-3/+3
|
* [SPARK-5006][Deploy]spark.port.maxRetries doesn't workWangTaoTheTonic2015-01-211-6/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | https://issues.apache.org/jira/browse/SPARK-5006 I think the issue is produced in https://github.com/apache/spark/pull/1777. Not digging mesos's backend yet. Maybe should add same logic either. Author: WangTaoTheTonic <barneystinson@aliyun.com> Author: WangTao <barneystinson@aliyun.com> Closes #3841 from WangTaoTheTonic/SPARK-5006 and squashes the following commits: 8cdf96d [WangTao] indent thing 2d86d65 [WangTaoTheTonic] fix line length 7cdfd98 [WangTaoTheTonic] fit for new HttpServer constructor 61a370d [WangTaoTheTonic] some minor fixes bc6e1ec [WangTaoTheTonic] rebase 67bcb46 [WangTaoTheTonic] put conf at 3rd position, modify suite class, add comments f450cd1 [WangTaoTheTonic] startServiceOnPort will use a SparkConf arg 29b751b [WangTaoTheTonic] rebase as ExecutorRunnableUtil changed to ExecutorRunnable 396c226 [WangTaoTheTonic] make the grammar more like scala 191face [WangTaoTheTonic] invalid value name 62ec336 [WangTaoTheTonic] spark.port.maxRetries doesn't work Conflicts: external/mqtt/src/test/scala/org/apache/spark/streaming/mqtt/MQTTStreamSuite.scala
* [SPARK-5289]: Backport publishing of repl, yarn into branch-1.2.Patrick Wendell2015-01-171-14/+0
| | | | | | | | | | | | This change was done in SPARK-4048 as part of a larger refactoring, but we need to backport this publishing of yarn and repl into Spark 1.2, so that we can cut a 1.2.1 release with these artifacts. Author: Patrick Wendell <patrick@databricks.com> Closes #4079 from pwendell/skip-deps and squashes the following commits: 807b833 [Patrick Wendell] [SPARK-5289]: Backport publishing of repl, yarn into branch-1.2.
* [YARN][SPARK-4929] Bug fix: fix the yarn-client code to support HAhuangzhaowei2015-01-071-1/+15
| | | | | | | | | | | | | | | | Nowadays, yarn-client will exit directly when the HA change happens no matter how many times the am should retry. The reason may be that the default final status only considerred the sys.exit, and the yarn-client HA cann't benefit from this. So we should distinct the default final status between client and cluster, because the SUCCEEDED status may cause the HA failed in client mode and UNDEFINED may cause the error reporter in cluster when using sys.exit. Author: huangzhaowei <carlmartinmax@gmail.com> Closes #3771 from SaintBacchus/YarnHA and squashes the following commits: c02bfcc [huangzhaowei] Improve the comment of the funciton 'getDefaultFinalStatus' 0e69924 [huangzhaowei] Bug fix: fix the yarn-client code to support HA (cherry picked from commit 5fde66163fe460d6f64b145047f76cc4ee33601a) Signed-off-by: Thomas Graves <tgraves@apache.org>
* [SPARK-4966][YARN]The MemoryOverhead value is setted not correctlymeiyoula2014-12-291-1/+2
| | | | | | | | | | | Author: meiyoula <1039320815@qq.com> Closes #3797 from XuTingjun/MemoryOverhead and squashes the following commits: 5a780fc [meiyoula] Update ClientArguments.scala (cherry picked from commit 14fa87bdf4b89cd392270864ee063ce01bd31887) Signed-off-by: Thomas Graves <tgraves@apache.org>
* [SPARK-4730][YARN] Warn against deprecated YARN settingsAndrew Or2014-12-231-1/+16
| | | | | | | | | | | | | | See https://issues.apache.org/jira/browse/SPARK-4730. Author: Andrew Or <andrew@databricks.com> Closes #3590 from andrewor14/yarn-settings and squashes the following commits: 36e0753 [Andrew Or] Merge branch 'master' of github.com:apache/spark into yarn-settings dcd1316 [Andrew Or] Warn against deprecated YARN settings (cherry picked from commit 27c5399f4dd542e36ea579956b8cb0613de25c6d) Signed-off-by: Josh Rosen <joshrosen@databricks.com>
* Preparing development version 1.2.1-SNAPSHOTPatrick Wendell2014-12-103-3/+3
|
* Preparing Spark release v1.2.0-rc2v1.2.0Patrick Wendell2014-12-103-3/+3
|
* Revert "Preparing Spark release v1.2.0-rc2"Patrick Wendell2014-12-103-3/+3
| | | | This reverts commit 2b72c569a674cccf79ebbe8d067b8dbaaf78007f.
* Revert "Preparing development version 1.2.1-SNAPSHOT"Patrick Wendell2014-12-103-3/+3
| | | | This reverts commit bc05df8a23ba7ad485f6844f28f96551b13ba461.
* Revert "SPARK-2624 add datanucleus jars to the container in yarn-cluster"Andrew Or2014-12-042-142/+0
| | | | This reverts commit a975dc32799bb8a14f9e1c76defaaa7cfbaf8b53.
* Revert "[HOT FIX] [YARN] Check whether `/lib` exists before listing its files"Andrew Or2014-12-041-15/+12
| | | | This reverts commit 38cb2c3a36a5c9ead4494cbc3dde008c2f0698ce.
* Preparing development version 1.2.1-SNAPSHOTPatrick Wendell2014-12-043-3/+3
|
* Preparing Spark release v1.2.0-rc2Patrick Wendell2014-12-043-3/+3
|
* [SPARK-4253] Ignore spark.driver.host in yarn-cluster and standalone-cluster ↵WangTaoTheTonic2014-12-041-1/+1
| | | | | | | | | | | | | | | | | | | | | | | modes In yarn-cluster and standalone-cluster modes, we don't know where driver will run until it is launched. If the `spark.driver.host` property is set on the submitting machine and propagated to the driver through SparkConf then this will lead to errors when the driver launches. This patch fixes this issue by dropping the `spark.driver.host` property in SparkSubmit when running in a cluster deploy mode. Author: WangTaoTheTonic <barneystinson@aliyun.com> Author: WangTao <barneystinson@aliyun.com> Closes #3112 from WangTaoTheTonic/SPARK4253 and squashes the following commits: ed1a25c [WangTaoTheTonic] revert unrelated formatting issue 02c4e49 [WangTao] add comment 32a3f3f [WangTaoTheTonic] ingore it in SparkSubmit instead of SparkContext 667cf24 [WangTaoTheTonic] document fix ff8d5f7 [WangTaoTheTonic] also ignore it in standalone cluster mode 2286e6b [WangTao] ignore spark.driver.host in yarn-cluster mode (cherry picked from commit 8106b1e36b2c2b9f5dc5d7252540e48cc3fc96d5) Signed-off-by: Josh Rosen <joshrosen@databricks.com>
* Revert "Preparing Spark release v1.2.0-rc1"Patrick Wendell2014-12-043-3/+3
| | | | This reverts commit 1056e9ec13203d0c51564265e94d77a054498fdb.
* Revert "Preparing development version 1.2.1-SNAPSHOT"Patrick Wendell2014-12-043-3/+3
| | | | This reverts commit 00316cc87983b844f6603f351a8f0b84fe1f6035.
* [HOT FIX] [YARN] Check whether `/lib` exists before listing its filesAndrew Or2014-12-031-12/+15
| | | | | | | | | | | | | This is caused by a975dc32799bb8a14f9e1c76defaaa7cfbaf8b53 Author: Andrew Or <andrew@databricks.com> Closes #3589 from andrewor14/yarn-hot-fix and squashes the following commits: a4fad5f [Andrew Or] Check whether lib directory exists before listing its files (cherry picked from commit 90ec643e9af4c8bbb9000edca08c07afb17939c7) Signed-off-by: Andrew Or <andrew@databricks.com>
* SPARK-2624 add datanucleus jars to the container in yarn-clusterJim Lim2014-12-032-0/+142
| | | | | | | | | | | | | | | | | If `spark-submit` finds the datanucleus jars, it adds them to the driver's classpath, but does not add it to the container. This patch modifies the yarn deployment class to copy all `datanucleus-*` jars found in `[spark-home]/libs` to the container. Author: Jim Lim <jim@quixey.com> Closes #3238 from jimjh/SPARK-2624 and squashes the following commits: 3633071 [Jim Lim] SPARK-2624 update documentation and comments fe95125 [Jim Lim] SPARK-2624 keep java imports together 6c31fe0 [Jim Lim] SPARK-2624 update documentation 6690fbf [Jim Lim] SPARK-2624 add tests d28d8e9 [Jim Lim] SPARK-2624 add spark.yarn.datanucleus.dir option 84e6cba [Jim Lim] SPARK-2624 add datanucleus jars to the container in yarn-cluster
* Preparing development version 1.2.1-SNAPSHOTPatrick Wendell2014-11-283-3/+3
|
* Preparing Spark release v1.2.0-rc1Patrick Wendell2014-11-283-3/+3
|
* Revert "Preparing Spark release v1.2.0-rc1"Patrick Wendell2014-11-283-3/+3
| | | | This reverts commit 39c7d1c1f9a7785285cf4c20dfbffd96f72d5634.
* Revert "Preparing development version 1.2.1-SNAPSHOT"Patrick Wendell2014-11-283-3/+3
| | | | This reverts commit fc7bff00ac731d2632213a98cd92dc5e84ce7dcd.
* Preparing development version 1.2.1-SNAPSHOTPatrick Wendell2014-11-283-3/+3
|
* Preparing Spark release v1.2.0-rc1Patrick Wendell2014-11-283-3/+3
|
* [SPARK-4584] [yarn] Remove security manager from Yarn AM.Marcelo Vanzin2014-11-281-46/+14
| | | | | | | | | | | | | | | | | | | | | | | | | | | The security manager adds a lot of overhead to the runtime of the app, and causes a severe performance regression. Even stubbing out all unneeded methods (all except checkExit()) does not help. So, instead, penalize users who do an explicit System.exit() by leaving them in "undefined behavior" territory: if they do that, the Yarn backend won't be able to report the final app status to the RM. The result is that the final status of the application might not match the user's expectations. One side-effect of the change is that users who do an explicit System.exit() will lose the AM retry functionality. Since there is no way to know if the exit was because of success or failure, the AM right now errs on the side of it being a successful exit. Author: Marcelo Vanzin <vanzin@cloudera.com> Closes #3484 from vanzin/SPARK-4584 and squashes the following commits: 21f2502 [Marcelo Vanzin] Do not retry apps that use System.exit(). 4198b3b [Marcelo Vanzin] [SPARK-4584] [yarn] Remove security manager from Yarn AM. (cherry picked from commit 915f8eeb3a493a0bb4b8d05d795ddd21f373d2ff) Signed-off-by: Patrick Wendell <pwendell@gmail.com>
* Revert "Preparing Spark release v1.2.0-rc1"Patrick Wendell2014-11-263-3/+3
| | | | This reverts commit cc2c05e4ee81d2f34873a2ebb9a5272867cb65c2.
* Revert "Preparing development version 1.2.1-SNAPSHOT"Patrick Wendell2014-11-263-3/+3
| | | | This reverts commit 380eba5f49eca1dbd4084e6c84e19866fffd4efa.
* Preparing development version 1.2.1-SNAPSHOTPatrick Wendell2014-11-263-3/+3
|
* Preparing Spark release v1.2.0-rc1Patrick Wendell2014-11-263-3/+3
|
* Revert "Preparing Spark release v1.2.0-rc1"Patrick Wendell2014-11-263-3/+3
| | | | This reverts commit 5247dd859b95a440baa562b9827bdeb26aa6530e.
* Revert "Preparing development version 1.2.1-SNAPSHOT"Patrick Wendell2014-11-263-3/+3
| | | | This reverts commit 79df6b43ae762263a8120f423ddb4a0811dd4b6f.
* Preparing development version 1.2.1-SNAPSHOTPatrick Wendell2014-11-263-3/+3
|
generated by cgit v1.2.3 (git 2.39.1) at 2024-09-16 07:26:47 +0000