aboutsummaryrefslogtreecommitdiff
path: root/yarn
Commit message (Collapse)AuthorAgeFilesLines
* Revert "[maven-release-plugin] prepare for next development iteration"Patrick Wendell2014-05-162-2/+2
| | | | This reverts commit e5436b8c1a79ce108f3af402455ac5f6dc5d1eb3.
* [maven-release-plugin] prepare for next development iterationPatrick Wendell2014-05-162-2/+2
|
* [maven-release-plugin] prepare release v1.0.0-rc8Patrick Wendell2014-05-162-2/+2
|
* Revert "[maven-release-plugin] prepare release v1.0.0-rc7"Patrick Wendell2014-05-162-2/+2
| | | | This reverts commit 9212b3e5bb5545ccfce242da8d89108e6fb1c464.
* Revert "[maven-release-plugin] prepare for next development iteration"Patrick Wendell2014-05-162-2/+2
| | | | This reverts commit c4746aa6fe4aaf383e69e34353114d36d1eb9ba6.
* [maven-release-plugin] prepare for next development iterationPatrick Wendell2014-05-152-2/+2
|
* [maven-release-plugin] prepare release v1.0.0-rc7Patrick Wendell2014-05-152-2/+2
|
* Revert "[maven-release-plugin] prepare release v1.0.0-rc6"Patrick Wendell2014-05-142-2/+2
| | | | This reverts commit 54133abdce0246f6643a1112a5204afb2c4caa82.
* Revert "[maven-release-plugin] prepare for next development iteration"Patrick Wendell2014-05-142-2/+2
| | | | This reverts commit e480bcfbd269ae1d7a6a92cfb50466cf192fe1fb.
* [maven-release-plugin] prepare for next development iterationPatrick Wendell2014-05-142-2/+2
|
* [maven-release-plugin] prepare release v1.0.0-rc6Patrick Wendell2014-05-142-2/+2
|
* Revert "[maven-release-plugin] prepare release v1.0.0-rc5"Patrick Wendell2014-05-142-2/+2
| | | | This reverts commit 18f062303303824139998e8fc8f4158217b0dbc3.
* Revert "[maven-release-plugin] prepare for next development iteration"Patrick Wendell2014-05-142-2/+2
| | | | This reverts commit d08e9604fc9958b7c768e91715c8152db2ed6fd0.
* [maven-release-plugin] prepare for next development iterationPatrick Wendell2014-05-132-2/+2
|
* [maven-release-plugin] prepare release v1.0.0-rc5Patrick Wendell2014-05-132-2/+2
|
* Revert "[maven-release-plugin] prepare release v1.0.0-rc4"Patrick Wendell2014-05-122-2/+2
| | | | This reverts commit 3d0a44833ab50360bf9feccc861cb5e8c44a4866.
* Revert "[maven-release-plugin] prepare for next development iteration"Patrick Wendell2014-05-122-2/+2
| | | | This reverts commit 9772d85c6f3893d42044f4bab0e16f8b6287613a.
* [maven-release-plugin] prepare for next development iterationPatrick Wendell2014-05-132-2/+2
|
* [maven-release-plugin] prepare release v1.0.0-rc4Patrick Wendell2014-05-132-2/+2
|
* Rollback versions for 1.0.0-rc4Patrick Wendell2014-05-122-2/+2
|
* [maven-release-plugin] prepare for next development iterationPatrick Wendell2014-05-122-2/+2
|
* [maven-release-plugin] prepare release v1.0.0-rc4Patrick Wendell2014-05-122-2/+2
|
* [SPARK-1774] Respect SparkSubmit --jars on YARN (client)Andrew Or2014-05-101-3/+1
| | | | | | | | | | | | | | | | | | | | | | | SparkSubmit ignores `--jars` for YARN client. This is a bug. This PR also automatically adds the application jar to `spark.jar`. Previously, when running as yarn-client, you must specify the jar additionally through `--files` (because `--jars` didn't work). Now you don't have to explicitly specify it through either. Tested on a YARN cluster. Author: Andrew Or <andrewor14@gmail.com> Closes #710 from andrewor14/yarn-jars and squashes the following commits: 35d1928 [Andrew Or] Merge branch 'master' of github.com:apache/spark into yarn-jars c27bf6c [Andrew Or] For yarn-cluster and python, do not add primaryResource to spark.jar c92c5bf [Andrew Or] Minor cleanups 269f9f3 [Andrew Or] Fix format 013d840 [Andrew Or] Fix tests 1407474 [Andrew Or] Merge branch 'master' of github.com:apache/spark into yarn-jars 3bb75e8 [Andrew Or] Allow SparkSubmit --jars to take effect in yarn-client mode (cherry picked from commit 83e0424d87022e7a967088365931a08aa06ffd9f) Signed-off-by: Patrick Wendell <pwendell@gmail.com>
* [SPARK-1631] Correctly set the Yarn app name when launching the AM.Marcelo Vanzin2014-05-081-3/+3
| | | | | | | | | | | Author: Marcelo Vanzin <vanzin@cloudera.com> Closes #539 from vanzin/yarn-app-name and squashes the following commits: 7d1ca4f [Marcelo Vanzin] [SPARK-1631] Correctly set the Yarn app name when launching the AM. (cherry picked from commit 3f779d872d8459b262b3db9e4d12b011910b6ce9) Signed-off-by: Patrick Wendell <pwendell@gmail.com>
* SPARK-1569 Spark on Yarn, authentication broken by pr299Thomas Graves2014-05-071-19/+30
| | | | | | | | | | | | | | | | | Pass the configs as java options since the executor needs to know before it registers whether to create the connection using authentication or not. We could see about passing only the authentication configs but for now I just had it pass them all. I also updating it to use a list to construct the command to make it the same as ClientBase and avoid any issues with spaces. Author: Thomas Graves <tgraves@apache.org> Closes #649 from tgravescs/SPARK-1569 and squashes the following commits: 0178ab8 [Thomas Graves] add akka settings 22a8735 [Thomas Graves] Change to only path spark.auth* configs 8ccc1d4 [Thomas Graves] SPARK-1569 Spark on Yarn, authentication broken (cherry picked from commit 4bec84b6a23e1e642708a70a6c7ef7b3d1df9b3e) Signed-off-by: Patrick Wendell <pwendell@gmail.com>
* SPARK-1474: Spark on yarn assembly doesn't include AmIpFilterThomas Graves2014-05-061-0/+4
| | | | | | | | | | | | | | | We use org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter in spark on yarn but are not included it in the assembly jar. I tested this on yarn cluster by removing the yarn jars from the classpath and spark runs fine now. Author: Thomas Graves <tgraves@apache.org> Closes #406 from tgravescs/SPARK-1474 and squashes the following commits: 1548bf9 [Thomas Graves] SPARK-1474: Spark on yarn assembly doesn't include org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter (cherry picked from commit 1e829905c791fbf1dfd8e0c1caa62ead7354605e) Signed-off-by: Patrick Wendell <pwendell@gmail.com>
* The default version of yarn is equal to the hadoop versionwitgo2014-05-031-3/+2
| | | | | | | | | | | | | | | | | | This is a part of [PR 590](https://github.com/apache/spark/pull/590) Author: witgo <witgo@qq.com> Closes #626 from witgo/yarn_version and squashes the following commits: c390631 [witgo] restore the yarn dependency declarations f8a4ad8 [witgo] revert remove the dependency of avro in yarn-alpha 2df6cf5 [witgo] review commit a1d876a [witgo] review commit 20e7e3e [witgo] review commit c76763b [witgo] The default value of yarn.version is equal to hadoop.version (cherry picked from commit fb0543224bcedb8ae3aab4a7ddcc6111a03378fe) Signed-off-by: Patrick Wendell <pwendell@gmail.com>
* [WIP] SPARK-1676: Cache Hadoop UGIs by default to prevent FileSystem leakThomas Graves2014-05-034-15/+21
| | | | | | | | | | | | | | | | | | | | | | Move the doAs in Executor higher up so that we only have 1 ugi and aren't leaking filesystems. Fix spark on yarn to work when the cluster is running as user "yarn" but the clients are launched as the user and want to read/write to hdfs as the user. Note this hasn't been fully tested yet. Need to test in standalone mode. Putting this up for people to look at and possibly test. I don't have access to a mesos cluster. This is alternative to https://github.com/apache/spark/pull/607 Author: Thomas Graves <tgraves@apache.org> Closes #621 from tgravescs/SPARK-1676 and squashes the following commits: 244d55a [Thomas Graves] fix line length 44163d4 [Thomas Graves] Rework 9398853 [Thomas Graves] change to have doAs in executor higher up. (cherry picked from commit 3d0a02dff3011e8894d98d903cd086bc95e56807) Signed-off-by: Aaron Davidson <aaron@databricks.com>
* SPARK-1588. Restore SPARK_YARN_USER_ENV and SPARK_JAVA_OPTS for YARN.Sandy Ryza2014-04-292-5/+15
| | | | | | | | | Author: Sandy Ryza <sandy@cloudera.com> Closes #586 from sryza/sandy-spark-1588 and squashes the following commits: 35eb38e [Sandy Ryza] Scalify b361684 [Sandy Ryza] SPARK-1588. Restore SPARK_YARN_USER_ENV and SPARK_JAVA_OPTS for YARN.
* [maven-release-plugin] prepare for next development iterationPatrick Wendell2014-04-292-2/+2
|
* [maven-release-plugin] prepare release v1.0.0-rc3Patrick Wendell2014-04-292-2/+2
|
* Manual revert of rc2 version changes.Patrick Wendell2014-04-282-2/+2
|
* Improved build configurationwitgo2014-04-283-37/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 1, Fix SPARK-1441: compile spark core error with hadoop 0.23.x 2, Fix SPARK-1491: maven hadoop-provided profile fails to build 3, Fix org.scala-lang: * ,org.apache.avro:* inconsistent versions dependency 4, A modified on the sql/catalyst/pom.xml,sql/hive/pom.xml,sql/core/pom.xml (Four spaces formatted into two spaces) Author: witgo <witgo@qq.com> Closes #480 from witgo/format_pom and squashes the following commits: 03f652f [witgo] review commit b452680 [witgo] Merge branch 'master' of https://github.com/apache/spark into format_pom bee920d [witgo] revert fix SPARK-1629: Spark Core missing commons-lang dependence 7382a07 [witgo] Merge branch 'master' of https://github.com/apache/spark into format_pom 6902c91 [witgo] fix SPARK-1629: Spark Core missing commons-lang dependence 0da4bc3 [witgo] merge master d1718ed [witgo] Merge branch 'master' of https://github.com/apache/spark into format_pom e345919 [witgo] add avro dependency to yarn-alpha 77fad08 [witgo] Merge branch 'master' of https://github.com/apache/spark into format_pom 62d0862 [witgo] Fix org.scala-lang: * inconsistent versions dependency 1a162d7 [witgo] Merge branch 'master' of https://github.com/apache/spark into format_pom 934f24d [witgo] review commit cf46edc [witgo] exclude jruby 06e7328 [witgo] Merge branch 'SparkBuild' into format_pom 99464d2 [witgo] fix maven hadoop-provided profile fails to build 0c6c1fc [witgo] Fix compile spark core error with hadoop 0.23.x 6851bec [witgo] Maintain consistent SparkBuild.scala, pom.xml (cherry picked from commit 030f2c2126d5075576cd6d83a1ee7462c48b953b) Conflicts: sql/catalyst/pom.xml sql/core/pom.xml sql/hive/pom.xml
* SPARK-1652: Remove incorrect deprecation warning in spark-submitPatrick Wendell2014-04-282-4/+8
| | | | | | | | | | | | | | | | | | This is a straightforward fix. Author: Patrick Wendell <pwendell@gmail.com> This patch had conflicts when merged, resolved by Committer: Patrick Wendell <pwendell@gmail.com> Closes #578 from pwendell/spark-submit-yarn and squashes the following commits: 96027c7 [Patrick Wendell] Test fixes b5be173 [Patrick Wendell] Review feedback 4ac9cac [Patrick Wendell] SPARK-1652: spark-submit for yarn prints warnings even though calling as expected (cherry picked from commit 9f7a095184d6c7a9b1bbac55efcc3d878f876768) Signed-off-by: Patrick Wendell <pwendell@gmail.com>
* SPARK-1607. HOTFIX: Fix syntax adapting Int result to ShortSean Owen2014-04-251-2/+2
| | | | | | | | | | | | | Sorry folks. This should make the change for SPARK-1607 compile again. Verified this time with the yarn build enabled. Author: Sean Owen <sowen@cloudera.com> Closes #556 from srowen/SPARK-1607.2 and squashes the following commits: e3fe7a3 [Sean Owen] Fix syntax adapting Int result to Short (cherry picked from commit df6d81425bf3b8830988288069f6863de873aee2) Signed-off-by: Aaron Davidson <aaron@databricks.com>
* SPARK-1607. Replace octal literals, removed in Scala 2.11, with hex literalsSean Owen2014-04-241-2/+4
| | | | | | | | | | | | | Octal literals like "0700" are deprecated in Scala 2.10, generating a warning. They have been removed entirely in 2.11. See https://issues.scala-lang.org/browse/SI-7618 This change simply replaces two uses of octals with hex literals, which seemed the next-best representation since they express a bit mask (file permission in particular) Author: Sean Owen <sowen@cloudera.com> Closes #529 from srowen/SPARK-1607 and squashes the following commits: 1ee0e67 [Sean Owen] Use Integer.parseInt(...,8) for octal literal instead of hex equivalent 0102f3d [Sean Owen] Replace octal literals, removed in Scala 2.11, with hex literals
* SPARK-1586 Windows build fixesMridul Muralidharan2014-04-241-1/+1
| | | | | | | | | | | | | | | | | | | | | | Unfortunately, this is not exhaustive - particularly hive tests still fail due to path issues. Author: Mridul Muralidharan <mridulm80@apache.org> This patch had conflicts when merged, resolved by Committer: Matei Zaharia <matei@databricks.com> Closes #505 from mridulm/windows_fixes and squashes the following commits: ef12283 [Mridul Muralidharan] Move to org.apache.commons.lang3 for StringEscapeUtils. Earlier version was buggy appparently cdae406 [Mridul Muralidharan] Remove leaked changes from > 2G fix branch 3267f4b [Mridul Muralidharan] Fix build failures 35b277a [Mridul Muralidharan] Fix Scalastyle failures bc69d14 [Mridul Muralidharan] Change from hardcoded path separator 10c4d78 [Mridul Muralidharan] Use explicit encoding while using getBytes 1337abd [Mridul Muralidharan] fix classpath while running in windows (cherry picked from commit 968c0187a12f5ae4a696c02c1ff088e998ed7edd) Signed-off-by: Matei Zaharia <matei@databricks.com>
* Fix Scala StyleSandeep2014-04-243-38/+41
| | | | | | | | | | | | | | | | | Any comments are welcome Author: Sandeep <sandeep@techaddict.me> Closes #531 from techaddict/stylefix-1 and squashes the following commits: 7492730 [Sandeep] Pass 4 98b2428 [Sandeep] fix rxin suggestions b5e2e6f [Sandeep] Pass 3 05932d7 [Sandeep] fix if else styling 2 08690e5 [Sandeep] fix if else styling (cherry picked from commit a03ac222d84025a1036750e1179136a13f75dea7) Signed-off-by: Reynold Xin <rxin@apache.org>
* [maven-release-plugin] prepare for next development iterationPatrick Wendell2014-04-242-2/+2
|
* [maven-release-plugin] prepare release v1.0.0-rc2Patrick Wendell2014-04-242-2/+2
|
* Assorted clean-up for Spark-on-YARN.Patrick Wendell2014-04-221-3/+5
| | | | | | | | | | | | | | | In particular when the HADOOP_CONF_DIR is not not specified. Author: Patrick Wendell <pwendell@gmail.com> Closes #488 from pwendell/hadoop-cleanup and squashes the following commits: fe95f13 [Patrick Wendell] Changes based on Andrew's feeback 18d09c1 [Patrick Wendell] Review comments from Andrew 17929cc [Patrick Wendell] Assorted clean-up for Spark-on-YARN. (cherry picked from commit 995fdc96bcd2c540804401eaab009a777d7d7aa9) Signed-off-by: Patrick Wendell <pwendell@gmail.com>
* Fix compilation on Hadoop 2.4.x.Marcelo Vanzin2014-04-221-1/+2
| | | | | | | | | | | Author: Marcelo Vanzin <vanzin@cloudera.com> Closes #483 from vanzin/yarn-2.4 and squashes the following commits: 0fc57d8 [Marcelo Vanzin] Fix compilation on Hadoop 2.4.x. (cherry picked from commit 0ea0b1a2d66044af97ab84bf3014ff8dfe91fee4) Signed-off-by: Patrick Wendell <pwendell@gmail.com>
* Updating versions for Spark 1.0Patrick Wendell2014-04-211-1/+1
|
* Revert "[maven-release-plugin] prepare release v1.0.0-rc1"Patrick Wendell2014-04-212-2/+2
| | | | This reverts commit 6cc698fc378256fee9111f66c691ced27f54e973.
* Revert "[maven-release-plugin] prepare for next development iteration"Patrick Wendell2014-04-212-2/+2
| | | | This reverts commit 188f7c3f68e93b3e9347ec02e21f5943874b4741.
* [maven-release-plugin] prepare for next development iterationPatrick Wendell2014-04-212-2/+2
|
* [maven-release-plugin] prepare release v1.0.0-rc1Patrick Wendell2014-04-212-2/+2
|
* Revert "[maven-release-plugin] prepare release v1.0.0"Patrick Wendell2014-04-212-2/+2
| | | | This reverts commit c3c6ea05d9d02a38b97388d583828ca82c5181db.
* Revert "[maven-release-plugin] prepare for next development iteration"Patrick Wendell2014-04-212-2/+2
| | | | This reverts commit 0b49305297033b70cbb525bef54b70a14deeb238.
* [maven-release-plugin] prepare for next development iterationPatrick Wendell2014-04-212-2/+2
|
generated by cgit v1.2.3 (git 2.39.1) at 2024-09-17 23:39:04 +0000