| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
#511 and #863 got left out of branch-1.0 since we were really close to the release. Now that they have been tested a little I see no reason to leave them out.
Author: Michael Armbrust <michael@databricks.com>
Author: witgo <witgo@qq.com>
Closes #1078 from marmbrus/branch-1.0 and squashes the following commits:
22be674 [witgo] [SPARK-1841]: update scalatest to version 2.1.5
fc8fc79 [Michael Armbrust] Include #1071 as well.
c5d0adf [Michael Armbrust] Update SparkSQL in branch-1.0 to match master.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Allow Spark to build against particular Pivotal HD distributions. For example to build Spark against Pivotal HD 2.0.1 one can run:
```
mvn -Pyarn -Phadoop-2.2 -Dhadoop.version=2.2.0-gphd-3.0.1.0 -DskipTests clean package
```
Author: tzolov <christian.tzolov@gmail.com>
Closes #942 from tzolov/master and squashes the following commits:
bc3e05a [tzolov] Add support for Pivotal HD in the Maven build and SBT build: [SPARK-1992]
(cherry picked from commit b1f285359a12345d94ae59f979d9a75482e732ce)
Signed-off-by: Matei Zaharia <matei@databricks.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
Author: witgo <witgo@qq.com>
Closes #786 from witgo/maven_plugin and squashes the following commits:
5de86a2 [witgo] Merge branch 'master' of https://github.com/apache/spark into maven_plugin
c35ef73 [witgo] Improve maven plugin configuration
Conflicts:
pom.xml
|
| |
|
|
|
|
|
|
|
|
|
| |
Author: Yin Huai <huai@cse.ohio-state.edu>
Closes #889 from yhuai/SPARK-1935 and squashes the following commits:
7d50ef1 [Yin Huai] Explicitly add commons-codec 1.5 as a dependency.
(cherry picked from commit 60b89fe6b09ff896a30d74204876da883e307de7)
Signed-off-by: Patrick Wendell <pwendell@gmail.com>
|
| | |
|
| | |
|
| |
|
|
| |
This reverts commit 2f1dc868e5714882cf40d2633fb66772baf34789.
|
| |
|
|
| |
This reverts commit 832dc594e7666f1d402334f8015ce29917d9c888.
|
| | |
|
| | |
|
| |
|
|
| |
This reverts commit d807023479ce10aec28ef3c1ab646ddefc2e663c.
|
| |
|
|
| |
This reverts commit 67dd53d2556f03ce292e6889128cf441f1aa48f8.
|
| | |
|
| | |
|
| |
|
|
| |
This reverts commit 920f947eb5a22a679c0c3186cf69ee75f6041c75.
|
| |
|
|
| |
This reverts commit f8e611955096c5c1c7db5764b9d2851b1d295f0d.
|
| | |
|
| | |
|
| |
|
|
| |
This reverts commit 80eea0f111c06260ffaa780d2f3f7facd09c17bc.
|
| |
|
|
| |
This reverts commit e5436b8c1a79ce108f3af402455ac5f6dc5d1eb3.
|
| | |
|
| | |
|
| |
|
|
| |
This reverts commit 9212b3e5bb5545ccfce242da8d89108e6fb1c464.
|
| |
|
|
| |
This reverts commit c4746aa6fe4aaf383e69e34353114d36d1eb9ba6.
|
| |
|
|
|
|
|
|
|
|
|
| |
Author: Patrick Wendell <pwendell@gmail.com>
Closes #803 from pwendell/mapr-support and squashes the following commits:
8df60e4 [Patrick Wendell] SPARK-1862: Support for MapR in the Maven build.
(cherry picked from commit 17702e280c4b0b030870962fcb3d50c3085ae862)
Signed-off-by: Patrick Wendell <pwendell@gmail.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
...s
Author: Sandy Ryza <sandy@cloudera.com>
Closes #795 from sryza/sandy-spark-1851 and squashes the following commits:
79c8227 [Sandy Ryza] SPARK-1851. Upgrade Avro dependency to 1.7.6 so Spark can read Avro files
(cherry picked from commit 08e7606a964e3d1ac1d565f33651ff0035c75044)
Signed-off-by: Patrick Wendell <pwendell@gmail.com>
|
| | |
|
| | |
|
| |
|
|
| |
This reverts commit 54133abdce0246f6643a1112a5204afb2c4caa82.
|
| |
|
|
| |
This reverts commit e480bcfbd269ae1d7a6a92cfb50466cf192fe1fb.
|
| | |
|
| | |
|
| |
|
|
| |
This reverts commit 18f062303303824139998e8fc8f4158217b0dbc3.
|
| |
|
|
| |
This reverts commit d08e9604fc9958b7c768e91715c8152db2ed6fd0.
|
| |
|
|
|
|
|
|
| |
Author: Marcelo Vanzin <vanzin@cloudera.com>
Closes #763 from vanzin/netty-dep-hell and squashes the following commits:
dfb6ce2 [Marcelo Vanzin] Fix dep exclusion: avro-ipc, not avro, depends on netty.
|
| | |
|
| | |
|
| |
|
|
| |
This reverts commit 3d0a44833ab50360bf9feccc861cb5e8c44a4866.
|
| |
|
|
| |
This reverts commit 9772d85c6f3893d42044f4bab0e16f8b6287613a.
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Following on a few more items from SPARK-1802 --
The first commit touches up a few similar problems remaining with the YARN profile. I think this is worth cherry-picking.
The second commit is more of the same for hadoop-client, although the fix is a little more complex. It may or may not be worth bothering with.
Author: Sean Owen <sowen@cloudera.com>
Closes #746 from srowen/SPARK-1802.2 and squashes the following commits:
52aeb41 [Sean Owen] Add more commons-logging, servlet excludes to avoid conflicts in assembly when building for YARN
(cherry picked from commit 4b31f4ec7efab8eabf956284a99bfd96a58b79f7)
Signed-off-by: Patrick Wendell <pwendell@gmail.com>
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This initial commit resolves the conflicts in the Hive profiles as noted in https://issues.apache.org/jira/browse/SPARK-1802 .
Most of the fix was to note that Hive drags in Avro, and so if the hive module depends on Spark's version of the `avro-*` dependencies, it will pull in our exclusions as needed too. But I found we need to copy some exclusions between the two Avro dependencies to get this right. And then had to squash some commons-logging intrusions.
This turned up another annoying find, that `hive-exec` is basically an "assembly" artifact that _also_ packages all of its transitive dependencies. This means the final assembly shows lots of collisions between itself and its dependencies, and even other project dependencies. I have a TODO to examine whether that is going to be a deal-breaker or not.
In the meantime I'm going to tack on a second commit to this PR that will also fix some similar, last collisions in the YARN profile.
Author: Sean Owen <sowen@cloudera.com>
Closes #744 from srowen/SPARK-1802 and squashes the following commits:
a856604 [Sean Owen] Resolve JAR version conflicts specific to Hive profile
(cherry picked from commit 8586bf564fe010dfc19ef26874472a6f85e355fb)
Signed-off-by: Patrick Wendell <pwendell@gmail.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Three issues related to temp files that tests generate – these should be touched up for hygiene but are not urgent.
Modules have a log4j.properties which directs the unit-test.log output file to a directory like `[module]/target/unit-test.log`. But this ends up creating `[module]/[module]/target/unit-test.log` instead of former.
The `work/` directory is not deleted by "mvn clean", in the parent and in modules. Neither is the `checkpoint/` directory created under the various external modules.
Many tests create a temp directory, which is not usually deleted. This can be largely resolved by calling `deleteOnExit()` at creation and trying to call `Utils.deleteRecursively` consistently to clean up, sometimes in an `@After` method.
_If anyone seconds the motion, I can create a more significant change that introduces a new test trait along the lines of `LocalSparkContext`, which provides management of temp directories for subclasses to take advantage of._
Author: Sean Owen <sowen@cloudera.com>
Closes #732 from srowen/SPARK-1798 and squashes the following commits:
5af578e [Sean Owen] Try to consistently delete test temp dirs and files, and set deleteOnExit() for each
b21b356 [Sean Owen] Remove work/ and checkpoint/ dirs with mvn clean
bdd0f41 [Sean Owen] Remove duplicate module dir in log4j.properties output path for tests
(cherry picked from commit 7120a2979d0a9f0f54a88b2416be7ca10e74f409)
Signed-off-by: Patrick Wendell <pwendell@gmail.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Enabled Mesos (0.18.1) dependency with shaded protobuf
Why is this needed?
Avoids any protobuf version collision between Mesos and any other
dependency in Spark e.g. Hadoop HDFS 2.2+ or 1.0.4.
Ticket: https://issues.apache.org/jira/browse/SPARK-1806
* Should close https://issues.apache.org/jira/browse/SPARK-1433
Author berngp
Author: Bernardo Gomez Palacio <bernardo.gomezpalacio@gmail.com>
Closes #741 from berngp/feature/SPARK-1806 and squashes the following commits:
5d70646 [Bernardo Gomez Palacio] SPARK-1806: Upgrade Mesos dependency to 0.18.1
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
failure
TL;DR is there is a bit of JAR hell trouble with Netty, that can be mostly resolved and will resolve a test failure.
I hit the error described at http://apache-spark-user-list.1001560.n3.nabble.com/SparkContext-startup-time-out-td1753.html while running FlumeStreamingSuite, and have for a short while (is it just me?)
velvia notes:
"I have found a workaround. If you add akka 2.2.4 to your dependencies, then everything works, probably because akka 2.2.4 brings in newer version of Jetty."
There are at least 3 versions of Netty in play in the build:
- the new Flume 1.4.0 dependency brings in io.netty:netty:3.4.0.Final, and that is the immediate problem
- the custom version of akka 2.2.3 depends on io.netty:netty:3.6.6.
- but, Spark Core directly uses io.netty:netty-all:4.0.17.Final
The POMs try to exclude other versions of netty, but are excluding org.jboss.netty:netty, when in fact older versions of io.netty:netty (not netty-all) are also an issue.
The org.jboss.netty:netty excludes are largely unnecessary. I replaced many of them with io.netty:netty exclusions until everything agreed on io.netty:netty-all:4.0.17.Final.
But this didn't work, since Akka 2.2.3 doesn't work with Netty 4.x. Down-grading to 3.6.6.Final across the board made some Spark code not compile.
If the build *keeps* io.netty:netty:3.6.6.Final as well, everything seems to work. Part of the reason seems to be that Netty 3.x used the old `org.jboss.netty` packages. This is less than ideal, but is no worse than the current situation.
So this PR resolves the issue and improves the JAR hell, even if it leaves the existing theoretical Netty 3-vs-4 conflict:
- Remove org.jboss.netty excludes where possible, for clarity; they're not needed except with Hadoop artifacts
- Add io.netty:netty excludes where needed -- except, let akka keep its io.netty:netty
- Change a bit of test code that actually depended on Netty 3.x, to use 4.x equivalent
- Update SBT build accordingly
A better change would be to update Akka far enough such that it agrees on Netty 4.x, but I don't know if that's feasible.
Author: Sean Owen <sowen@cloudera.com>
Closes #723 from srowen/SPARK-1789 and squashes the following commits:
43661b7 [Sean Owen] Update and add Netty excludes to prevent some JAR conflicts that cause test issues
(cherry picked from commit 2b7bd29eb6ee5baf739eec143044ecfc296b9b1f)
Signed-off-by: Patrick Wendell <pwendell@gmail.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
I think we are hitting this issue in some perf tests: https://github.com/Parquet/parquet-mr/commit/6aed5288fd4a1398063a5a219b2ae4a9f71b02cf
Credit to @aarondav !
Author: Michael Armbrust <michael@databricks.com>
Closes #684 from marmbrus/upgradeParquet and squashes the following commits:
e10a619 [Michael Armbrust] Upgrade parquet library.
(cherry picked from commit 4d6055329846f5e09472e5f844127a5ab5880e15)
Signed-off-by: Patrick Wendell <pwendell@gmail.com>
|
