aboutsummaryrefslogtreecommitdiff
path: root/external
Commit message (Collapse)AuthorAgeFilesLines
* Preparing Spark release v1.3.1-rc3v1.3.1Patrick Wendell2015-04-117-7/+7
|
* Revert "Preparing Spark release v1.3.1-rc2"Patrick Wendell2015-04-107-7/+7
| | | | This reverts commit 7c4473aa5a7f5de0323394aaedeefbf9738e8eb5.
* Revert "Preparing development version 1.3.2-SNAPSHOT"Patrick Wendell2015-04-107-7/+7
| | | | This reverts commit cdef7d080aa3f473f5ea06ba816c01b41a0239eb.
* Preparing development version 1.3.2-SNAPSHOTPatrick Wendell2015-04-087-7/+7
|
* Preparing Spark release v1.3.1-rc2Patrick Wendell2015-04-087-7/+7
|
* Revert "Preparing Spark release v1.3.1-rc1"Patrick Wendell2015-04-077-7/+7
| | | | This reverts commit 0dcb5d9f31b713ed90bcec63ebc4e530cbb69851.
* Revert "Preparing development version 1.3.2-SNAPSHOT"Patrick Wendell2015-04-077-7/+7
| | | | This reverts commit 728c1f927822eb6b12f04dc47109feb6fbe02ec2.
* Preparing development version 1.3.2-SNAPSHOTPatrick Wendell2015-04-047-7/+7
|
* Preparing Spark release v1.3.1-rc1Patrick Wendell2015-04-047-7/+7
|
* [SPARK-5559] [Streaming] [Test] Remove oppotunity we met flakiness when ↵Kousuke Saruta2015-03-242-3/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | running FlumeStreamSuite When we run FlumeStreamSuite on Jenkins, sometimes we get error like as follows. sbt.ForkMain$ForkError: The code passed to eventually never returned normally. Attempted 52 times over 10.094849836 seconds. Last failure message: Error connecting to localhost/127.0.0.1:23456. at org.scalatest.concurrent.Eventually$class.tryTryAgain$1(Eventually.scala:420) at org.scalatest.concurrent.Eventually$class.eventually(Eventually.scala:438) at org.scalatest.concurrent.Eventually$.eventually(Eventually.scala:478) at org.scalatest.concurrent.Eventually$class.eventually(Eventually.scala:307) at org.scalatest.concurrent.Eventually$.eventually(Eventually.scala:478) at org.apache.spark.streaming.flume.FlumeStreamSuite.writeAndVerify(FlumeStreamSuite.scala:116) at org.apache.spark.streaming.flume.FlumeStreamSuite.org$apache$spark$streaming$flume$FlumeStreamSuite$$testFlumeStream(FlumeStreamSuite.scala:74) at org.apache.spark.streaming.flume.FlumeStreamSuite$$anonfun$3.apply$mcV$sp(FlumeStreamSuite.scala:66) at org.apache.spark.streaming.flume.FlumeStreamSuite$$anonfun$3.apply(FlumeStreamSuite.scala:66) at org.apache.spark.streaming.flume.FlumeStreamSuite$$anonfun$3.apply(FlumeStreamSuite.scala:66) at org.scalatest.Transformer$$anonfun$apply$1.apply$mcV$sp(Transformer.scala:22) at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85) at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104) at org.scalatest.Transformer.apply(Transformer.scala:22) at org.scalatest.Transformer.apply(Transformer.scala:20) at org.scalatest.FunSuiteLike$$anon$1.apply(FunSuiteLike.scala:166) at org.scalatest.Suite$class.withFixture(Suite.scala:1122) at org.scalatest.FunSuite.withFixture(FunSuite.scala:1555) at org.scalatest.FunSuiteLike$class.invokeWithFixture$1(FunSuiteLike.scala:163) at org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:175) at org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:175) at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306) at org.scalatest.FunSuiteLike$class.runTest(FunSuiteLike.scala:175) This error is caused by check-then-act logic when it find free-port . /** Find a free port */ private def findFreePort(): Int = { Utils.startServiceOnPort(23456, (trialPort: Int) => { val socket = new ServerSocket(trialPort) socket.close() (null, trialPort) }, conf)._2 } Removing the check-then-act is not easy but we can reduce the chance of having the error by choosing random value for initial port instead of 23456. Author: Kousuke Saruta <sarutak@oss.nttdata.co.jp> Closes #4337 from sarutak/SPARK-5559 and squashes the following commits: 16f109f [Kousuke Saruta] Added `require` to Utils#startServiceOnPort c39d8b6 [Kousuke Saruta] Merge branch 'SPARK-5559' of github.com:sarutak/spark into SPARK-5559 1610ba2 [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into SPARK-5559 33357e3 [Kousuke Saruta] Changed "findFreePort" method in MQTTStreamSuite and FlumeStreamSuite so that it can choose valid random port a9029fe [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into SPARK-5559 9489ef9 [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into SPARK-5559 8212e42 [Kousuke Saruta] Modified default port used in FlumeStreamSuite from 23456 to random value (cherry picked from commit 85cf0636825d1997d64d0bdc04618f29b7222da1) Signed-off-by: Sean Owen <sowen@cloudera.com>
* Preparing development version 1.3.1-SNAPSHOTPatrick Wendell2015-03-057-7/+7
|
* Preparing Spark release v1.3.0-rc3v1.3.0Patrick Wendell2015-03-057-7/+7
|
* Revert "Preparing Spark release v1.3.0-rc3"Patrick Wendell2015-03-057-7/+7
| | | | This reverts commit 6fb4af2fbeb3d1b888191a2fa1042c80e3ef2d60.
* Revert "Preparing development version 1.3.1-SNAPSHOT"Patrick Wendell2015-03-057-7/+7
| | | | This reverts commit 5097f869efbdb75d3b87bcbd8e621e7c12356942.
* Preparing development version 1.3.1-SNAPSHOTPatrick Wendell2015-03-057-7/+7
|
* Preparing Spark release v1.3.0-rc3Patrick Wendell2015-03-057-7/+7
|
* SPARK-6182 [BUILD] spark-parent pom needs to be published for both 2.10 and 2.11Sean Owen2015-03-057-7/+7
| | | | | | | | | | | | | Option 1 of 2: Convert spark-parent module name to spark-parent_2.10 / spark-parent_2.11 Author: Sean Owen <sowen@cloudera.com> Closes #4912 from srowen/SPARK-6182.1 and squashes the following commits: eff60de [Sean Owen] Convert spark-parent module name to spark-parent_2.10 / spark-parent_2.11 (cherry picked from commit c9cfba0cebe3eb546e3e96f3e5b9b89a74c5b7de) Signed-off-by: Patrick Wendell <patrick@databricks.com>
* Revert "Preparing Spark release v1.3.0-rc3"Patrick Wendell2015-03-047-7/+7
| | | | This reverts commit 430a879699d2d41bc65f5c00b1f239d15fd5e549.
* Revert "Preparing development version 1.3.1-SNAPSHOT"Patrick Wendell2015-03-047-7/+7
| | | | This reverts commit 0ecab40e4391d0674ac86595ec09af3b9a4ac50d.
* Preparing development version 1.3.1-SNAPSHOTPatrick Wendell2015-03-057-7/+7
|
* Preparing Spark release v1.3.0-rc3Patrick Wendell2015-03-057-7/+7
|
* Revert "Preparing Spark release v1.3.0-rc2"Patrick Wendell2015-03-047-7/+7
| | | | This reverts commit 3af26870e5163438868c4eb2df88380a533bb232.
* Revert "Preparing development version 1.3.1-SNAPSHOT"Patrick Wendell2015-03-047-7/+7
| | | | This reverts commit 05d5a29eb3193aeb57d177bafe39eb75edce72a1.
* Preparing development version 1.3.1-SNAPSHOTPatrick Wendell2015-03-037-7/+7
|
* Preparing Spark release v1.3.0-rc2Patrick Wendell2015-03-037-7/+7
|
* Revert "Preparing Spark release v1.3.0-rc1"Patrick Wendell2015-03-037-7/+7
| | | | This reverts commit f97b0d4a6b26504916816d7aefcf3132cd1da6c2.
* Revert "Preparing development version 1.3.1-SNAPSHOT"Patrick Wendell2015-03-037-7/+7
| | | | This reverts commit 2ab0ba04f66683be25cbe0e83cecf2bdcb0f13ba.
* [Streaming][Minor] Remove useless type signature of Java Kafka direct stream APISaisai Shao2015-02-271-1/+1
| | | | | | | | | | | | | cc tdas . Author: Saisai Shao <saisai.shao@intel.com> Closes #4817 from jerryshao/signature-minor-fix and squashes the following commits: eebfaac [Saisai Shao] Remove useless type parameter (cherry picked from commit 5f7f3b938e1776168be866fc9ee87dc7494696cc) Signed-off-by: Tathagata Das <tathagata.das1565@gmail.com>
* [SPARK-6027][SPARK-5546] Fixed --jar and --packages not working for ↵Tathagata Das2015-02-261-1/+28
| | | | | | | | | | | | | | | | | | | | | KafkaUtils and improved error message The problem with SPARK-6027 in short is that JARs like the kafka-assembly.jar does not work in python as the added JAR is not visible in the classloader used by Py4J. Py4J uses Class.forName(), which does not uses the systemclassloader, but the JARs are only visible in the Thread's contextclassloader. So this back uses the context class loader to create the KafkaUtils dstream object. This works for both cases where the Kafka libraries are added with --jars spark-streaming-kafka-assembly.jar or with --packages spark-streaming-kafka Also improves the error message. davies Author: Tathagata Das <tathagata.das1565@gmail.com> Closes #4779 from tdas/kafka-python-fix and squashes the following commits: fb16b04 [Tathagata Das] Removed import c1fdf35 [Tathagata Das] Fixed long line and improved documentation 7b88be8 [Tathagata Das] Fixed --jar not working for KafkaUtils and improved error message (cherry picked from commit aa63f633d39efa8c29095295f161eaad5495071d) Signed-off-by: Andrew Or <andrew@databricks.com>
* [SPARK-5993][Streaming][Build] Fix assembly jar location of kafka-assemblyTathagata Das2015-02-241-4/+0
| | | | | | | | | | | | | | | | | | Published Kafka-assembly JAR was empty in 1.3.0-RC1 This is because the maven build generated two Jars- 1. an empty JAR file (since kafka-assembly has no code of its own) 2. a assembly JAR file containing everything in a different location as 1 The maven publishing plugin uploaded 1 and not 2. Instead if 2 is not configure to generate in a different location, there is only 1 jar containing everything, which gets published. Author: Tathagata Das <tathagata.das1565@gmail.com> Closes #4753 from tdas/SPARK-5993 and squashes the following commits: c390db8 [Tathagata Das] Fix assembly jar location of kafka-assembly (cherry picked from commit 922b43b3cc1cca04e0313bf9e31c5f944ac06d1f) Signed-off-by: Tathagata Das <tathagata.das1565@gmail.com>
* SPARK-4682 [CORE] Consolidate various 'Clock' classesSean Owen2015-02-195-8/+7
| | | | | | | | | | | | | | | | | | Another one from JoshRosen 's wish list. The first commit is much smaller and removes 2 of the 4 Clock classes. The second is much larger, necessary for consolidating the streaming one. I put together implementations in the way that seemed simplest. Almost all the change is standardizing class and method names. Author: Sean Owen <sowen@cloudera.com> Closes #4514 from srowen/SPARK-4682 and squashes the following commits: 5ed3a03 [Sean Owen] Javadoc Clock classes; make ManualClock private[spark] 169dd13 [Sean Owen] Add support for legacy org.apache.spark.streaming clock class names 277785a [Sean Owen] Reduce the net change in this patch by reversing some unnecessary syntax changes along the way b5e53df [Sean Owen] FakeClock -> ManualClock; getTime() -> getTimeMillis() 160863a [Sean Owen] Consolidate Streaming Clock class into common util Clock 7c956b2 [Sean Owen] Consolidate Clocks except for Streaming Clock (cherry picked from commit 34b7c35380c88569a1396fb4ed991a0bed4288e7) Signed-off-by: Andrew Or <andrew@databricks.com>
* [SPARK-5731][Streaming][Test] Fix incorrect test in DirectKafkaStreamSuiteTathagata Das2015-02-171-12/+16
| | | | | | | | | | | | | | | | | | The test was incorrect. Instead of counting the number of records, it counted the number of partitions of RDD generated by DStream. Which is not its intention. I will be testing this patch multiple times to understand its flakiness. PS: This was caused by my refactoring in https://github.com/apache/spark/pull/4384/ koeninger check it out. Author: Tathagata Das <tathagata.das1565@gmail.com> Closes #4597 from tdas/kafka-flaky-test and squashes the following commits: d236235 [Tathagata Das] Unignored last test. e9a1820 [Tathagata Das] fix test (cherry picked from commit 3912d332464dcd124c60b734724c34d9742466a4) Signed-off-by: Tathagata Das <tathagata.das1565@gmail.com>
* Preparing development version 1.3.1-SNAPSHOTPatrick Wendell2015-02-187-7/+7
|
* Preparing Spark release v1.3.0-rc1Patrick Wendell2015-02-187-7/+7
|
* Revert "Preparing Spark release v1.3.0-snapshot1"Patrick Wendell2015-02-177-7/+7
| | | | This reverts commit d97bfc6f28ec4b7acfb36410c7c167d8d3c145ec.
* Revert "Preparing development version 1.3.1-SNAPSHOT"Patrick Wendell2015-02-177-7/+7
| | | | This reverts commit e57c81b8c1a6581c2588973eaf30d3c7ae90ed0c.
* SPARK-5850: Remove experimental label for Scala 2.11 and FlumePollingStreamPatrick Wendell2015-02-161-8/+0
| | | | | | | | Author: Patrick Wendell <patrick@databricks.com> Closes #4638 from pwendell/SPARK-5850 and squashes the following commits: 386126f [Patrick Wendell] SPARK-5850: Remove experimental label for Scala 2.11 and FlumePollingStream.
* [HOTFIX] Ignore DirectKafkaStreamSuite.Reynold Xin2015-02-151-4/+4
|
* SPARK-5728 [STREAMING] MQTTStreamSuite leaves behind ActiveMQ database filesSean Owen2015-02-111-0/+1
| | | | | | | | | | | | | Use temp dir for ActiveMQ database Author: Sean Owen <sowen@cloudera.com> Closes #4517 from srowen/SPARK-5728 and squashes the following commits: 1d3aeb8 [Sean Owen] Use temp dir for ActiveMQ database (cherry picked from commit da89720bf4023392436e75b6ed5e10ed8588a132) Signed-off-by: Sean Owen <sowen@cloudera.com>
* [SPARK-4964] [Streaming] refactor createRDD to take leaders via map instead ↵cody koeninger2015-02-114-66/+287
| | | | | | | | | | | | | | | | | of array Author: cody koeninger <cody@koeninger.org> Closes #4511 from koeninger/kafkaRdd-leader-to-broker and squashes the following commits: f7151d4 [cody koeninger] [SPARK-4964] test refactoring 6f8680b [cody koeninger] [SPARK-4964] add test of the scala api for KafkaUtils.createRDD f81e016 [cody koeninger] [SPARK-4964] leave KafkaStreamSuite host and port as private 5173f3f [cody koeninger] [SPARK-4964] test the Java variations of createRDD e9cece4 [cody koeninger] [SPARK-4964] pass leaders as a map to ensure 1 leader per TopicPartition (cherry picked from commit 658687b25491047f30ee8558733d11e5a0572070) Signed-off-by: Tathagata Das <tathagata.das1565@gmail.com>
* Preparing development version 1.3.1-SNAPSHOTPatrick Wendell2015-02-117-7/+7
|
* Preparing Spark release v1.3.0-snapshot1Patrick Wendell2015-02-117-7/+7
|
* Revert "Preparing Spark release v1.3.0-snapshot1"Patrick Wendell2015-02-107-7/+7
| | | | This reverts commit 53068f56f40bf03b7fc52e5980fb7e205903fc8b.
* Revert "Preparing development version 1.3.1-SNAPSHOT"Patrick Wendell2015-02-107-7/+7
| | | | This reverts commit ba12b793f1f4f432e71439e2a7ebacce74d9c472.
* Preparing development version 1.3.1-SNAPSHOTPatrick Wendell2015-02-117-7/+7
|
* Preparing Spark release v1.3.0-snapshot1Patrick Wendell2015-02-117-7/+7
|
* Revert "Preparing Spark release v1.3.0-snapshot1"Patrick Wendell2015-02-107-7/+7
| | | | This reverts commit c2e4001030cfb881ff33d448fc0aeaf4f05dad0f.
* Revert "Preparing development version 1.3.1-SNAPSHOT"Patrick Wendell2015-02-107-7/+7
| | | | This reverts commit db80d0fe21daa3202ff217cbefb999ce77c5aa9e.
* Preparing development version 1.3.1-SNAPSHOTPatrick Wendell2015-02-117-7/+7
|
* Preparing Spark release v1.3.0-snapshot1Patrick Wendell2015-02-117-7/+7
|
generated by cgit v1.2.3 (git 2.39.1) at 2024-09-20 00:44:43 +0000