aboutsummaryrefslogtreecommitdiff
path: root/external
Commit message (Collapse)AuthorAgeFilesLines
* Preparing Spark release v1.2.2-rc1v1.2.2Patrick Wendell2015-04-056-6/+6
|
* [SPARK-5559] [Streaming] [Test] Remove oppotunity we met flakiness when ↵Kousuke Saruta2015-03-242-3/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | running FlumeStreamSuite When we run FlumeStreamSuite on Jenkins, sometimes we get error like as follows. sbt.ForkMain$ForkError: The code passed to eventually never returned normally. Attempted 52 times over 10.094849836 seconds. Last failure message: Error connecting to localhost/127.0.0.1:23456. at org.scalatest.concurrent.Eventually$class.tryTryAgain$1(Eventually.scala:420) at org.scalatest.concurrent.Eventually$class.eventually(Eventually.scala:438) at org.scalatest.concurrent.Eventually$.eventually(Eventually.scala:478) at org.scalatest.concurrent.Eventually$class.eventually(Eventually.scala:307) at org.scalatest.concurrent.Eventually$.eventually(Eventually.scala:478) at org.apache.spark.streaming.flume.FlumeStreamSuite.writeAndVerify(FlumeStreamSuite.scala:116) at org.apache.spark.streaming.flume.FlumeStreamSuite.org$apache$spark$streaming$flume$FlumeStreamSuite$$testFlumeStream(FlumeStreamSuite.scala:74) at org.apache.spark.streaming.flume.FlumeStreamSuite$$anonfun$3.apply$mcV$sp(FlumeStreamSuite.scala:66) at org.apache.spark.streaming.flume.FlumeStreamSuite$$anonfun$3.apply(FlumeStreamSuite.scala:66) at org.apache.spark.streaming.flume.FlumeStreamSuite$$anonfun$3.apply(FlumeStreamSuite.scala:66) at org.scalatest.Transformer$$anonfun$apply$1.apply$mcV$sp(Transformer.scala:22) at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85) at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104) at org.scalatest.Transformer.apply(Transformer.scala:22) at org.scalatest.Transformer.apply(Transformer.scala:20) at org.scalatest.FunSuiteLike$$anon$1.apply(FunSuiteLike.scala:166) at org.scalatest.Suite$class.withFixture(Suite.scala:1122) at org.scalatest.FunSuite.withFixture(FunSuite.scala:1555) at org.scalatest.FunSuiteLike$class.invokeWithFixture$1(FunSuiteLike.scala:163) at org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:175) at org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:175) at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306) at org.scalatest.FunSuiteLike$class.runTest(FunSuiteLike.scala:175) This error is caused by check-then-act logic when it find free-port . /** Find a free port */ private def findFreePort(): Int = { Utils.startServiceOnPort(23456, (trialPort: Int) => { val socket = new ServerSocket(trialPort) socket.close() (null, trialPort) }, conf)._2 } Removing the check-then-act is not easy but we can reduce the chance of having the error by choosing random value for initial port instead of 23456. Author: Kousuke Saruta <sarutak@oss.nttdata.co.jp> Closes #4337 from sarutak/SPARK-5559 and squashes the following commits: 16f109f [Kousuke Saruta] Added `require` to Utils#startServiceOnPort c39d8b6 [Kousuke Saruta] Merge branch 'SPARK-5559' of github.com:sarutak/spark into SPARK-5559 1610ba2 [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into SPARK-5559 33357e3 [Kousuke Saruta] Changed "findFreePort" method in MQTTStreamSuite and FlumeStreamSuite so that it can choose valid random port a9029fe [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into SPARK-5559 9489ef9 [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into SPARK-5559 8212e42 [Kousuke Saruta] Modified default port used in FlumeStreamSuite from 23456 to random value (cherry picked from commit 85cf0636825d1997d64d0bdc04618f29b7222da1) Signed-off-by: Sean Owen <sowen@cloudera.com>
* SPARK-5728 [STREAMING] MQTTStreamSuite leaves behind ActiveMQ database filesSean Owen2015-02-131-0/+1
| | | | | | | | | | | | | Use temp dir for ActiveMQ database Author: Sean Owen <sowen@cloudera.com> Closes #4517 from srowen/SPARK-5728 and squashes the following commits: 1d3aeb8 [Sean Owen] Use temp dir for ActiveMQ database (cherry picked from commit da89720bf4023392436e75b6ed5e10ed8588a132) Signed-off-by: Sean Owen <sowen@cloudera.com>
* [SPARK-4631][streaming][FIX] Wait for a receiver to start before publishing ↵Iulian Dragos2015-02-132-18/+54
| | | | | | | | | | | | | | | | | | | | | | | | | | test data. This fixes two sources of non-deterministic failures in this test: - wait for a receiver to be up before pushing data through MQTT - gracefully handle the case where the MQTT client is overloaded. There’s a hard-coded limit of 10 in-flight messages, and this test may hit it. Instead of crashing, we retry sending the message. Both of these are needed to make the test pass reliably on my machine. Author: Iulian Dragos <jaguarul@gmail.com> Closes #4270 from dragos/issue/fix-flaky-test-SPARK-4631 and squashes the following commits: f66c482 [Iulian Dragos] [SPARK-4631][streaming] Wait for a receiver to start before publishing test data. d408a8e [Iulian Dragos] Install callback before connecting to MQTT broker. (cherry picked from commit e908322cd5991e6cbdaaafb8cd494759dac01225) Signed-off-by: Sean Owen <sowen@cloudera.com> # Conflicts: # external/mqtt/src/test/scala/org/apache/spark/streaming/mqtt/MQTTStreamSuite.scala
* [SPARK-4905][STREAMING] FlumeStreamSuite fix.Hari Shreedharan2015-02-091-4/+3
| | | | | | | | | | | | | | | Using String constructor instead of CharsetDecoder to see if it fixes the issue of empty strings in Flume test output. Author: Hari Shreedharan <hshreedharan@apache.org> Closes #4371 from harishreedharan/Flume-stream-attempted-fix and squashes the following commits: 550d363 [Hari Shreedharan] Fix imports. 8695950 [Hari Shreedharan] Use Charsets.UTF_8 instead of "UTF-8" in String constructors. af3ba14 [Hari Shreedharan] [SPARK-4905][STREAMING] FlumeStreamSuite fix. (cherry picked from commit 0765af9b21e9204c410c7a849c7201bc3eda8cc3) Signed-off-by: Josh Rosen <joshrosen@databricks.com>
* [SPARK-5153][Streaming][Test] Increased timeout to deal with flaky ↵Tathagata Das2015-02-031-1/+1
| | | | | | | | | | | | | | | KafkaStreamSuite Timeout increased to allow overloaded Jenkins to cope with delay in topic creation. Author: Tathagata Das <tathagata.das1565@gmail.com> Closes #4342 from tdas/SPARK-5153 and squashes the following commits: dc42762 [Tathagata Das] Increased timeout to deal with delays in overloaded Jenkins. (cherry picked from commit 681f9df47ff40f7b0d9175d835e9758d33a13a06) Signed-off-by: Tathagata Das <tathagata.das1565@gmail.com>
* Preparing development version 1.2.2-SNAPSHOTPatrick Wendell2015-02-036-6/+6
|
* Preparing Spark release v1.2.1-rc3v1.2.1Patrick Wendell2015-02-036-6/+6
|
* Revert "Preparing Spark release v1.2.1-rc2"Patrick Wendell2015-02-026-6/+6
| | | | This reverts commit b77f87673d1f9f03d4c83cf583158227c551359b.
* Revert "Preparing development version 1.2.2-SNAPSHOT"Patrick Wendell2015-02-026-6/+6
| | | | This reverts commit 0a16abadc59082b7d3a24d7f3625236658632813.
* Preparing development version 1.2.2-SNAPSHOTPatrick Wendell2015-01-286-6/+6
|
* Preparing Spark release v1.2.1-rc2Patrick Wendell2015-01-286-6/+6
|
* Revert "Preparing Spark release v1.2.1-rc1"Patrick Wendell2015-01-276-6/+6
| | | | This reverts commit 3e2d7d310b76c293b9ac787f204e6880f508f6ec.
* Revert "Preparing development version 1.2.2-SNAPSHOT"Patrick Wendell2015-01-276-6/+6
| | | | This reverts commit f53a4319ba5f0843c077e64ae5a41e2fac835a5b.
* Preparing development version 1.2.2-SNAPSHOTPatrick Wendell2015-01-276-6/+6
|
* Preparing Spark release v1.2.1-rc1Patrick Wendell2015-01-276-6/+6
|
* Revert "Preparing Spark release v1.2.1-rc1"Patrick Wendell2015-01-266-6/+6
| | | | This reverts commit e87eb2b42f137c22194cfbca2abf06fecdf943da.
* Revert "Preparing development version 1.2.2-SNAPSHOT"Patrick Wendell2015-01-266-6/+6
| | | | This reverts commit adfed7086f10fa8db4eeac7996c84cf98f625e9a.
* Preparing development version 1.2.2-SNAPSHOTUbuntu2015-01-276-6/+6
|
* Preparing Spark release v1.2.1-rc1Ubuntu2015-01-276-6/+6
|
* [SPARK-5006][Deploy]spark.port.maxRetries doesn't workWangTaoTheTonic2015-01-212-3/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | https://issues.apache.org/jira/browse/SPARK-5006 I think the issue is produced in https://github.com/apache/spark/pull/1777. Not digging mesos's backend yet. Maybe should add same logic either. Author: WangTaoTheTonic <barneystinson@aliyun.com> Author: WangTao <barneystinson@aliyun.com> Closes #3841 from WangTaoTheTonic/SPARK-5006 and squashes the following commits: 8cdf96d [WangTao] indent thing 2d86d65 [WangTaoTheTonic] fix line length 7cdfd98 [WangTaoTheTonic] fit for new HttpServer constructor 61a370d [WangTaoTheTonic] some minor fixes bc6e1ec [WangTaoTheTonic] rebase 67bcb46 [WangTaoTheTonic] put conf at 3rd position, modify suite class, add comments f450cd1 [WangTaoTheTonic] startServiceOnPort will use a SparkConf arg 29b751b [WangTaoTheTonic] rebase as ExecutorRunnableUtil changed to ExecutorRunnable 396c226 [WangTaoTheTonic] make the grammar more like scala 191face [WangTaoTheTonic] invalid value name 62ec336 [WangTaoTheTonic] spark.port.maxRetries doesn't work Conflicts: external/mqtt/src/test/scala/org/apache/spark/streaming/mqtt/MQTTStreamSuite.scala
* [SPARK-4631] unit test for MQTTbilna2015-01-042-15/+101
| | | | | | | | | | | | | | | | | | | | | | | | | | | Please review the unit test for MQTT Author: bilna <bilnap@am.amrita.edu> Author: Bilna P <bilna.p@gmail.com> Closes #3844 from Bilna/master and squashes the following commits: acea3a3 [bilna] Adding dependency with scope test 28681fa [bilna] Merge remote-tracking branch 'upstream/master' fac3904 [bilna] Correction in Indentation and coding style ed9db4c [bilna] Merge remote-tracking branch 'upstream/master' 4b34ee7 [Bilna P] Update MQTTStreamSuite.scala 04503cf [bilna] Added embedded broker service for mqtt test 89d804e [bilna] Merge remote-tracking branch 'upstream/master' fc8eb28 [bilna] Merge remote-tracking branch 'upstream/master' 4b58094 [Bilna P] Update MQTTStreamSuite.scala b1ac4ad [bilna] Added BeforeAndAfter 5f6bfd2 [bilna] Added BeforeAndAfter e8b6623 [Bilna P] Update MQTTStreamSuite.scala 5ca6691 [Bilna P] Update MQTTStreamSuite.scala 8616495 [bilna] [SPARK-4631] unit test for MQTT (cherry picked from commit e767d7ddac5c2330af553f2a74b8575dfc7afb67) Signed-off-by: Tathagata Das <tathagata.das1565@gmail.com>
* [SPARK-1010] Clean up uses of System.setProperty in unit testsJosh Rosen2014-12-314-8/+24
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Several of our tests call System.setProperty (or test code which implicitly sets system properties) and don't always reset/clear the modified properties, which can create ordering dependencies between tests and cause hard-to-diagnose failures. This patch removes most uses of System.setProperty from our tests, since in most cases we can use SparkConf to set these configurations (there are a few exceptions, including the tests of SparkConf itself). For the cases where we continue to use System.setProperty, this patch introduces a `ResetSystemProperties` ScalaTest mixin class which snapshots the system properties before individual tests and to automatically restores them on test completion / failure. See the block comment at the top of the ResetSystemProperties class for more details. Author: Josh Rosen <joshrosen@databricks.com> Closes #3739 from JoshRosen/cleanup-system-properties-in-tests and squashes the following commits: 0236d66 [Josh Rosen] Replace setProperty uses in two example programs / tools 3888fe3 [Josh Rosen] Remove setProperty use in LocalJavaStreamingContext 4f4031d [Josh Rosen] Add note on why SparkSubmitSuite needs ResetSystemProperties 4742a5b [Josh Rosen] Clarify ResetSystemProperties trait inheritance ordering. 0eaf0b6 [Josh Rosen] Remove setProperty call in TaskResultGetterSuite. 7a3d224 [Josh Rosen] Fix trait ordering 3fdb554 [Josh Rosen] Remove setProperty call in TaskSchedulerImplSuite bee20df [Josh Rosen] Remove setProperty calls in SparkContextSchedulerCreationSuite 655587c [Josh Rosen] Remove setProperty calls in JobCancellationSuite 3f2f955 [Josh Rosen] Remove System.setProperty calls in DistributedSuite cfe9cce [Josh Rosen] Remove use of system properties in SparkContextSuite 8783ab0 [Josh Rosen] Remove TestUtils.setSystemProperty, since it is subsumed by the ResetSystemProperties trait. 633a84a [Josh Rosen] Remove use of system properties in FileServerSuite 25bfce2 [Josh Rosen] Use ResetSystemProperties in UtilsSuite 1d1aa5a [Josh Rosen] Use ResetSystemProperties in SizeEstimatorSuite dd9492b [Josh Rosen] Use ResetSystemProperties in AkkaUtilsSuite b0daff2 [Josh Rosen] Use ResetSystemProperties in BlockManagerSuite e9ded62 [Josh Rosen] Use ResetSystemProperties in TaskSchedulerImplSuite 5b3cb54 [Josh Rosen] Use ResetSystemProperties in SparkListenerSuite 0995c4b [Josh Rosen] Use ResetSystemProperties in SparkContextSchedulerCreationSuite c83ded8 [Josh Rosen] Use ResetSystemProperties in SparkConfSuite 51aa870 [Josh Rosen] Use withSystemProperty in ShuffleSuite 60a63a1 [Josh Rosen] Use ResetSystemProperties in JobCancellationSuite 14a92e4 [Josh Rosen] Use withSystemProperty in FileServerSuite 628f46c [Josh Rosen] Use ResetSystemProperties in DistributedSuite 9e3e0dd [Josh Rosen] Add ResetSystemProperties test fixture mixin; use it in SparkSubmitSuite. 4dcea38 [Josh Rosen] Move withSystemProperty to TestUtils class. (cherry picked from commit 352ed6bbe3c3b67e52e298e7c535ae414d96beca) Signed-off-by: Josh Rosen <joshrosen@databricks.com>
* fixed spelling errors in documentationPeter Klipfel2014-12-141-3/+3
| | | | | | | | | | | | | changed "form" to "from" in 3 documentation entries for Kafka integration Author: Peter Klipfel <peter@klipfel.me> Closes #3691 from peterklipfel/master and squashes the following commits: 0fe7fc5 [Peter Klipfel] fixed spelling errors in documentation (cherry picked from commit 2a2983f7c5edce8e4d1d7592adcf227cbd462ae9) Signed-off-by: Josh Rosen <joshrosen@databricks.com>
* Preparing development version 1.2.1-SNAPSHOTPatrick Wendell2014-12-106-6/+6
|
* Preparing Spark release v1.2.0-rc2v1.2.0Patrick Wendell2014-12-106-6/+6
|
* Revert "Preparing Spark release v1.2.0-rc2"Patrick Wendell2014-12-106-6/+6
| | | | This reverts commit 2b72c569a674cccf79ebbe8d067b8dbaaf78007f.
* Revert "Preparing development version 1.2.1-SNAPSHOT"Patrick Wendell2014-12-106-6/+6
| | | | This reverts commit bc05df8a23ba7ad485f6844f28f96551b13ba461.
* Preparing development version 1.2.1-SNAPSHOTPatrick Wendell2014-12-046-6/+6
|
* Preparing Spark release v1.2.0-rc2Patrick Wendell2014-12-046-6/+6
|
* Revert "Preparing Spark release v1.2.0-rc1"Patrick Wendell2014-12-046-6/+6
| | | | This reverts commit 1056e9ec13203d0c51564265e94d77a054498fdb.
* Revert "Preparing development version 1.2.1-SNAPSHOT"Patrick Wendell2014-12-046-6/+6
| | | | This reverts commit 00316cc87983b844f6603f351a8f0b84fe1f6035.
* Preparing development version 1.2.1-SNAPSHOTPatrick Wendell2014-11-286-6/+6
|
* Preparing Spark release v1.2.0-rc1Patrick Wendell2014-11-286-6/+6
|
* Revert "Preparing Spark release v1.2.0-rc1"Patrick Wendell2014-11-286-6/+6
| | | | This reverts commit 39c7d1c1f9a7785285cf4c20dfbffd96f72d5634.
* Revert "Preparing development version 1.2.1-SNAPSHOT"Patrick Wendell2014-11-286-6/+6
| | | | This reverts commit fc7bff00ac731d2632213a98cd92dc5e84ce7dcd.
* Preparing development version 1.2.1-SNAPSHOTPatrick Wendell2014-11-286-6/+6
|
* Preparing Spark release v1.2.0-rc1Patrick Wendell2014-11-286-6/+6
|
* Revert "Preparing Spark release v1.2.0-rc1"Patrick Wendell2014-11-266-6/+6
| | | | This reverts commit cc2c05e4ee81d2f34873a2ebb9a5272867cb65c2.
* Revert "Preparing development version 1.2.1-SNAPSHOT"Patrick Wendell2014-11-266-6/+6
| | | | This reverts commit 380eba5f49eca1dbd4084e6c84e19866fffd4efa.
* Preparing development version 1.2.1-SNAPSHOTPatrick Wendell2014-11-266-6/+6
|
* Preparing Spark release v1.2.0-rc1Patrick Wendell2014-11-266-6/+6
|
* Revert "Preparing Spark release v1.2.0-rc1"Patrick Wendell2014-11-266-6/+6
| | | | This reverts commit 5247dd859b95a440baa562b9827bdeb26aa6530e.
* Revert "Preparing development version 1.2.1-SNAPSHOT"Patrick Wendell2014-11-266-6/+6
| | | | This reverts commit 79df6b43ae762263a8120f423ddb4a0811dd4b6f.
* Preparing development version 1.2.1-SNAPSHOTPatrick Wendell2014-11-266-6/+6
|
* Preparing Spark release v1.2.0-rc1Patrick Wendell2014-11-266-6/+6
|
* Revert "Preparing Spark release v1.2.0-rc1"Patrick Wendell2014-11-266-6/+6
| | | | This reverts commit db7f4a898af22a02b36428507f8ef2b429d78dc1.
* Revert "Preparing development version 1.2.1-SNAPSHOT"Patrick Wendell2014-11-266-6/+6
| | | | This reverts commit d7b1ecb25676d228deb6fe05efdb4e2ab9c3e30b.
* Preparing development version 1.2.1-SNAPSHOTUbuntu2014-11-266-6/+6
|
* Preparing Spark release v1.2.0-rc1Ubuntu2014-11-266-6/+6
|
generated by cgit v1.2.3 (git 2.39.1) at 2024-09-06 19:47:57 +0000