diff options
| author | Burak Yavuz <brkyvz@gmail.com> | 2015-12-04 12:08:42 -0800 |
|---|---|---|
| committer | Tathagata Das <tathagata.das1565@gmail.com> | 2015-12-04 12:08:42 -0800 |
| commit | 302d68de87dbaf1974accf49de26fc01fc0eb089 (patch) | |
| tree | 1ac91946df43759ee620e3e0e633f8636d62ee76 /python/pyspark/streaming/tests.py | |
| parent | d0d82227785dcd6c49a986431c476c7838a9541c (diff) | |
| download | spark-302d68de87dbaf1974accf49de26fc01fc0eb089.tar.gz spark-302d68de87dbaf1974accf49de26fc01fc0eb089.tar.bz2 spark-302d68de87dbaf1974accf49de26fc01fc0eb089.zip | |
[SPARK-12058][STREAMING][KINESIS][TESTS] fix Kinesis python tests
Python tests require access to the `KinesisTestUtils` file. When this file exists under src/test, python can't access it, since it is not available in the assembly jar.
However, if we move KinesisTestUtils to src/main, we need to add the KinesisProducerLibrary as a dependency. In order to avoid this, I moved KinesisTestUtils to src/main, and extended it with ExtendedKinesisTestUtils which is under src/test that adds support for the KPL.
cc zsxwing tdas
Author: Burak Yavuz <brkyvz@gmail.com>
Closes #10050 from brkyvz/kinesis-py.
Diffstat (limited to 'python/pyspark/streaming/tests.py')
| -rw-r--r-- | python/pyspark/streaming/tests.py | 1 |
1 files changed, 0 insertions, 1 deletions
diff --git a/python/pyspark/streaming/tests.py b/python/pyspark/streaming/tests.py index d50c6b8d4a..a2bfd79e1a 100644 --- a/python/pyspark/streaming/tests.py +++ b/python/pyspark/streaming/tests.py @@ -1458,7 +1458,6 @@ class KinesisStreamTests(PySparkStreamingTestCase): InitialPositionInStream.LATEST, 2, StorageLevel.MEMORY_AND_DISK_2, "awsAccessKey", "awsSecretKey") - @unittest.skip("Enable it when we fix SPAKR-12058") def test_kinesis_stream(self): if not are_kinesis_tests_enabled: sys.stderr.write( |