diff options
author | Josh Rosen <joshrosen@databricks.com> | 2015-06-26 08:12:22 -0700 |
---|---|---|
committer | Davies Liu <davies@databricks.com> | 2015-06-26 08:12:22 -0700 |
commit | 41afa16500e682475eaa80e31c0434b7ab66abcb (patch) | |
tree | 298d6e9e0739fa91a99422c1308eb15ff6ad12f3 /python/pyspark/profiler.py | |
parent | 37bf76a2de2143ec6348a3d43b782227849520cc (diff) | |
download | spark-41afa16500e682475eaa80e31c0434b7ab66abcb.tar.gz spark-41afa16500e682475eaa80e31c0434b7ab66abcb.tar.bz2 spark-41afa16500e682475eaa80e31c0434b7ab66abcb.zip |
[SPARK-8652] [PYSPARK] Check return value for all uses of doctest.testmod()
This patch addresses a critical issue in the PySpark tests:
Several of our Python modules' `__main__` methods call `doctest.testmod()` in order to run doctests but forget to check and handle its return value. As a result, some PySpark test failures can go unnoticed because they will not fail the build.
Fortunately, there was only one test failure which was masked by this bug: a `pyspark.profiler` doctest was failing due to changes in RDD pipelining.
Author: Josh Rosen <joshrosen@databricks.com>
Closes #7032 from JoshRosen/testmod-fix and squashes the following commits:
60dbdc0 [Josh Rosen] Account for int vs. long formatting change in Python 3
8b8d80a [Josh Rosen] Fix failing test.
e6423f9 [Josh Rosen] Check return code for all uses of doctest.testmod().
Diffstat (limited to 'python/pyspark/profiler.py')
-rw-r--r-- | python/pyspark/profiler.py | 8 |
1 files changed, 6 insertions, 2 deletions
diff --git a/python/pyspark/profiler.py b/python/pyspark/profiler.py index d18daaabfc..44d17bd629 100644 --- a/python/pyspark/profiler.py +++ b/python/pyspark/profiler.py @@ -90,9 +90,11 @@ class Profiler(object): >>> sc = SparkContext('local', 'test', conf=conf, profiler_cls=MyCustomProfiler) >>> sc.parallelize(range(1000)).map(lambda x: 2 * x).take(10) [0, 2, 4, 6, 8, 10, 12, 14, 16, 18] + >>> sc.parallelize(range(1000)).count() + 1000 >>> sc.show_profiles() My custom profiles for RDD:1 - My custom profiles for RDD:2 + My custom profiles for RDD:3 >>> sc.stop() """ @@ -169,4 +171,6 @@ class BasicProfiler(Profiler): if __name__ == "__main__": import doctest - doctest.testmod() + (failure_count, test_count) = doctest.testmod() + if failure_count: + exit(-1) |