diff options
| author | Reynold Xin <rxin@databricks.com> | 2016-06-30 16:51:11 -0700 |
|---|---|---|
| committer | Reynold Xin <rxin@databricks.com> | 2016-06-30 16:51:11 -0700 |
| commit | 3d75a5b2a76eba0855d73476dc2fd579c612d521 (patch) | |
| tree | e7ea7bca28678511cc83122a6b07c12bdbd3b27a /python | |
| parent | fb41670c9263a89ec233861cc91a19cf1bb19073 (diff) | |
| download | spark-3d75a5b2a76eba0855d73476dc2fd579c612d521.tar.gz spark-3d75a5b2a76eba0855d73476dc2fd579c612d521.tar.bz2 spark-3d75a5b2a76eba0855d73476dc2fd579c612d521.zip | |
[SPARK-16313][SQL] Spark should not silently drop exceptions in file listing
## What changes were proposed in this pull request?
Spark silently drops exceptions during file listing. This is a very bad behavior because it can mask legitimate errors and the resulting plan will silently have 0 rows. This patch changes it to not silently drop the errors.
## How was this patch tested?
Manually verified.
Author: Reynold Xin <rxin@databricks.com>
Closes #13987 from rxin/SPARK-16313.
Diffstat (limited to 'python')
| -rw-r--r-- | python/pyspark/sql/context.py | 2 | ||||
| -rw-r--r-- | python/pyspark/sql/streaming.py | 2 |
2 files changed, 2 insertions, 2 deletions
diff --git a/python/pyspark/sql/context.py b/python/pyspark/sql/context.py index 3503fb90c3..8c984b36b7 100644 --- a/python/pyspark/sql/context.py +++ b/python/pyspark/sql/context.py @@ -440,7 +440,7 @@ class SQLContext(object): :return: :class:`DataStreamReader` - >>> text_sdf = sqlContext.readStream.text(os.path.join(tempfile.mkdtemp(), 'data')) + >>> text_sdf = sqlContext.readStream.text(tempfile.mkdtemp()) >>> text_sdf.isStreaming True """ diff --git a/python/pyspark/sql/streaming.py b/python/pyspark/sql/streaming.py index 8cf70983a4..bffe398247 100644 --- a/python/pyspark/sql/streaming.py +++ b/python/pyspark/sql/streaming.py @@ -437,7 +437,7 @@ class DataStreamReader(OptionUtils): :param paths: string, or list of strings, for input path(s). - >>> text_sdf = spark.readStream.text(os.path.join(tempfile.mkdtemp(), 'data')) + >>> text_sdf = spark.readStream.text(tempfile.mkdtemp()) >>> text_sdf.isStreaming True >>> "value" in str(text_sdf.schema) |