diff options
| author | jiangxingbo <jiangxb1987@gmail.com> | 2016-08-24 23:36:04 -0700 |
|---|---|---|
| committer | Reynold Xin <rxin@databricks.com> | 2016-08-24 23:36:04 -0700 |
| commit | 5f02d2e5b4d37f554629cbd0e488e856fffd7b6b (patch) | |
| tree | 00d1c3d44c017f3960888b37c15b08ec2c6d603e /python/pyspark/sql/streaming.py | |
| parent | 4d0706d616176dc29ff3562e40cb00dd4eb9c302 (diff) | |
| download | spark-5f02d2e5b4d37f554629cbd0e488e856fffd7b6b.tar.gz spark-5f02d2e5b4d37f554629cbd0e488e856fffd7b6b.tar.bz2 spark-5f02d2e5b4d37f554629cbd0e488e856fffd7b6b.zip | |
[SPARK-17215][SQL] Method `SQLContext.parseDataType(dataTypeString: String)` could be removed.
## What changes were proposed in this pull request?
Method `SQLContext.parseDataType(dataTypeString: String)` could be removed, we should use `SparkSession.parseDataType(dataTypeString: String)` instead.
This require updating PySpark.
## How was this patch tested?
Existing test cases.
Author: jiangxingbo <jiangxb1987@gmail.com>
Closes #14790 from jiangxb1987/parseDataType.
Diffstat (limited to 'python/pyspark/sql/streaming.py')
| -rw-r--r-- | python/pyspark/sql/streaming.py | 4 |
1 files changed, 3 insertions, 1 deletions
diff --git a/python/pyspark/sql/streaming.py b/python/pyspark/sql/streaming.py index 3761d2b199..a0ba5825f3 100644 --- a/python/pyspark/sql/streaming.py +++ b/python/pyspark/sql/streaming.py @@ -273,9 +273,11 @@ class DataStreamReader(OptionUtils): >>> s = spark.readStream.schema(sdf_schema) """ + from pyspark.sql import SparkSession if not isinstance(schema, StructType): raise TypeError("schema should be StructType") - jschema = self._spark._ssql_ctx.parseDataType(schema.json()) + spark = SparkSession.builder.getOrCreate() + jschema = spark._jsparkSession.parseDataType(schema.json()) self._jreader = self._jreader.schema(jschema) return self |