diff options
| author | Yin Huai <huai@cse.ohio-state.edu> | 2014-08-05 18:56:10 -0700 |
|---|---|---|
| committer | Michael Armbrust <michael@databricks.com> | 2014-08-05 18:56:10 -0700 |
| commit | 69ec678d3aaeb6ece85e5e82353bf083bfc83667 (patch) | |
| tree | 5bb73c4135b8971c8bbaf8811b7efaf6e68449d2 /sql | |
| parent | d0ae3f3912104a8227cd964c42e229a297a48ffa (diff) | |
| download | spark-69ec678d3aaeb6ece85e5e82353bf083bfc83667.tar.gz spark-69ec678d3aaeb6ece85e5e82353bf083bfc83667.tar.bz2 spark-69ec678d3aaeb6ece85e5e82353bf083bfc83667.zip | |
[SPARK-2854][SQL] Finalize _acceptable_types in pyspark.sql
This PR aims to finalize accepted data value types in Python RDDs provided to Python `applySchema`.
JIRA: https://issues.apache.org/jira/browse/SPARK-2854
Author: Yin Huai <huai@cse.ohio-state.edu>
Closes #1793 from yhuai/SPARK-2854 and squashes the following commits:
32f0708 [Yin Huai] LongType only accepts long values.
c2b23dd [Yin Huai] Do data type conversions based on the specified Spark SQL data type.
Diffstat (limited to 'sql')
| -rw-r--r-- | sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala | 3 |
1 files changed, 3 insertions, 0 deletions
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala b/sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala index ecd5fbaa0b..71d338d21d 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala @@ -491,7 +491,10 @@ class SQLContext(@transient val sparkContext: SparkContext) new java.sql.Timestamp(c.getTime().getTime()) case (c: Int, ByteType) => c.toByte + case (c: Long, ByteType) => c.toByte case (c: Int, ShortType) => c.toShort + case (c: Long, ShortType) => c.toShort + case (c: Long, IntegerType) => c.toInt case (c: Double, FloatType) => c.toFloat case (c, StringType) if !c.isInstanceOf[String] => c.toString |