diff options
| author | Liang-Chi Hsieh <viirya@gmail.com> | 2015-02-01 17:52:18 -0800 |
|---|---|---|
| committer | Michael Armbrust <michael@databricks.com> | 2015-02-01 17:52:18 -0800 |
| commit | ef89b82d831d1d35dfaa6387ff2077ea2f2073cc (patch) | |
| tree | 8c2479ece3cd5519791bf6d34e9e823ba2e9a897 /sql/core | |
| parent | 883bc88d520b27bdeb74a1837b45ef0b59753568 (diff) | |
| download | spark-ef89b82d831d1d35dfaa6387ff2077ea2f2073cc.tar.gz spark-ef89b82d831d1d35dfaa6387ff2077ea2f2073cc.tar.bz2 spark-ef89b82d831d1d35dfaa6387ff2077ea2f2073cc.zip | |
[Minor][SQL] Little refactor DataFrame related codes
Simplify some codes related to DataFrame.
* Calling `toAttributes` instead of a `map`.
* Original `createDataFrame` creates the `StructType` and its attributes in a redundant way. Refactored it to create `StructType` and call `toAttributes` on it directly.
Author: Liang-Chi Hsieh <viirya@gmail.com>
Closes #4298 from viirya/refactor_df and squashes the following commits:
1d61c64 [Liang-Chi Hsieh] Revert it.
f36efb5 [Liang-Chi Hsieh] Relax the constraint of toDataFrame.
2c9f370 [Liang-Chi Hsieh] Just refactor DataFrame codes.
Diffstat (limited to 'sql/core')
| -rw-r--r-- | sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala | 4 |
1 files changed, 2 insertions, 2 deletions
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala b/sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala index f87fde4ed8..84933dd944 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala @@ -168,8 +168,8 @@ class SQLContext(@transient val sparkContext: SparkContext) */ implicit def createDataFrame[A <: Product: TypeTag](rdd: RDD[A]): DataFrame = { SparkPlan.currentContext.set(self) - val attributeSeq = ScalaReflection.attributesFor[A] - val schema = StructType.fromAttributes(attributeSeq) + val schema = ScalaReflection.schemaFor[A].dataType.asInstanceOf[StructType] + val attributeSeq = schema.toAttributes val rowRDD = RDDConversions.productToRowRdd(rdd, schema) new DataFrame(this, LogicalRDD(attributeSeq, rowRDD)(self)) } |