diff options
| author | Reynold Xin <rxin@databricks.com> | 2015-11-11 10:17:54 -0800 |
|---|---|---|
| committer | Reynold Xin <rxin@databricks.com> | 2015-11-11 10:17:54 -0800 |
| commit | 95daff6459fc749949c2d71a0b7ab1c5be854f70 (patch) | |
| tree | bb448664ae5fbe1806c4206f4325a4d26a5e8be6 /core/src/main/scala/org/apache/spark/SparkContext.scala | |
| parent | 27524a3a9ccee6fbe56149180ebfb3f74e0957e7 (diff) | |
| download | spark-95daff6459fc749949c2d71a0b7ab1c5be854f70.tar.gz spark-95daff6459fc749949c2d71a0b7ab1c5be854f70.tar.bz2 spark-95daff6459fc749949c2d71a0b7ab1c5be854f70.zip | |
[SPARK-11646] WholeTextFileRDD should return Text rather than String
If it returns Text, we can reuse this in Spark SQL to provide a WholeTextFile data source and directly convert the Text into UTF8String without extra string decoding and encoding.
Author: Reynold Xin <rxin@databricks.com>
Closes #9622 from rxin/SPARK-11646.
Diffstat (limited to 'core/src/main/scala/org/apache/spark/SparkContext.scala')
| -rw-r--r-- | core/src/main/scala/org/apache/spark/SparkContext.scala | 6 |
1 files changed, 3 insertions, 3 deletions
diff --git a/core/src/main/scala/org/apache/spark/SparkContext.scala b/core/src/main/scala/org/apache/spark/SparkContext.scala index 67270c38fa..43a241686f 100644 --- a/core/src/main/scala/org/apache/spark/SparkContext.scala +++ b/core/src/main/scala/org/apache/spark/SparkContext.scala @@ -863,10 +863,10 @@ class SparkContext(config: SparkConf) extends Logging with ExecutorAllocationCli new WholeTextFileRDD( this, classOf[WholeTextFileInputFormat], - classOf[String], - classOf[String], + classOf[Text], + classOf[Text], updateConf, - minPartitions).setName(path) + minPartitions).setName(path).map(record => (record._1.toString, record._2.toString)) } /** |