diff options
author | Reynold Xin <rxin@databricks.com> | 2016-06-15 20:05:08 -0700 |
---|---|---|
committer | Reynold Xin <rxin@databricks.com> | 2016-06-15 20:05:08 -0700 |
commit | 865e7cc38d2b7cf2d4f7e7b04ccb7b17791a693b (patch) | |
tree | 193154d715dcb186f802939a395ebd86a9b9eb0f /mllib/src/main | |
parent | 3e6d567a4688f064f2a2259c8e436b7c628a431c (diff) | |
download | spark-865e7cc38d2b7cf2d4f7e7b04ccb7b17791a693b.tar.gz spark-865e7cc38d2b7cf2d4f7e7b04ccb7b17791a693b.tar.bz2 spark-865e7cc38d2b7cf2d4f7e7b04ccb7b17791a693b.zip |
[SPARK-15979][SQL] Rename various Parquet support classes.
## What changes were proposed in this pull request?
This patch renames various Parquet support classes from CatalystAbc to ParquetAbc. This new naming makes more sense for two reasons:
1. These are not optimizer related (i.e. Catalyst) classes.
2. We are in the Spark code base, and as a result it'd be more clear to call out these are Parquet support classes, rather than some Spark classes.
## How was this patch tested?
Renamed test cases as well.
Author: Reynold Xin <rxin@databricks.com>
Closes #13696 from rxin/parquet-rename.
Diffstat (limited to 'mllib/src/main')
-rw-r--r-- | mllib/src/main/scala/org/apache/spark/ml/source/libsvm/LibSVMRelation.scala | 6 |
1 files changed, 1 insertions, 5 deletions
diff --git a/mllib/src/main/scala/org/apache/spark/ml/source/libsvm/LibSVMRelation.scala b/mllib/src/main/scala/org/apache/spark/ml/source/libsvm/LibSVMRelation.scala index b5b2a681e9..62e09d2e0c 100644 --- a/mllib/src/main/scala/org/apache/spark/ml/source/libsvm/LibSVMRelation.scala +++ b/mllib/src/main/scala/org/apache/spark/ml/source/libsvm/LibSVMRelation.scala @@ -141,7 +141,7 @@ class LibSVMFileFormat extends TextBasedFileFormat with DataSourceRegister { sparkSession: SparkSession, options: Map[String, String], files: Seq[FileStatus]): Map[String, String] = { - def computeNumFeatures(): Int = { + val numFeatures = options.get("numFeatures").filter(_.toInt > 0).getOrElse { val dataFiles = files.filterNot(_.getPath.getName startsWith "_") val path = if (dataFiles.length == 1) { dataFiles.head.getPath.toUri.toString @@ -156,10 +156,6 @@ class LibSVMFileFormat extends TextBasedFileFormat with DataSourceRegister { MLUtils.computeNumFeatures(parsed) } - val numFeatures = options.get("numFeatures").filter(_.toInt > 0).getOrElse { - computeNumFeatures() - } - new CaseInsensitiveMap(options + ("numFeatures" -> numFeatures.toString)) } |