aboutsummaryrefslogtreecommitdiff
path: root/mllib
diff options
context:
space:
mode:
authorReynold Xin <rxin@databricks.com>2016-06-15 20:05:08 -0700
committerReynold Xin <rxin@databricks.com>2016-06-15 20:05:08 -0700
commit865e7cc38d2b7cf2d4f7e7b04ccb7b17791a693b (patch)
tree193154d715dcb186f802939a395ebd86a9b9eb0f /mllib
parent3e6d567a4688f064f2a2259c8e436b7c628a431c (diff)
downloadspark-865e7cc38d2b7cf2d4f7e7b04ccb7b17791a693b.tar.gz
spark-865e7cc38d2b7cf2d4f7e7b04ccb7b17791a693b.tar.bz2
spark-865e7cc38d2b7cf2d4f7e7b04ccb7b17791a693b.zip
[SPARK-15979][SQL] Rename various Parquet support classes.
## What changes were proposed in this pull request? This patch renames various Parquet support classes from CatalystAbc to ParquetAbc. This new naming makes more sense for two reasons: 1. These are not optimizer related (i.e. Catalyst) classes. 2. We are in the Spark code base, and as a result it'd be more clear to call out these are Parquet support classes, rather than some Spark classes. ## How was this patch tested? Renamed test cases as well. Author: Reynold Xin <rxin@databricks.com> Closes #13696 from rxin/parquet-rename.
Diffstat (limited to 'mllib')
-rw-r--r--mllib/src/main/scala/org/apache/spark/ml/source/libsvm/LibSVMRelation.scala6
1 files changed, 1 insertions, 5 deletions
diff --git a/mllib/src/main/scala/org/apache/spark/ml/source/libsvm/LibSVMRelation.scala b/mllib/src/main/scala/org/apache/spark/ml/source/libsvm/LibSVMRelation.scala
index b5b2a681e9..62e09d2e0c 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/source/libsvm/LibSVMRelation.scala
+++ b/mllib/src/main/scala/org/apache/spark/ml/source/libsvm/LibSVMRelation.scala
@@ -141,7 +141,7 @@ class LibSVMFileFormat extends TextBasedFileFormat with DataSourceRegister {
sparkSession: SparkSession,
options: Map[String, String],
files: Seq[FileStatus]): Map[String, String] = {
- def computeNumFeatures(): Int = {
+ val numFeatures = options.get("numFeatures").filter(_.toInt > 0).getOrElse {
val dataFiles = files.filterNot(_.getPath.getName startsWith "_")
val path = if (dataFiles.length == 1) {
dataFiles.head.getPath.toUri.toString
@@ -156,10 +156,6 @@ class LibSVMFileFormat extends TextBasedFileFormat with DataSourceRegister {
MLUtils.computeNumFeatures(parsed)
}
- val numFeatures = options.get("numFeatures").filter(_.toInt > 0).getOrElse {
- computeNumFeatures()
- }
-
new CaseInsensitiveMap(options + ("numFeatures" -> numFeatures.toString))
}