diff options
| author | hyukjinkwon <gurwls223@gmail.com> | 2016-05-13 09:04:37 -0700 |
|---|---|---|
| committer | Reynold Xin <rxin@databricks.com> | 2016-05-13 09:04:37 -0700 |
| commit | 3ded5bc4db2badc9ff49554e73421021d854306b (patch) | |
| tree | c8e8bbb95806cddb899c971c333a6088e443fbe9 /mllib/src/main/scala | |
| parent | 10a838967455db80d750ef84a1c6b3088b19fd9f (diff) | |
| download | spark-3ded5bc4db2badc9ff49554e73421021d854306b.tar.gz spark-3ded5bc4db2badc9ff49554e73421021d854306b.tar.bz2 spark-3ded5bc4db2badc9ff49554e73421021d854306b.zip | |
[SPARK-15267][SQL] Refactor options for JDBC and ORC data sources and change default compression for ORC
## What changes were proposed in this pull request?
Currently, Parquet, JSON and CSV data sources have a class for thier options, (`ParquetOptions`, `JSONOptions` and `CSVOptions`).
It is convenient to manage options for sources to gather options into a class. Currently, `JDBC`, `Text`, `libsvm` and `ORC` datasources do not have this class. This might be nicer if these options are in a unified format so that options can be added and
This PR refactors the options in Spark internal data sources adding new classes, `OrcOptions`, `TextOptions`, `JDBCOptions` and `LibSVMOptions`.
Also, this PR change the default compression codec for ORC from `NONE` to `SNAPPY`.
## How was this patch tested?
Existing tests should cover this for refactoring and unittests in `OrcHadoopFsRelationSuite` for changing the default compression codec for ORC.
Author: hyukjinkwon <gurwls223@gmail.com>
Closes #13048 from HyukjinKwon/SPARK-15267.
Diffstat (limited to 'mllib/src/main/scala')
| -rw-r--r-- | mllib/src/main/scala/org/apache/spark/ml/source/libsvm/LibSVMRelation.scala | 2 |
1 files changed, 1 insertions, 1 deletions
diff --git a/mllib/src/main/scala/org/apache/spark/ml/source/libsvm/LibSVMRelation.scala b/mllib/src/main/scala/org/apache/spark/ml/source/libsvm/LibSVMRelation.scala index 68a855c99f..39bdd1afad 100644 --- a/mllib/src/main/scala/org/apache/spark/ml/source/libsvm/LibSVMRelation.scala +++ b/mllib/src/main/scala/org/apache/spark/ml/source/libsvm/LibSVMRelation.scala @@ -32,7 +32,7 @@ import org.apache.spark.mllib.util.MLUtils import org.apache.spark.sql.{DataFrame, DataFrameReader, Row, SparkSession} import org.apache.spark.sql.catalyst.InternalRow import org.apache.spark.sql.catalyst.encoders.RowEncoder -import org.apache.spark.sql.catalyst.expressions.{AttributeReference, JoinedRow} +import org.apache.spark.sql.catalyst.expressions.AttributeReference import org.apache.spark.sql.catalyst.expressions.codegen.GenerateUnsafeProjection import org.apache.spark.sql.execution.datasources._ import org.apache.spark.sql.sources._ |