diff options
| author | Huaxin Gao <huaxing@oc0558782468.ibm.com> | 2015-11-05 09:41:14 -0800 |
|---|---|---|
| committer | Reynold Xin <rxin@databricks.com> | 2015-11-05 09:41:14 -0800 |
| commit | b072ff4d1d05fc212cd7036d1897a032a395f0b3 (patch) | |
| tree | 50a10a4bdd4c9b4f9a0ed0e537b2b8a598ea9e67 /sql | |
| parent | a4b5cefcf1a196e6b257e6127d6b43a7e50200ac (diff) | |
| download | spark-b072ff4d1d05fc212cd7036d1897a032a395f0b3.tar.gz spark-b072ff4d1d05fc212cd7036d1897a032a395f0b3.tar.bz2 spark-b072ff4d1d05fc212cd7036d1897a032a395f0b3.zip | |
[SPARK-11474][SQL] change fetchSize to fetchsize
In DefaultDataSource.scala, it has
override def createRelation(
sqlContext: SQLContext,
parameters: Map[String, String]): BaseRelation
The parameters is CaseInsensitiveMap.
After this line
parameters.foreach(kv => properties.setProperty(kv._1, kv._2))
properties is set to all lower case key/value pairs and fetchSize becomes fetchsize.
However, in compute method in JDBCRDD, it has
val fetchSize = properties.getProperty("fetchSize", "0").toInt
so fetchSize value is always 0 and never gets set correctly.
Author: Huaxin Gao <huaxing@oc0558782468.ibm.com>
Closes #9473 from huaxingao/spark-11474.
Diffstat (limited to 'sql')
| -rw-r--r-- | sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCRDD.scala | 3 |
1 files changed, 2 insertions, 1 deletions
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCRDD.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCRDD.scala index 730d88b024..018a009fbd 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCRDD.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCRDD.scala @@ -347,6 +347,7 @@ private[sql] class JDBCRDD( /** * Runs the SQL query against the JDBC driver. + * */ override def compute(thePart: Partition, context: TaskContext): Iterator[InternalRow] = new Iterator[InternalRow] { @@ -368,7 +369,7 @@ private[sql] class JDBCRDD( val sqlText = s"SELECT $columnList FROM $fqTable $myWhereClause" val stmt = conn.prepareStatement(sqlText, ResultSet.TYPE_FORWARD_ONLY, ResultSet.CONCUR_READ_ONLY) - val fetchSize = properties.getProperty("fetchSize", "0").toInt + val fetchSize = properties.getProperty("fetchsize", "0").toInt stmt.setFetchSize(fetchSize) val rs = stmt.executeQuery() |