[SPARK-13266] [SQL] None read/writer options were not transalated to "null"

## What changes were proposed in this pull request?

In Python, the `option` and `options` method of `DataFrameReader` and `DataFrameWriter` were sending the string "None" instead of `null` when passed `None`, therefore making it impossible to send an actual `null`. This fixes that problem.

This is based on #11305 from mathieulongtin.

## How was this patch tested?

Added test to readwriter.py.

Author: Liang-Chi Hsieh <simonh@tw.ibm.com>
Author: mathieu longtin <mathieu.longtin@nuance.com>

Closes #12494 from viirya/py-df-none-option.

1 files changed, 5 insertions, 1 deletions

diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVOptions.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVOptions.scala
index 7b9d3b605a..80a0ad7856 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVOptions.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVOptions.scala
@@ -29,6 +29,7 @@ private[sql] class CSVOptions(@transient private val parameters: Map[String, Str
     val paramValue = parameters.get(paramName)
     paramValue match {
       case None => default
+      case Some(null) => default
       case Some(value) if value.length == 0 => '\u0000'
       case Some(value) if value.length == 1 => value.charAt(0)
       case _ => throw new RuntimeException(s"$paramName cannot be more than one character")
@@ -39,6 +40,7 @@ private[sql] class CSVOptions(@transient private val parameters: Map[String, Str
     val paramValue = parameters.get(paramName)
     paramValue match {
       case None => default
+      case Some(null) => default
       case Some(value) => try {
         value.toInt
       } catch {
@@ -50,7 +52,9 @@ private[sql] class CSVOptions(@transient private val parameters: Map[String, Str
 
   private def getBool(paramName: String, default: Boolean = false): Boolean = {
     val param = parameters.getOrElse(paramName, default.toString)
-    if (param.toLowerCase == "true") {
+    if (param == null) {
+      default
+    } else if (param.toLowerCase == "true") {
       true
     } else if (param.toLowerCase == "false") {
       false