diff options
| author | Jurriaan Pruis <email@jurriaanpruis.nl> | 2016-05-25 12:40:16 -0700 |
|---|---|---|
| committer | Reynold Xin <rxin@databricks.com> | 2016-05-25 12:40:16 -0700 |
| commit | c875d81a3de3f209b9eb03adf96b7c740b2c7b52 (patch) | |
| tree | f09a2b335d592b1c40e42b2f557e8f643768dc3e /dev | |
| parent | 4b88067416ce922ae15a1445cf953fb9b5c43427 (diff) | |
| download | spark-c875d81a3de3f209b9eb03adf96b7c740b2c7b52.tar.gz spark-c875d81a3de3f209b9eb03adf96b7c740b2c7b52.tar.bz2 spark-c875d81a3de3f209b9eb03adf96b7c740b2c7b52.zip | |
[SPARK-15493][SQL] default QuoteEscapingEnabled flag to true when writing CSV
## What changes were proposed in this pull request?
Default QuoteEscapingEnabled flag to true when writing CSV and add an escapeQuotes option to be able to change this.
See https://github.com/uniVocity/univocity-parsers/blob/f3eb2af26374940e60d91d1703bde54619f50c51/src/main/java/com/univocity/parsers/csv/CsvWriterSettings.java#L231-L247
This change is needed to be able to write RFC 4180 compatible CSV files (https://tools.ietf.org/html/rfc4180#section-2)
https://issues.apache.org/jira/browse/SPARK-15493
## How was this patch tested?
Added a test that verifies the output is quoted correctly.
Author: Jurriaan Pruis <email@jurriaanpruis.nl>
Closes #13267 from jurriaan/quote-escaping.
Diffstat (limited to 'dev')
| -rw-r--r-- | dev/deps/spark-deps-hadoop-2.2 | 2 | ||||
| -rw-r--r-- | dev/deps/spark-deps-hadoop-2.3 | 2 | ||||
| -rw-r--r-- | dev/deps/spark-deps-hadoop-2.4 | 2 | ||||
| -rw-r--r-- | dev/deps/spark-deps-hadoop-2.6 | 2 | ||||
| -rw-r--r-- | dev/deps/spark-deps-hadoop-2.7 | 2 |
5 files changed, 5 insertions, 5 deletions
diff --git a/dev/deps/spark-deps-hadoop-2.2 b/dev/deps/spark-deps-hadoop-2.2 index a9068da8b2..0ac1c00154 100644 --- a/dev/deps/spark-deps-hadoop-2.2 +++ b/dev/deps/spark-deps-hadoop-2.2 @@ -159,7 +159,7 @@ stax-api-1.0.1.jar stream-2.7.0.jar stringtemplate-3.2.1.jar super-csv-2.2.0.jar -univocity-parsers-2.1.0.jar +univocity-parsers-2.1.1.jar validation-api-1.1.0.Final.jar xbean-asm5-shaded-4.4.jar xmlenc-0.52.jar diff --git a/dev/deps/spark-deps-hadoop-2.3 b/dev/deps/spark-deps-hadoop-2.3 index 7e60a313ae..fa35fa7051 100644 --- a/dev/deps/spark-deps-hadoop-2.3 +++ b/dev/deps/spark-deps-hadoop-2.3 @@ -167,7 +167,7 @@ stax-api-1.0.1.jar stream-2.7.0.jar stringtemplate-3.2.1.jar super-csv-2.2.0.jar -univocity-parsers-2.1.0.jar +univocity-parsers-2.1.1.jar validation-api-1.1.0.Final.jar xbean-asm5-shaded-4.4.jar xmlenc-0.52.jar diff --git a/dev/deps/spark-deps-hadoop-2.4 b/dev/deps/spark-deps-hadoop-2.4 index 70d33b4f48..99dffa93bb 100644 --- a/dev/deps/spark-deps-hadoop-2.4 +++ b/dev/deps/spark-deps-hadoop-2.4 @@ -167,7 +167,7 @@ stax-api-1.0.1.jar stream-2.7.0.jar stringtemplate-3.2.1.jar super-csv-2.2.0.jar -univocity-parsers-2.1.0.jar +univocity-parsers-2.1.1.jar validation-api-1.1.0.Final.jar xbean-asm5-shaded-4.4.jar xmlenc-0.52.jar diff --git a/dev/deps/spark-deps-hadoop-2.6 b/dev/deps/spark-deps-hadoop-2.6 index a80f6bc2a4..a3bee36ce5 100644 --- a/dev/deps/spark-deps-hadoop-2.6 +++ b/dev/deps/spark-deps-hadoop-2.6 @@ -175,7 +175,7 @@ stax-api-1.0.1.jar stream-2.7.0.jar stringtemplate-3.2.1.jar super-csv-2.2.0.jar -univocity-parsers-2.1.0.jar +univocity-parsers-2.1.1.jar validation-api-1.1.0.Final.jar xbean-asm5-shaded-4.4.jar xercesImpl-2.9.1.jar diff --git a/dev/deps/spark-deps-hadoop-2.7 b/dev/deps/spark-deps-hadoop-2.7 index c0b53f73cd..dbd7a8e0bf 100644 --- a/dev/deps/spark-deps-hadoop-2.7 +++ b/dev/deps/spark-deps-hadoop-2.7 @@ -176,7 +176,7 @@ stax-api-1.0.1.jar stream-2.7.0.jar stringtemplate-3.2.1.jar super-csv-2.2.0.jar -univocity-parsers-2.1.0.jar +univocity-parsers-2.1.1.jar validation-api-1.1.0.Final.jar xbean-asm5-shaded-4.4.jar xercesImpl-2.9.1.jar |