diff options
| author | Daoyuan Wang <daoyuan.wang@intel.com> | 2016-07-18 13:58:12 -0700 |
|---|---|---|
| committer | Yin Huai <yhuai@databricks.com> | 2016-07-18 13:58:12 -0700 |
| commit | 96e9afaae93318250334211cc80ed0fee3d055b9 (patch) | |
| tree | 095e699d596da28fca8d1239f705dea02f9b045b /docs/sparkr.md | |
| parent | 2877f1a5224c38c1fa0b85ef633ff935fae9dd83 (diff) | |
| download | spark-96e9afaae93318250334211cc80ed0fee3d055b9.tar.gz spark-96e9afaae93318250334211cc80ed0fee3d055b9.tar.bz2 spark-96e9afaae93318250334211cc80ed0fee3d055b9.zip | |
[SPARK-16515][SQL] set default record reader and writer for script transformation
## What changes were proposed in this pull request?
In ScriptInputOutputSchema, we read default RecordReader and RecordWriter from conf. Since Spark 2.0 has deleted those config keys from hive conf, we have to set default reader/writer class name by ourselves. Otherwise we will get None for LazySimpleSerde, the data written would not be able to read by script. The test case added worked fine with previous version of Spark, but would fail now.
## How was this patch tested?
added a test case in SQLQuerySuite.
Closes #14169
Author: Daoyuan Wang <daoyuan.wang@intel.com>
Author: Yin Huai <yhuai@databricks.com>
Closes #14249 from yhuai/scriptTransformation.
Diffstat (limited to 'docs/sparkr.md')
0 files changed, 0 insertions, 0 deletions