diff options
| author | Cheng Hao <hao.cheng@intel.com> | 2015-08-05 22:35:55 +0800 |
|---|---|---|
| committer | Cheng Lian <lian@databricks.com> | 2015-08-05 22:35:55 +0800 |
| commit | 519cf6d3f764a977770266784d6902fe205a070f (patch) | |
| tree | 2bd9e34a52613ee3c06757aa3dec2a653e7f2ee1 /docs/mllib-data-types.md | |
| parent | eb8bfa3eaa0846d685e4d12f9ee2e4273b85edcf (diff) | |
| download | spark-519cf6d3f764a977770266784d6902fe205a070f.tar.gz spark-519cf6d3f764a977770266784d6902fe205a070f.tar.bz2 spark-519cf6d3f764a977770266784d6902fe205a070f.zip | |
[SPARK-9381] [SQL] Migrate JSON data source to the new partitioning data source
Support partitioning for the JSON data source.
Still 2 open issues for the `HadoopFsRelation`
- `refresh()` will invoke the `discoveryPartition()`, which will auto infer the data type for the partition columns, and maybe conflict with the given partition columns. (TODO enable `HadoopFsRelationSuite.Partition column type casting"
- When insert data into a cached HadoopFsRelation based table, we need to invalidate the cache after the insertion (TODO enable `InsertSuite.Caching`)
Author: Cheng Hao <hao.cheng@intel.com>
Closes #7696 from chenghao-intel/json and squashes the following commits:
d90b104 [Cheng Hao] revert the change for JacksonGenerator.apply
307111d [Cheng Hao] fix bug in the unit test
8738c8a [Cheng Hao] fix bug in unit testing
35f2cde [Cheng Hao] support partition for json format
Diffstat (limited to 'docs/mllib-data-types.md')
0 files changed, 0 insertions, 0 deletions