diff options
| author | hyukjinkwon <gurwls223@gmail.com> | 2015-11-11 16:46:04 +0800 |
|---|---|---|
| committer | Cheng Lian <lian@databricks.com> | 2015-11-11 16:46:04 +0800 |
| commit | 1bc41125ee6306e627be212969854f639969c440 (patch) | |
| tree | f2ddd4fdd429ed284f508086848dc1cba0f8be1c /licenses | |
| parent | 99f5f988612b3093d73d9ce98819767e822fcbff (diff) | |
| download | spark-1bc41125ee6306e627be212969854f639969c440.tar.gz spark-1bc41125ee6306e627be212969854f639969c440.tar.bz2 spark-1bc41125ee6306e627be212969854f639969c440.zip | |
[SPARK-11500][SQL] Not deterministic order of columns when using merging schemas.
https://issues.apache.org/jira/browse/SPARK-11500
As filed in SPARK-11500, if merging schemas is enabled, the order of files to touch is a matter which might affect the ordering of the output columns.
This was mostly because of the use of `Set` and `Map` so I replaced them to `LinkedHashSet` and `LinkedHashMap` to keep the insertion order.
Also, I changed `reduceOption` to `reduceLeftOption`, and replaced the order of `filesToTouch` from `metadataStatuses ++ commonMetadataStatuses ++ needMerged` to `needMerged ++ metadataStatuses ++ commonMetadataStatuses` in order to touch the part-files first which always have the schema in footers whereas the others might not exist.
One nit is, If merging schemas is not enabled, but when multiple files are given, there is no guarantee of the output order, since there might not be a summary file for the first file, which ends up putting ahead the columns of the other files.
However, I thought this should be okay since disabling merging schemas means (assumes) all the files have the same schemas.
In addition, in the test code for this, I only checked the names of fields.
Author: hyukjinkwon <gurwls223@gmail.com>
Closes #9517 from HyukjinKwon/SPARK-11500.
Diffstat (limited to 'licenses')
0 files changed, 0 insertions, 0 deletions