diff options
| author | Reynold Xin <rxin@databricks.com> | 2016-04-26 10:58:56 -0700 |
|---|---|---|
| committer | Yin Huai <yhuai@databricks.com> | 2016-04-26 10:58:56 -0700 |
| commit | 5cb03220a02c70d343e82d69cfd30edb894595a1 (patch) | |
| tree | 41232454ab88f2c9c1feadd13d43b8cc46873cf5 /python/pyspark/ml/util.py | |
| parent | 92f66331b4ba3634f54f57ddb5e7962b14aa4ca1 (diff) | |
| download | spark-5cb03220a02c70d343e82d69cfd30edb894595a1.tar.gz spark-5cb03220a02c70d343e82d69cfd30edb894595a1.tar.bz2 spark-5cb03220a02c70d343e82d69cfd30edb894595a1.zip | |
[SPARK-14912][SQL] Propagate data source options to Hadoop configuration
## What changes were proposed in this pull request?
We currently have no way for users to propagate options to the underlying library that rely in Hadoop configurations to work. For example, there are various options in parquet-mr that users might want to set, but the data source API does not expose a per-job way to set it. This patch propagates the user-specified options also into Hadoop Configuration.
## How was this patch tested?
Used a mock data source implementation to test both the read path and the write path.
Author: Reynold Xin <rxin@databricks.com>
Closes #12688 from rxin/SPARK-14912.
Diffstat (limited to 'python/pyspark/ml/util.py')
0 files changed, 0 insertions, 0 deletions