aboutsummaryrefslogtreecommitdiff
path: root/docs/mllib-data-types.md
diff options
context:
space:
mode:
authorReynold Xin <rxin@databricks.com>2015-05-15 22:00:31 -0700
committerReynold Xin <rxin@databricks.com>2015-05-15 22:00:31 -0700
commit578bfeeff514228f6fd4b07a536815fbb3510f7e (patch)
tree97964df2b0b7ada4f019f2cd9617ba6af1d59f52 /docs/mllib-data-types.md
parentdeb411335a09b91eb1f75421d77e1c3686719621 (diff)
downloadspark-578bfeeff514228f6fd4b07a536815fbb3510f7e.tar.gz
spark-578bfeeff514228f6fd4b07a536815fbb3510f7e.tar.bz2
spark-578bfeeff514228f6fd4b07a536815fbb3510f7e.zip
[SPARK-7654][SQL] DataFrameReader and DataFrameWriter for input/output API
This patch introduces DataFrameWriter and DataFrameReader. DataFrameReader interface, accessible through SQLContext.read, contains methods that create DataFrames. These methods used to reside in SQLContext. Example usage: ```scala sqlContext.read.json("...") sqlContext.read.parquet("...") ``` DataFrameWriter interface, accessible through DataFrame.write, implements a builder pattern to avoid the proliferation of options in writing DataFrame out. It currently implements: - mode - format (e.g. "parquet", "json") - options (generic options passed down into data sources) - partitionBy (partitioning columns) Example usage: ```scala df.write.mode("append").format("json").partitionBy("date").saveAsTable("myJsonTable") ``` TODO: - [ ] Documentation update - [ ] Move JDBC into reader / writer? - [ ] Deprecate the old interfaces - [ ] Move the generic load interface into reader. - [ ] Update example code and documentation Author: Reynold Xin <rxin@databricks.com> Closes #6175 from rxin/reader-writer and squashes the following commits: b146c95 [Reynold Xin] Deprecation of old APIs. bd8abdf [Reynold Xin] Fixed merge conflict. 26abea2 [Reynold Xin] Added general load methods. 244fbec [Reynold Xin] Added equivalent to example. 4f15d92 [Reynold Xin] Added documentation for partitionBy. 7e91611 [Reynold Xin] [SPARK-7654][SQL] DataFrameReader and DataFrameWriter for input/output API.
Diffstat (limited to 'docs/mllib-data-types.md')
0 files changed, 0 insertions, 0 deletions