From 1b2a918e59addcdccdf8e011bce075cc9dd07b93 Mon Sep 17 00:00:00 2001 From: Sun Rui Date: Wed, 20 Jan 2016 21:08:15 -0800 Subject: [SPARK-12204][SPARKR] Implement drop method for DataFrame in SparkR. Author: Sun Rui Closes #10201 from sun-rui/SPARK-12204. --- docs/sql-programming-guide.md | 13 +++++++++++++ 1 file changed, 13 insertions(+) (limited to 'docs/sql-programming-guide.md') diff --git a/docs/sql-programming-guide.md b/docs/sql-programming-guide.md index bc89c78156..fddc513794 100644 --- a/docs/sql-programming-guide.md +++ b/docs/sql-programming-guide.md @@ -2150,6 +2150,8 @@ options. --conf spark.sql.hive.thriftServer.singleSession=true \ ... {% endhighlight %} + - Since 1.6.1, withColumn method in sparkR supports adding a new column to or replacing existing columns + of the same name of a DataFrame. - From Spark 1.6, LongType casts to TimestampType expect seconds instead of microseconds. This change was made to match the behavior of Hive 1.2 for more consistent type casting to TimestampType @@ -2183,6 +2185,7 @@ options. users can use `REFRESH TABLE` SQL command or `HiveContext`'s `refreshTable` method to include those new files to the table. For a DataFrame representing a JSON dataset, users need to recreate the DataFrame and the new DataFrame will include new files. + - DataFrame.withColumn method in pySpark supports adding a new column or replacing existing columns of the same name. ## Upgrading from Spark SQL 1.3 to 1.4 @@ -2262,6 +2265,16 @@ sqlContext.setConf("spark.sql.retainGroupColumns", "false") +#### Behavior change on DataFrame.withColumn + +Prior to 1.4, DataFrame.withColumn() supports adding a column only. The column will always be added +as a new column with its specified name in the result DataFrame even if there may be any existing +columns of the same name. Since 1.4, DataFrame.withColumn() supports adding a column of a different +name from names of all existing columns or replacing existing columns of the same name. + +Note that this change is only for Scala API, not for PySpark and SparkR. + + ## Upgrading from Spark SQL 1.0-1.2 to 1.3 In Spark 1.3 we removed the "Alpha" label from Spark SQL and as part of this did a cleanup of the -- cgit v1.2.3