diff options
author | Sun Rui <rui.sun@intel.com> | 2016-01-20 21:08:15 -0800 |
---|---|---|
committer | Shivaram Venkataraman <shivaram@cs.berkeley.edu> | 2016-01-20 21:08:15 -0800 |
commit | 1b2a918e59addcdccdf8e011bce075cc9dd07b93 (patch) | |
tree | a4fe927c31db877acf9af143e88081b95633891b /docs/sql-programming-guide.md | |
parent | d7415991a1c65f44ba385bc697b458125366523f (diff) | |
download | spark-1b2a918e59addcdccdf8e011bce075cc9dd07b93.tar.gz spark-1b2a918e59addcdccdf8e011bce075cc9dd07b93.tar.bz2 spark-1b2a918e59addcdccdf8e011bce075cc9dd07b93.zip |
[SPARK-12204][SPARKR] Implement drop method for DataFrame in SparkR.
Author: Sun Rui <rui.sun@intel.com>
Closes #10201 from sun-rui/SPARK-12204.
Diffstat (limited to 'docs/sql-programming-guide.md')
-rw-r--r-- | docs/sql-programming-guide.md | 13 |
1 files changed, 13 insertions, 0 deletions
diff --git a/docs/sql-programming-guide.md b/docs/sql-programming-guide.md index bc89c78156..fddc513794 100644 --- a/docs/sql-programming-guide.md +++ b/docs/sql-programming-guide.md @@ -2150,6 +2150,8 @@ options. --conf spark.sql.hive.thriftServer.singleSession=true \ ... {% endhighlight %} + - Since 1.6.1, withColumn method in sparkR supports adding a new column to or replacing existing columns + of the same name of a DataFrame. - From Spark 1.6, LongType casts to TimestampType expect seconds instead of microseconds. This change was made to match the behavior of Hive 1.2 for more consistent type casting to TimestampType @@ -2183,6 +2185,7 @@ options. users can use `REFRESH TABLE` SQL command or `HiveContext`'s `refreshTable` method to include those new files to the table. For a DataFrame representing a JSON dataset, users need to recreate the DataFrame and the new DataFrame will include new files. + - DataFrame.withColumn method in pySpark supports adding a new column or replacing existing columns of the same name. ## Upgrading from Spark SQL 1.3 to 1.4 @@ -2262,6 +2265,16 @@ sqlContext.setConf("spark.sql.retainGroupColumns", "false") </div> +#### Behavior change on DataFrame.withColumn + +Prior to 1.4, DataFrame.withColumn() supports adding a column only. The column will always be added +as a new column with its specified name in the result DataFrame even if there may be any existing +columns of the same name. Since 1.4, DataFrame.withColumn() supports adding a column of a different +name from names of all existing columns or replacing existing columns of the same name. + +Note that this change is only for Scala API, not for PySpark and SparkR. + + ## Upgrading from Spark SQL 1.0-1.2 to 1.3 In Spark 1.3 we removed the "Alpha" label from Spark SQL and as part of this did a cleanup of the |