diff options
author | Yin Huai <yhuai@databricks.com> | 2015-02-06 12:38:07 -0800 |
---|---|---|
committer | Michael Armbrust <michael@databricks.com> | 2015-02-06 12:38:07 -0800 |
commit | 3eccf29ce061559c86e6f7338851932fc89a9afd (patch) | |
tree | 9d8dca5ab32cd82f04d6e96287e1d56d8d56559e /sql/hive | |
parent | 0b7eb3f3b700080bf6cb810d092709a8a468e5db (diff) | |
download | spark-3eccf29ce061559c86e6f7338851932fc89a9afd.tar.gz spark-3eccf29ce061559c86e6f7338851932fc89a9afd.tar.bz2 spark-3eccf29ce061559c86e6f7338851932fc89a9afd.zip |
[SPARK-5595][SPARK-5603][SQL] Add a rule to do PreInsert type casting and field renaming and invalidating in memory cache after INSERT
This PR adds a rule to Analyzer that will add preinsert data type casting and field renaming to the select clause in an `INSERT INTO/OVERWRITE` statement. Also, with the change of this PR, we always invalidate our in memory data cache after inserting into a BaseRelation.
cc marmbrus liancheng
Author: Yin Huai <yhuai@databricks.com>
Closes #4373 from yhuai/insertFollowUp and squashes the following commits:
08237a7 [Yin Huai] Merge remote-tracking branch 'upstream/master' into insertFollowUp
316542e [Yin Huai] Doc update.
c9ccfeb [Yin Huai] Revert a unnecessary change.
84aecc4 [Yin Huai] Address comments.
1951fe1 [Yin Huai] Merge remote-tracking branch 'upstream/master'
c18da34 [Yin Huai] Invalidate cache after insert.
727f21a [Yin Huai] Preinsert casting and renaming.
Diffstat (limited to 'sql/hive')
-rw-r--r-- | sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveContext.scala | 1 |
1 files changed, 1 insertions, 0 deletions
diff --git a/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveContext.scala b/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveContext.scala index d2371d4a55..ad37b7d0e6 100644 --- a/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveContext.scala +++ b/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveContext.scala @@ -324,6 +324,7 @@ class HiveContext(sc: SparkContext) extends SQLContext(sc) { catalog.PreInsertionCasts :: ExtractPythonUdfs :: ResolveUdtfsAlias :: + sources.PreInsertCastAndRename :: Nil } |