aboutsummaryrefslogtreecommitdiff
path: root/sql/hive
diff options
context:
space:
mode:
authorWenchen Fan <wenchen@databricks.com>2016-06-02 17:39:56 -0700
committerAndrew Or <andrew@databricks.com>2016-06-02 17:39:56 -0700
commitf34aadc54ca1a9fd4236a928d342324b26fb3a12 (patch)
tree15dd45cdfe1f0cd64e265984156e8061d8242dcc /sql/hive
parent229f90225748343972d7202c5567b45364cd8497 (diff)
downloadspark-f34aadc54ca1a9fd4236a928d342324b26fb3a12.tar.gz
spark-f34aadc54ca1a9fd4236a928d342324b26fb3a12.tar.bz2
spark-f34aadc54ca1a9fd4236a928d342324b26fb3a12.zip
[SPARK-15718][SQL] better error message for writing bucketed data
## What changes were proposed in this pull request? Currently we don't support bucketing for `save` and `insertInto`. For `save`, we just write the data out into a directory users specified, and it's not a table, we don't keep its metadata. When we read it back, we have no idea if the data is bucketed or not, so it doesn't make sense to use `save` to write bucketed data, as we can't use the bucket information anyway. We can support it in the future, once we have features like bucket discovery, or we save bucket information in the data directory too, so that we don't need to rely on a metastore. For `insertInto`, it inserts data into an existing table, so it doesn't make sense to specify bucket information, as we should get the bucket information from the existing table. This PR improves the error message for the above 2 cases. ## How was this patch tested? new test in `BukctedWriteSuite` Author: Wenchen Fan <wenchen@databricks.com> Closes #13452 from cloud-fan/error-msg.
Diffstat (limited to 'sql/hive')
-rw-r--r--sql/hive/src/test/scala/org/apache/spark/sql/sources/BucketedWriteSuite.scala19
1 files changed, 15 insertions, 4 deletions
diff --git a/sql/hive/src/test/scala/org/apache/spark/sql/sources/BucketedWriteSuite.scala b/sql/hive/src/test/scala/org/apache/spark/sql/sources/BucketedWriteSuite.scala
index ff44c6f294..61a281db85 100644
--- a/sql/hive/src/test/scala/org/apache/spark/sql/sources/BucketedWriteSuite.scala
+++ b/sql/hive/src/test/scala/org/apache/spark/sql/sources/BucketedWriteSuite.scala
@@ -59,11 +59,22 @@ class BucketedWriteSuite extends QueryTest with SQLTestUtils with TestHiveSingle
intercept[SparkException](df.write.bucketBy(3, "i").format("text").saveAsTable("tt"))
}
- test("write bucketed data to non-hive-table or existing hive table") {
+ test("write bucketed data using save()") {
val df = Seq(1 -> "a", 2 -> "b").toDF("i", "j")
- intercept[IllegalArgumentException](df.write.bucketBy(2, "i").parquet("/tmp/path"))
- intercept[IllegalArgumentException](df.write.bucketBy(2, "i").json("/tmp/path"))
- intercept[IllegalArgumentException](df.write.bucketBy(2, "i").insertInto("tt"))
+
+ val e = intercept[IllegalArgumentException] {
+ df.write.bucketBy(2, "i").parquet("/tmp/path")
+ }
+ assert(e.getMessage == "'save' does not support bucketing right now.")
+ }
+
+ test("write bucketed data using insertInto()") {
+ val df = Seq(1 -> "a", 2 -> "b").toDF("i", "j")
+
+ val e = intercept[IllegalArgumentException] {
+ df.write.bucketBy(2, "i").insertInto("tt")
+ }
+ assert(e.getMessage == "'insertInto' does not support bucketing right now.")
}
private val df = (0 until 50).map(i => (i % 5, i % 13, i.toString)).toDF("i", "j", "k")