diff options
| author | Tejas Patil <tejasp@fb.com> | 2017-02-15 22:45:58 -0800 |
|---|---|---|
| committer | Wenchen Fan <wenchen@databricks.com> | 2017-02-15 22:45:58 -0800 |
| commit | f041e55eefe1d8a995fed321c66bccbd8b8e5255 (patch) | |
| tree | b82f5402f3a5abf38127322be8f5e5657654f7d7 /sql/catalyst/src | |
| parent | 8487902a98caf727ba3f9820452b01276d20ede3 (diff) | |
| download | spark-f041e55eefe1d8a995fed321c66bccbd8b8e5255.tar.gz spark-f041e55eefe1d8a995fed321c66bccbd8b8e5255.tar.bz2 spark-f041e55eefe1d8a995fed321c66bccbd8b8e5255.zip | |
[SPARK-19618][SQL] Inconsistency wrt max. buckets allowed from Dataframe API vs SQL
## What changes were proposed in this pull request?
Jira: https://issues.apache.org/jira/browse/SPARK-19618
Moved the check for validating number of buckets from `DataFrameWriter` to `BucketSpec` creation
## How was this patch tested?
- Added more unit tests
Author: Tejas Patil <tejasp@fb.com>
Closes #16948 from tejasapatil/SPARK-19618_max_buckets.
Diffstat (limited to 'sql/catalyst/src')
| -rw-r--r-- | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala | 5 |
1 files changed, 3 insertions, 2 deletions
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala index 353e5954fd..2b3b575b4c 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala @@ -135,8 +135,9 @@ case class BucketSpec( numBuckets: Int, bucketColumnNames: Seq[String], sortColumnNames: Seq[String]) { - if (numBuckets <= 0) { - throw new AnalysisException(s"Expected positive number of buckets, but got `$numBuckets`.") + if (numBuckets <= 0 || numBuckets >= 100000) { + throw new AnalysisException( + s"Number of buckets should be greater than 0 but less than 100000. Got `$numBuckets`") } override def toString: String = { |