aboutsummaryrefslogtreecommitdiff
diff options
context:
space:
mode:
authorSean Owen <sowen@cloudera.com>2016-09-19 09:38:25 +0100
committerSean Owen <sowen@cloudera.com>2016-09-19 09:38:25 +0100
commitd720a4019460b6c284d0473249303c349df60a1f (patch)
tree258b33aa1cf4f5fcde93903997bb749aec022855
parent8f0c35a4d0dd458719627be5f524792bf244d70a (diff)
downloadspark-d720a4019460b6c284d0473249303c349df60a1f.tar.gz
spark-d720a4019460b6c284d0473249303c349df60a1f.tar.bz2
spark-d720a4019460b6c284d0473249303c349df60a1f.zip
[SPARK-17297][DOCS] Clarify window/slide duration as absolute time, not relative to a calendar
## What changes were proposed in this pull request? Clarify that slide and window duration are absolute, and not relative to a calendar. ## How was this patch tested? Doc build (no functional change) Author: Sean Owen <sowen@cloudera.com> Closes #15142 from srowen/SPARK-17297.
-rw-r--r--R/pkg/R/functions.R8
-rw-r--r--sql/core/src/main/scala/org/apache/spark/sql/functions.scala15
2 files changed, 17 insertions, 6 deletions
diff --git a/R/pkg/R/functions.R b/R/pkg/R/functions.R
index ceedbe7671..4d94b4cd05 100644
--- a/R/pkg/R/functions.R
+++ b/R/pkg/R/functions.R
@@ -2713,11 +2713,15 @@ setMethod("from_unixtime", signature(x = "Column"),
#' @param x a time Column. Must be of TimestampType.
#' @param windowDuration a string specifying the width of the window, e.g. '1 second',
#' '1 day 12 hours', '2 minutes'. Valid interval strings are 'week',
-#' 'day', 'hour', 'minute', 'second', 'millisecond', 'microsecond'.
+#' 'day', 'hour', 'minute', 'second', 'millisecond', 'microsecond'. Note that
+#' the duration is a fixed length of time, and does not vary over time
+#' according to a calendar. For example, '1 day' always means 86,400,000
+#' milliseconds, not a calendar day.
#' @param slideDuration a string specifying the sliding interval of the window. Same format as
#' \code{windowDuration}. A new window will be generated every
#' \code{slideDuration}. Must be less than or equal to
-#' the \code{windowDuration}.
+#' the \code{windowDuration}. This duration is likewise absolute, and does not
+#' vary according to a calendar.
#' @param startTime the offset with respect to 1970-01-01 00:00:00 UTC with which to start
#' window intervals. For example, in order to have hourly tumbling windows
#' that start 15 minutes past the hour, e.g. 12:15-13:15, 13:15-14:15... provide
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/functions.scala b/sql/core/src/main/scala/org/apache/spark/sql/functions.scala
index 18e736ab69..960c87f60e 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/functions.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/functions.scala
@@ -2606,12 +2606,15 @@ object functions {
* The time column must be of TimestampType.
* @param windowDuration A string specifying the width of the window, e.g. `10 minutes`,
* `1 second`. Check [[org.apache.spark.unsafe.types.CalendarInterval]] for
- * valid duration identifiers.
+ * valid duration identifiers. Note that the duration is a fixed length of
+ * time, and does not vary over time according to a calendar. For example,
+ * `1 day` always means 86,400,000 milliseconds, not a calendar day.
* @param slideDuration A string specifying the sliding interval of the window, e.g. `1 minute`.
* A new window will be generated every `slideDuration`. Must be less than
* or equal to the `windowDuration`. Check
* [[org.apache.spark.unsafe.types.CalendarInterval]] for valid duration
- * identifiers.
+ * identifiers. This duration is likewise absolute, and does not vary
+ * according to a calendar.
* @param startTime The offset with respect to 1970-01-01 00:00:00 UTC with which to start
* window intervals. For example, in order to have hourly tumbling windows that
* start 15 minutes past the hour, e.g. 12:15-13:15, 13:15-14:15... provide
@@ -2660,11 +2663,15 @@ object functions {
* The time column must be of TimestampType.
* @param windowDuration A string specifying the width of the window, e.g. `10 minutes`,
* `1 second`. Check [[org.apache.spark.unsafe.types.CalendarInterval]] for
- * valid duration identifiers.
+ * valid duration identifiers. Note that the duration is a fixed length of
+ * time, and does not vary over time according to a calendar. For example,
+ * `1 day` always means 86,400,000 milliseconds, not a calendar day.
* @param slideDuration A string specifying the sliding interval of the window, e.g. `1 minute`.
* A new window will be generated every `slideDuration`. Must be less than
* or equal to the `windowDuration`. Check
- * [[org.apache.spark.unsafe.types.CalendarInterval]] for valid duration.
+ * [[org.apache.spark.unsafe.types.CalendarInterval]] for valid duration
+ * identifiers. This duration is likewise absolute, and does not vary
+ * according to a calendar.
*
* @group datetime_funcs
* @since 2.0.0