diff options
| author | Davies Liu <davies@databricks.com> | 2015-08-27 16:38:00 -0700 |
|---|---|---|
| committer | Michael Armbrust <michael@databricks.com> | 2015-08-27 16:38:10 -0700 |
| commit | fc4c3bf43626ecce75a909d9d0f1acd973f75fbf (patch) | |
| tree | 6eec7a736781c78fda8c7858d869bae4f0599ff3 | |
| parent | 351e849bbeaeee9dcf95d465ada1270a059da2f1 (diff) | |
| download | spark-fc4c3bf43626ecce75a909d9d0f1acd973f75fbf.tar.gz spark-fc4c3bf43626ecce75a909d9d0f1acd973f75fbf.tar.bz2 spark-fc4c3bf43626ecce75a909d9d0f1acd973f75fbf.zip | |
[SPARK-10321] sizeInBytes in HadoopFsRelation
Having sizeInBytes in HadoopFsRelation to enable broadcast join.
cc marmbrus
Author: Davies Liu <davies@databricks.com>
Closes #8490 from davies/sizeInByte.
(cherry picked from commit 54cda0deb6bebf1470f16ba5bcc6c4fb842bdac1)
Signed-off-by: Michael Armbrust <michael@databricks.com>
| -rw-r--r-- | sql/core/src/main/scala/org/apache/spark/sql/sources/interfaces.scala | 2 |
1 files changed, 2 insertions, 0 deletions
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/sources/interfaces.scala b/sql/core/src/main/scala/org/apache/spark/sql/sources/interfaces.scala index dff726b33f..7b030b7d73 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/sources/interfaces.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/sources/interfaces.scala @@ -518,6 +518,8 @@ abstract class HadoopFsRelation private[sql](maybePartitionSpec: Option[Partitio override def inputFiles: Array[String] = cachedLeafStatuses().map(_.getPath.toString).toArray + override def sizeInBytes: Long = cachedLeafStatuses().map(_.getLen).sum + /** * Partition columns. Can be either defined by [[userDefinedPartitionColumns]] or automatically * discovered. Note that they should always be nullable. |