diff options
author | Davies Liu <davies@databricks.com> | 2015-08-27 16:38:00 -0700 |
---|---|---|
committer | Michael Armbrust <michael@databricks.com> | 2015-08-27 16:38:00 -0700 |
commit | 54cda0deb6bebf1470f16ba5bcc6c4fb842bdac1 (patch) | |
tree | c16f23655ea73e7403bc885763124c8a299585f5 | |
parent | b3dd569ad40905f8861a547a1e25ed3ca8e1d272 (diff) | |
download | spark-54cda0deb6bebf1470f16ba5bcc6c4fb842bdac1.tar.gz spark-54cda0deb6bebf1470f16ba5bcc6c4fb842bdac1.tar.bz2 spark-54cda0deb6bebf1470f16ba5bcc6c4fb842bdac1.zip |
[SPARK-10321] sizeInBytes in HadoopFsRelation
Having sizeInBytes in HadoopFsRelation to enable broadcast join.
cc marmbrus
Author: Davies Liu <davies@databricks.com>
Closes #8490 from davies/sizeInByte.
-rw-r--r-- | sql/core/src/main/scala/org/apache/spark/sql/sources/interfaces.scala | 2 |
1 files changed, 2 insertions, 0 deletions
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/sources/interfaces.scala b/sql/core/src/main/scala/org/apache/spark/sql/sources/interfaces.scala index dff726b33f..7b030b7d73 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/sources/interfaces.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/sources/interfaces.scala @@ -518,6 +518,8 @@ abstract class HadoopFsRelation private[sql](maybePartitionSpec: Option[Partitio override def inputFiles: Array[String] = cachedLeafStatuses().map(_.getPath.toString).toArray + override def sizeInBytes: Long = cachedLeafStatuses().map(_.getLen).sum + /** * Partition columns. Can be either defined by [[userDefinedPartitionColumns]] or automatically * discovered. Note that they should always be nullable. |