diff options
| author | Narine Kokhlikyan <narine.kokhlikyan@gmail.com> | 2016-06-10 17:17:47 -0700 |
|---|---|---|
| committer | Cheng Lian <lian@databricks.com> | 2016-06-10 17:17:47 -0700 |
| commit | 54f758b5fc60ecb0da6b191939a72ef5829be38c (patch) | |
| tree | fce99965ddbd6df03008c302bee5e38f73d5aec9 /sql/catalyst | |
| parent | 2022afe57dbf8cb0c9909399962c4a3649e0601c (diff) | |
| download | spark-54f758b5fc60ecb0da6b191939a72ef5829be38c.tar.gz spark-54f758b5fc60ecb0da6b191939a72ef5829be38c.tar.bz2 spark-54f758b5fc60ecb0da6b191939a72ef5829be38c.zip | |
[SPARK-15884][SPARKR][SQL] Overriding stringArgs in MapPartitionsInR
## What changes were proposed in this pull request?
As discussed in https://github.com/apache/spark/pull/12836
we need to override stringArgs method in MapPartitionsInR in order to avoid too large strings generated by "stringArgs" method based on the input arguments.
In this case exclude some of the input arguments: serialized R objects.
## How was this patch tested?
Existing test cases
Author: Narine Kokhlikyan <narine.kokhlikyan@gmail.com>
Closes #13610 from NarineK/dapply_MapPartitionsInR_stringArgs.
Diffstat (limited to 'sql/catalyst')
| -rw-r--r-- | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/object.scala | 3 |
1 files changed, 3 insertions, 0 deletions
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/object.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/object.scala index 55d8adf040..78e8822b64 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/object.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/object.scala @@ -141,6 +141,9 @@ case class MapPartitionsInR( outputObjAttr: Attribute, child: LogicalPlan) extends ObjectConsumer with ObjectProducer { override lazy val schema = outputSchema + + override protected def stringArgs: Iterator[Any] = Iterator(inputSchema, outputSchema, + outputObjAttr, child) } object MapElements { |