diff options
author | Dongjoon Hyun <dongjoon@apache.org> | 2017-02-03 20:26:53 +0800 |
---|---|---|
committer | Wenchen Fan <wenchen@databricks.com> | 2017-02-03 20:26:53 +0800 |
commit | 52d4f61941b0ac8dcdca04aac042a492141c2068 (patch) | |
tree | c9591a0100b111710b54edd284207c8e76f57ae6 | |
parent | 20b4ca1402f921102a4fa47c68ebe1d7a2cff0a0 (diff) | |
download | spark-52d4f61941b0ac8dcdca04aac042a492141c2068.tar.gz spark-52d4f61941b0ac8dcdca04aac042a492141c2068.tar.bz2 spark-52d4f61941b0ac8dcdca04aac042a492141c2068.zip |
[SPARK-18909][SQL] The error messages in `ExpressionEncoder.toRow/fromRow` are too verbose
## What changes were proposed in this pull request?
In `ExpressionEncoder.toRow` and `fromRow`, we catch the exception and output `treeString` of serializer/deserializer expressions in the error message. However, encoder can be very complex and the serializer/deserializer expressions can be very large trees and blow up the log files(e.g. generate over 500mb logs for this single error message.) As a first attempt, this PR try to use `simpleString` instead.
**BEFORE**
```scala
scala> :paste
// Entering paste mode (ctrl-D to finish)
case class TestCaseClass(value: Int)
import spark.implicits._
Seq(TestCaseClass(1)).toDS().collect()
// Exiting paste mode, now interpreting.
java.lang.RuntimeException: Error while decoding: java.lang.NullPointerException
newInstance(class TestCaseClass)
+- assertnotnull(input[0, int, false], - field (class: "scala.Int", name: "value"), - root class: "TestCaseClass")
+- input[0, int, false]
at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.fromRow(ExpressionEncoder.scala:303)
...
```
**AFTER**
```scala
...
// Exiting paste mode, now interpreting.
java.lang.RuntimeException: Error while decoding: java.lang.NullPointerException
newInstance(class TestCaseClass)
at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.fromRow(ExpressionEncoder.scala:303)
...
```
## How was this patch tested?
Manual.
Author: Dongjoon Hyun <dongjoon@apache.org>
Closes #16701 from dongjoon-hyun/SPARK-18909-EXPR-ERROR.
-rw-r--r-- | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/encoders/ExpressionEncoder.scala | 4 |
1 files changed, 2 insertions, 2 deletions
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/encoders/ExpressionEncoder.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/encoders/ExpressionEncoder.scala index 3757eccfa2..0782143d46 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/encoders/ExpressionEncoder.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/encoders/ExpressionEncoder.scala @@ -288,7 +288,7 @@ case class ExpressionEncoder[T]( } catch { case e: Exception => throw new RuntimeException( - s"Error while encoding: $e\n${serializer.map(_.treeString).mkString("\n")}", e) + s"Error while encoding: $e\n${serializer.map(_.simpleString).mkString("\n")}", e) } /** @@ -300,7 +300,7 @@ case class ExpressionEncoder[T]( constructProjection(row).get(0, ObjectType(clsTag.runtimeClass)).asInstanceOf[T] } catch { case e: Exception => - throw new RuntimeException(s"Error while decoding: $e\n${deserializer.treeString}", e) + throw new RuntimeException(s"Error while decoding: $e\n${deserializer.simpleString}", e) } /** |