aboutsummaryrefslogtreecommitdiff
path: root/core/src/main
diff options
context:
space:
mode:
authorDongjoon Hyun <dongjoon@apache.org>2017-02-03 20:26:53 +0800
committerWenchen Fan <wenchen@databricks.com>2017-02-03 20:26:53 +0800
commit52d4f61941b0ac8dcdca04aac042a492141c2068 (patch)
treec9591a0100b111710b54edd284207c8e76f57ae6 /core/src/main
parent20b4ca1402f921102a4fa47c68ebe1d7a2cff0a0 (diff)
downloadspark-52d4f61941b0ac8dcdca04aac042a492141c2068.tar.gz
spark-52d4f61941b0ac8dcdca04aac042a492141c2068.tar.bz2
spark-52d4f61941b0ac8dcdca04aac042a492141c2068.zip
[SPARK-18909][SQL] The error messages in `ExpressionEncoder.toRow/fromRow` are too verbose
## What changes were proposed in this pull request? In `ExpressionEncoder.toRow` and `fromRow`, we catch the exception and output `treeString` of serializer/deserializer expressions in the error message. However, encoder can be very complex and the serializer/deserializer expressions can be very large trees and blow up the log files(e.g. generate over 500mb logs for this single error message.) As a first attempt, this PR try to use `simpleString` instead. **BEFORE** ```scala scala> :paste // Entering paste mode (ctrl-D to finish) case class TestCaseClass(value: Int) import spark.implicits._ Seq(TestCaseClass(1)).toDS().collect() // Exiting paste mode, now interpreting. java.lang.RuntimeException: Error while decoding: java.lang.NullPointerException newInstance(class TestCaseClass) +- assertnotnull(input[0, int, false], - field (class: "scala.Int", name: "value"), - root class: "TestCaseClass") +- input[0, int, false] at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.fromRow(ExpressionEncoder.scala:303) ... ``` **AFTER** ```scala ... // Exiting paste mode, now interpreting. java.lang.RuntimeException: Error while decoding: java.lang.NullPointerException newInstance(class TestCaseClass) at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.fromRow(ExpressionEncoder.scala:303) ... ``` ## How was this patch tested? Manual. Author: Dongjoon Hyun <dongjoon@apache.org> Closes #16701 from dongjoon-hyun/SPARK-18909-EXPR-ERROR.
Diffstat (limited to 'core/src/main')
0 files changed, 0 insertions, 0 deletions