aboutsummaryrefslogtreecommitdiff
path: root/docs/programming-guide.md
diff options
context:
space:
mode:
authorReynold Xin <rxin@databricks.com>2015-08-18 22:08:15 -0700
committerReynold Xin <rxin@databricks.com>2015-08-18 22:08:15 -0700
commit1ff0580eda90f9247a5233809667f5cebaea290e (patch)
tree29d85b57bf0870f316f510c9823f5805fe31dc2a /docs/programming-guide.md
parent270ee677750a1f2adaf24b5816857194e61782ff (diff)
downloadspark-1ff0580eda90f9247a5233809667f5cebaea290e.tar.gz
spark-1ff0580eda90f9247a5233809667f5cebaea290e.tar.bz2
spark-1ff0580eda90f9247a5233809667f5cebaea290e.zip
[SPARK-10093] [SPARK-10096] [SQL] Avoid transformation on executors & fix UDFs on complex types
This is kind of a weird case, but given a sufficiently complex query plan (in this case a TungstenProject with an Exchange underneath), we could have NPEs on the executors due to the time when we were calling transformAllExpressions In general we should ensure that all transformations occur on the driver and not on the executors. Some reasons for avoid executor side transformations include: * (this case) Some operator constructors require state such as access to the Spark/SQL conf so doing a makeCopy on the executor can fail. * (unrelated reason for avoid executor transformations) ExprIds are calculated using an atomic integer, so you can violate their uniqueness constraint by constructing them anywhere other than the driver. This subsumes #8285. Author: Reynold Xin <rxin@databricks.com> Author: Michael Armbrust <michael@databricks.com> Closes #8295 from rxin/SPARK-10096.
Diffstat (limited to 'docs/programming-guide.md')
0 files changed, 0 insertions, 0 deletions