diff options
| author | yzhou2001 <yzhou_1999@yahoo.com> | 2016-05-03 13:41:04 -0700 |
|---|---|---|
| committer | Davies Liu <davies.liu@gmail.com> | 2016-05-03 13:41:04 -0700 |
| commit | a4aed71719b4fc728de93afc623aef05d27bc89a (patch) | |
| tree | 5663ce28289fe28e1484a1eca6c17788170e6886 /core | |
| parent | 659f635d3bd0c0d025bf514dfb1747ed7386ba45 (diff) | |
| download | spark-a4aed71719b4fc728de93afc623aef05d27bc89a.tar.gz spark-a4aed71719b4fc728de93afc623aef05d27bc89a.tar.bz2 spark-a4aed71719b4fc728de93afc623aef05d27bc89a.zip | |
[SPARK-14521] [SQL] StackOverflowError in Kryo when executing TPC-DS
## What changes were proposed in this pull request?
Observed stackOverflowError in Kryo when executing TPC-DS Query27. Spark thrift server disables kryo reference tracking (if not specified in conf). When "spark.kryo.referenceTracking" is set to true explicitly in spark-defaults.conf, query executes successfully. The root cause is that the TaskMemoryManager inside MemoryConsumer and LongToUnsafeRowMap were not transient and thus were serialized and broadcast around from within LongHashedRelation, which could potentially cause circular reference inside Kryo. But the TaskMemoryManager is per task and should not be passed around at the first place. This fix makes it transient.
## How was this patch tested?
core/test, hive/test, sql/test, catalyst/test, dev/lint-scala, org.apache.spark.sql.hive.execution.HiveCompatibilitySuite, dev/scalastyle,
manual test of TBC-DS Query 27 with 1GB data but without the "limit 100" which would cause a NPE due to SPARK-14752.
Author: yzhou2001 <yzhou_1999@yahoo.com>
Closes #12598 from yzhou2001/master.
Diffstat (limited to 'core')
0 files changed, 0 insertions, 0 deletions