aboutsummaryrefslogtreecommitdiff
path: root/sbin/spark-daemon.sh
diff options
context:
space:
mode:
authorMichael Armbrust <michael@databricks.com>2015-08-05 09:01:45 -0700
committerYin Huai <yhuai@databricks.com>2015-08-05 09:01:45 -0700
commit23d982204bb9ef74d3b788a32ce6608116968719 (patch)
tree4faef48a8551c8da604463613c2c1b6fc9762d30 /sbin/spark-daemon.sh
parent34dcf10104460816382908b2b8eeb6c925e862bf (diff)
downloadspark-23d982204bb9ef74d3b788a32ce6608116968719.tar.gz
spark-23d982204bb9ef74d3b788a32ce6608116968719.tar.bz2
spark-23d982204bb9ef74d3b788a32ce6608116968719.zip
[SPARK-9141] [SQL] Remove project collapsing from DataFrame API
Currently we collapse successive projections that are added by `withColumn`. However, this optimization violates the constraint that adding nodes to a plan will never change its analyzed form and thus breaks caching. Instead of doing early optimization, in this PR I just fix some low-hanging slowness in the analyzer. In particular, I add a mechanism for skipping already analyzed subplans, `resolveOperators` and `resolveExpression`. Since trees are generally immutable after construction, it's safe to annotate a plan as already analyzed as any transformation will create a new tree with this bit no longer set. Together these result in a faster analyzer than before, even with added timing instrumentation. ``` Original Code [info] 3430ms [info] 2205ms [info] 1973ms [info] 1982ms [info] 1916ms Without Project Collapsing in DataFrame [info] 44610ms [info] 45977ms [info] 46423ms [info] 46306ms [info] 54723ms With analyzer optimizations [info] 6394ms [info] 4630ms [info] 4388ms [info] 4093ms [info] 4113ms With resolveOperators [info] 2495ms [info] 1380ms [info] 1685ms [info] 1414ms [info] 1240ms ``` Author: Michael Armbrust <michael@databricks.com> Closes #7920 from marmbrus/withColumnCache and squashes the following commits: 2145031 [Michael Armbrust] fix hive udfs tests 5a5a525 [Michael Armbrust] remove wrong comment 7a507d5 [Michael Armbrust] style b59d710 [Michael Armbrust] revert small change 1fa5949 [Michael Armbrust] move logic into LogicalPlan, add tests 0e2cb43 [Michael Armbrust] Merge remote-tracking branch 'origin/master' into withColumnCache c926e24 [Michael Armbrust] naming e593a2d [Michael Armbrust] style f5a929e [Michael Armbrust] [SPARK-9141][SQL] Remove project collapsing from DataFrame API 38b1c83 [Michael Armbrust] WIP
Diffstat (limited to 'sbin/spark-daemon.sh')
0 files changed, 0 insertions, 0 deletions