diff options
| author | Oleksii Kostyliev <etander@gmail.com> | 2015-05-15 11:19:56 -0700 |
|---|---|---|
| committer | Andrew Or <andrew@databricks.com> | 2015-05-15 11:20:19 -0700 |
| commit | b1b9d5802e3d185f42711ab043a21c9d1eb4763f (patch) | |
| tree | fd226968c5bc6b7a6ab85f821562cadcce6566f7 /sql | |
| parent | 8f4aaba0e4e3350ab152a476d08ff60e9495c6d2 (diff) | |
| download | spark-b1b9d5802e3d185f42711ab043a21c9d1eb4763f.tar.gz spark-b1b9d5802e3d185f42711ab043a21c9d1eb4763f.tar.bz2 spark-b1b9d5802e3d185f42711ab043a21c9d1eb4763f.zip | |
[SPARK-7233] [CORE] Detect REPL mode once
<h3>Description</h3>
Detect REPL mode once per JVM lifespan.
Previous behavior was to check presence of interpreter mode every time a job was submitted. In the case of execution of multiple short-living jobs this was causing massive mutual blocks between submission threads.
For more details please refer to https://issues.apache.org/jira/browse/SPARK-7233.
<h3>Notes</h3>
* I inverted the return value in case of catching an exception from `true` to `false`. It seems more logical to assume that if the REPL class is not found, we aren't in the interpreter mode.
* I'd personally would call `classForName` with just a Spark classloader (`org.apache.spark.util.Utils#getSparkClassLoader`) but `org.apache.spark.util.Utils#getContextOrSparkClassLoader` is said to be preferable.
* I struggled to come up with a concise, readable and clear unit test. Suggestions are welcome if you feel necessary.
Author: Oleksii Kostyliev <etander@gmail.com>
Author: Oleksii Kostyliev <okostyliev@thunderhead.com>
Closes #5835 from preeze/SPARK-7233 and squashes the following commits:
69bb9e4 [Oleksii Kostyliev] SPARK-7527: fixed explanatory comment to meet style-checker requirements
26dcc24 [Oleksii Kostyliev] SPARK-7527: fixed explanatory comment to meet style-checker requirements
c6f9685 [Oleksii Kostyliev] Merge remote-tracking branch 'remotes/upstream/master' into SPARK-7233
b78a983 [Oleksii Kostyliev] SPARK-7527: revert the fix and let it be addressed separately at a later stage
b64d441 [Oleksii Kostyliev] SPARK-7233: inline inInterpreter parameter into instantiateClass
86e2606 [Oleksii Kostyliev] SPARK-7233, SPARK-7527: Handle interpreter mode properly.
c7ee69c [Oleksii Kostyliev] Merge remote-tracking branch 'upstream/master' into SPARK-7233
d6c07fc [Oleksii Kostyliev] SPARK-7233: properly handle the inverted meaning of isInInterpreter
c319039 [Oleksii Kostyliev] SPARK-7233: move inInterpreter to Utils and make it lazy
Diffstat (limited to 'sql')
0 files changed, 0 insertions, 0 deletions