diff options
| author | Ilya Ganelin <ilya.ganelin@capitalone.com> | 2015-04-17 18:28:42 -0700 |
|---|---|---|
| committer | Andrew Or <andrew@databricks.com> | 2015-04-17 18:28:42 -0700 |
| commit | c5ed510135aee3a1a0402057b3b5229892aa6f3a (patch) | |
| tree | b1daa6fe6fb42c924059c9442e10b8c4595d148b /project | |
| parent | a452c59210cf2c8ff8601cdb11401eea6dc14973 (diff) | |
| download | spark-c5ed510135aee3a1a0402057b3b5229892aa6f3a.tar.gz spark-c5ed510135aee3a1a0402057b3b5229892aa6f3a.tar.bz2 spark-c5ed510135aee3a1a0402057b3b5229892aa6f3a.zip | |
[SPARK-6703][Core] Provide a way to discover existing SparkContext's
I've added a static getOrCreate method to the static SparkContext object that allows one to either retrieve a previously created SparkContext or to instantiate a new one with the provided config. The method accepts an optional SparkConf to make usage intuitive.
Still working on a test for this, basically want to create a new context from scratch, then ensure that subsequent calls don't overwrite that.
Author: Ilya Ganelin <ilya.ganelin@capitalone.com>
Closes #5501 from ilganeli/SPARK-6703 and squashes the following commits:
db9a963 [Ilya Ganelin] Closing second spark context
1dc0444 [Ilya Ganelin] Added ref equality check
8c884fa [Ilya Ganelin] Made getOrCreate synchronized
cb0c6b7 [Ilya Ganelin] Doc updates and code cleanup
270cfe3 [Ilya Ganelin] [SPARK-6703] Documentation fixes
15e8dea [Ilya Ganelin] Updated comments and added MiMa Exclude
0e1567c [Ilya Ganelin] Got rid of unecessary option for AtomicReference
dfec4da [Ilya Ganelin] Changed activeContext to AtomicReference
733ec9f [Ilya Ganelin] Fixed some bugs in test code
8be2f83 [Ilya Ganelin] Replaced match with if
e92caf7 [Ilya Ganelin] [SPARK-6703] Added test to ensure that getOrCreate both allows creation, retrieval, and a second context if desired
a99032f [Ilya Ganelin] Spacing fix
d7a06b8 [Ilya Ganelin] Updated SparkConf class to add getOrCreate method. Started test suite implementation
Diffstat (limited to 'project')
| -rw-r--r-- | project/MimaExcludes.scala | 4 |
1 files changed, 4 insertions, 0 deletions
diff --git a/project/MimaExcludes.scala b/project/MimaExcludes.scala index 1564babefa..7ef363a2f0 100644 --- a/project/MimaExcludes.scala +++ b/project/MimaExcludes.scala @@ -68,6 +68,10 @@ object MimaExcludes { // SPARK-6693 add tostring with max lines and width for matrix ProblemFilters.exclude[MissingMethodProblem]( "org.apache.spark.mllib.linalg.Matrix.toString") + )++ Seq( + // SPARK-6703 Add getOrCreate method to SparkContext + ProblemFilters.exclude[IncompatibleResultTypeProblem] + ("org.apache.spark.SparkContext.org$apache$spark$SparkContext$$activeContext") ) case v if v.startsWith("1.3") => |