diff options
| author | Yin Huai <yhuai@databricks.com> | 2016-06-28 07:54:44 -0700 |
|---|---|---|
| committer | Davies Liu <davies.liu@gmail.com> | 2016-06-28 07:54:44 -0700 |
| commit | 0923c4f5676691e28e70ecb05890e123540b91f0 (patch) | |
| tree | fc4b3fff3d5ab0f07080ec90cabeae9786bae147 /python/pyspark/context.py | |
| parent | e158478a9fff5e63ae0336a54b3f360d0cd38921 (diff) | |
| download | spark-0923c4f5676691e28e70ecb05890e123540b91f0.tar.gz spark-0923c4f5676691e28e70ecb05890e123540b91f0.tar.bz2 spark-0923c4f5676691e28e70ecb05890e123540b91f0.zip | |
[SPARK-16224] [SQL] [PYSPARK] SparkSession builder's configs need to be set to the existing Scala SparkContext's SparkConf
## What changes were proposed in this pull request?
When we create a SparkSession at the Python side, it is possible that a SparkContext has been created. For this case, we need to set configs of the SparkSession builder to the Scala SparkContext's SparkConf (we need to do so because conf changes on a active Python SparkContext will not be propagated to the JVM side). Otherwise, we may create a wrong SparkSession (e.g. Hive support is not enabled even if enableHiveSupport is called).
## How was this patch tested?
New tests and manual tests.
Author: Yin Huai <yhuai@databricks.com>
Closes #13931 from yhuai/SPARK-16224.
Diffstat (limited to 'python/pyspark/context.py')
| -rw-r--r-- | python/pyspark/context.py | 2 |
1 files changed, 2 insertions, 0 deletions
diff --git a/python/pyspark/context.py b/python/pyspark/context.py index 7217a9907a..6e9f24ef10 100644 --- a/python/pyspark/context.py +++ b/python/pyspark/context.py @@ -166,6 +166,8 @@ class SparkContext(object): # Create the Java SparkContext through Py4J self._jsc = jsc or self._initialize_context(self._conf._jconf) + # Reset the SparkConf to the one actually used by the SparkContext in JVM. + self._conf = SparkConf(_jconf=self._jsc.sc().conf()) # Create a single Accumulator in Java that we'll send all our updates through; # they will be passed back to us through a TCP server |