diff options
author | Jacek Lewandowski <lewandowski.jacek@gmail.com> | 2015-02-07 19:16:07 -0800 |
---|---|---|
committer | Josh Rosen <joshrosen@databricks.com> | 2015-02-07 19:16:07 -0800 |
commit | 4bad85485de8d4afc8f0446ef300f217f9e6b6b5 (patch) | |
tree | 468c984389bc67b2cc9dc6e00e0ca63333ee4941 /examples | |
parent | d89964f86e288dbdd67e08f4eb97f374c4d437f3 (diff) | |
download | spark-4bad85485de8d4afc8f0446ef300f217f9e6b6b5.tar.gz spark-4bad85485de8d4afc8f0446ef300f217f9e6b6b5.tar.bz2 spark-4bad85485de8d4afc8f0446ef300f217f9e6b6b5.zip |
SPARK-5425: Use synchronised methods in system properties to create SparkConf
SPARK-5425: Fixed usages of system properties
This patch fixes few problems caused by the fact that the Scala wrapper over system properties is not thread-safe and is basically invalid because it doesn't take into account the default values which could have been set in the properties object. The problem is fixed by modifying `Utils.getSystemProperties` method so that it uses `stringPropertyNames` method of the `Properties` class, which is thread-safe (internally it creates a defensive copy in a synchronized method) and returns keys of the properties which were set explicitly and which are defined as defaults.
The other related problem, which is fixed here. was in `ResetSystemProperties` mix-in. It created a copy of the system properties in the wrong way.
This patch also introduces a test case for thread-safeness of SparkConf creation.
Refer to the discussion in https://github.com/apache/spark/pull/4220 for more details.
Author: Jacek Lewandowski <lewandowski.jacek@gmail.com>
Closes #4221 from jacek-lewandowski/SPARK-5425-1.2 and squashes the following commits:
87951a2 [Jacek Lewandowski] SPARK-5425: Modified Utils.getSystemProperties to return a map of all system properties - explicit + defaults
01dd5cb [Jacek Lewandowski] SPARK-5425: Use SerializationUtils to save properties in ResetSystemProperties trait
94aeacf [Jacek Lewandowski] SPARK-5425: Use synchronised methods in system properties to create SparkConf
Diffstat (limited to 'examples')
-rw-r--r-- | examples/src/main/scala/org/apache/spark/examples/DriverSubmissionTest.scala | 4 |
1 files changed, 3 insertions, 1 deletions
diff --git a/examples/src/main/scala/org/apache/spark/examples/DriverSubmissionTest.scala b/examples/src/main/scala/org/apache/spark/examples/DriverSubmissionTest.scala index 65251e9319..e757283823 100644 --- a/examples/src/main/scala/org/apache/spark/examples/DriverSubmissionTest.scala +++ b/examples/src/main/scala/org/apache/spark/examples/DriverSubmissionTest.scala @@ -19,6 +19,8 @@ package org.apache.spark.examples import scala.collection.JavaConversions._ +import org.apache.spark.util.Utils + /** Prints out environmental information, sleeps, and then exits. Made to * test driver submission in the standalone scheduler. */ object DriverSubmissionTest { @@ -30,7 +32,7 @@ object DriverSubmissionTest { val numSecondsToSleep = args(0).toInt val env = System.getenv() - val properties = System.getProperties() + val properties = Utils.getSystemProperties println("Environment variables containing SPARK_TEST:") env.filter{case (k, v) => k.contains("SPARK_TEST")}.foreach(println) |