diff options
| author | Hossein <hossein@databricks.com> | 2014-07-26 01:04:56 -0700 |
|---|---|---|
| committer | Matei Zaharia <matei@databricks.com> | 2014-07-26 01:04:56 -0700 |
| commit | 66f26a4610aede57322cb7e193a50aecb6c57d22 (patch) | |
| tree | e45cb6dbf2a6970f0b8b341a0384352b2106122d /core | |
| parent | cf3e9fd84dc64f8a57ecbcfdd6b22f5492d41bd7 (diff) | |
| download | spark-66f26a4610aede57322cb7e193a50aecb6c57d22.tar.gz spark-66f26a4610aede57322cb7e193a50aecb6c57d22.tar.bz2 spark-66f26a4610aede57322cb7e193a50aecb6c57d22.zip | |
[SPARK-2696] Reduce default value of spark.serializer.objectStreamReset
The current default value of spark.serializer.objectStreamReset is 10,000.
When trying to re-partition (e.g., to 64 partitions) a large file (e.g., 500MB), containing 1MB records, the serializer will cache 10000 x 1MB x 64 ~= 640 GB which will cause out of memory errors.
This patch sets the default value to a more reasonable default value (100).
Author: Hossein <hossein@databricks.com>
Closes #1595 from falaki/objectStreamReset and squashes the following commits:
650a935 [Hossein] Updated documentation
1aa0df8 [Hossein] Reduce default value of spark.serializer.objectStreamReset
Diffstat (limited to 'core')
| -rw-r--r-- | core/src/main/scala/org/apache/spark/serializer/JavaSerializer.scala | 2 |
1 files changed, 1 insertions, 1 deletions
diff --git a/core/src/main/scala/org/apache/spark/serializer/JavaSerializer.scala b/core/src/main/scala/org/apache/spark/serializer/JavaSerializer.scala index 0a7e1ec539..a7fa057ee0 100644 --- a/core/src/main/scala/org/apache/spark/serializer/JavaSerializer.scala +++ b/core/src/main/scala/org/apache/spark/serializer/JavaSerializer.scala @@ -108,7 +108,7 @@ private[spark] class JavaSerializerInstance(counterReset: Int) extends Serialize */ @DeveloperApi class JavaSerializer(conf: SparkConf) extends Serializer with Externalizable { - private var counterReset = conf.getInt("spark.serializer.objectStreamReset", 10000) + private var counterReset = conf.getInt("spark.serializer.objectStreamReset", 100) def newInstance(): SerializerInstance = new JavaSerializerInstance(counterReset) |