diff options
| author | Davies Liu <davies.liu@gmail.com> | 2014-07-26 01:07:08 -0700 |
|---|---|---|
| committer | Matei Zaharia <matei@databricks.com> | 2014-07-26 01:07:08 -0700 |
| commit | 75663b57f90bb173f0c6c288944ec568c4719b2a (patch) | |
| tree | 295929b4fcaa15d91359e88234ffd47e40487c15 /core | |
| parent | 66f26a4610aede57322cb7e193a50aecb6c57d22 (diff) | |
| download | spark-75663b57f90bb173f0c6c288944ec568c4719b2a.tar.gz spark-75663b57f90bb173f0c6c288944ec568c4719b2a.tar.bz2 spark-75663b57f90bb173f0c6c288944ec568c4719b2a.zip | |
[SPARK-2652] [PySpark] Turning some default configs for PySpark
Add several default configs for PySpark, related to serialization in JVM.
spark.serializer = org.apache.spark.serializer.KryoSerializer
spark.serializer.objectStreamReset = 100
spark.rdd.compress = True
This will help to reduce the memory usage during RDD.partitionBy()
Author: Davies Liu <davies.liu@gmail.com>
Closes #1568 from davies/conf and squashes the following commits:
cd316f1 [Davies Liu] remove duplicated line
f71a355 [Davies Liu] rebase to master, add spark.rdd.compress = True
8f63f45 [Davies Liu] Merge branch 'master' into conf
8bc9f08 [Davies Liu] fix unittest
c04a83d [Davies Liu] some default configs for PySpark
Diffstat (limited to 'core')
0 files changed, 0 insertions, 0 deletions