diff options
| author | Cheng Lian <lian@databricks.com> | 2014-12-05 10:27:40 -0800 |
|---|---|---|
| committer | Patrick Wendell <pwendell@gmail.com> | 2014-12-05 10:27:55 -0800 |
| commit | e8d8077bfc3e667a61dc261d2bee80d2a9f1eed3 (patch) | |
| tree | cd320e70b66f1e9d096927e5b7c3aab0713fa80b /sql | |
| parent | d12ea49f56e9ffa9576a94cda99c066910c1425d (diff) | |
| download | spark-e8d8077bfc3e667a61dc261d2bee80d2a9f1eed3.tar.gz spark-e8d8077bfc3e667a61dc261d2bee80d2a9f1eed3.tar.bz2 spark-e8d8077bfc3e667a61dc261d2bee80d2a9f1eed3.zip | |
[SPARK-4761][SQL] Enables Kryo by default in Spark SQL Thrift server
Enables Kryo and disables reference tracking by default in Spark SQL Thrift server. Configurations explicitly defined by users in `spark-defaults.conf` are respected (the Thrift server is started by `spark-submit`, which handles configuration properties properly).
<!-- Reviewable:start -->
[<img src="https://reviewable.io/review_button.png" height=40 alt="Review on Reviewable"/>](https://reviewable.io/reviews/apache/spark/3621)
<!-- Reviewable:end -->
Author: Cheng Lian <lian@databricks.com>
Closes #3621 from liancheng/kryo-by-default and squashes the following commits:
70c2775 [Cheng Lian] Enables Kryo by default in Spark SQL Thrift server
(cherry picked from commit 6f61e1f961826a6c9e98a66d10b271b7e3c7dd55)
Signed-off-by: Patrick Wendell <pwendell@gmail.com>
Diffstat (limited to 'sql')
| -rw-r--r-- | sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkSQLEnv.scala | 14 |
1 files changed, 12 insertions, 2 deletions
diff --git a/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkSQLEnv.scala b/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkSQLEnv.scala index 89732c939b..158c225159 100644 --- a/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkSQLEnv.scala +++ b/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkSQLEnv.scala @@ -32,11 +32,21 @@ private[hive] object SparkSQLEnv extends Logging { def init() { if (hiveContext == null) { - val sparkConf = new SparkConf() + val sparkConf = new SparkConf(loadDefaults = true) + val maybeSerializer = sparkConf.getOption("spark.serializer") + val maybeKryoReferenceTracking = sparkConf.getOption("spark.kryo.referenceTracking") + + sparkConf .setAppName(s"SparkSQL::${java.net.InetAddress.getLocalHost.getHostName}") .set("spark.sql.hive.version", HiveShim.version) - sparkContext = new SparkContext(sparkConf) + .set( + "spark.serializer", + maybeSerializer.getOrElse("org.apache.spark.serializer.KryoSerializer")) + .set( + "spark.kryo.referenceTracking", + maybeKryoReferenceTracking.getOrElse("false")) + sparkContext = new SparkContext(sparkConf) sparkContext.addSparkListener(new StatsReportListener()) hiveContext = new HiveContext(sparkContext) |