diff options
| author | Davies Liu <davies@databricks.com> | 2015-12-21 14:21:43 -0800 |
|---|---|---|
| committer | Davies Liu <davies.liu@gmail.com> | 2015-12-21 14:21:43 -0800 |
| commit | 29cecd4a42f6969613e5b2a40f2724f99e7eec01 (patch) | |
| tree | 261b40272c991649ce584b7cec2056dcd2b6cf1c /sql | |
| parent | d655d37ddf59d7fb6db529324ac8044d53b2622a (diff) | |
| download | spark-29cecd4a42f6969613e5b2a40f2724f99e7eec01.tar.gz spark-29cecd4a42f6969613e5b2a40f2724f99e7eec01.tar.bz2 spark-29cecd4a42f6969613e5b2a40f2724f99e7eec01.zip | |
[SPARK-12388] change default compression to lz4
According the benchmark [1], LZ4-java could be 80% (or 30%) faster than Snappy.
After changing the compressor to LZ4, I saw 20% improvement on end-to-end time for a TPCDS query (Q4).
[1] https://github.com/ning/jvm-compressor-benchmark/wiki
cc rxin
Author: Davies Liu <davies@databricks.com>
Closes #10342 from davies/lz4.
Diffstat (limited to 'sql')
| -rw-r--r-- | sql/core/src/test/scala/org/apache/spark/sql/execution/ExchangeCoordinatorSuite.scala | 4 |
1 files changed, 2 insertions, 2 deletions
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/execution/ExchangeCoordinatorSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/execution/ExchangeCoordinatorSuite.scala index 101cf50d80..2715179e85 100644 --- a/sql/core/src/test/scala/org/apache/spark/sql/execution/ExchangeCoordinatorSuite.scala +++ b/sql/core/src/test/scala/org/apache/spark/sql/execution/ExchangeCoordinatorSuite.scala @@ -319,7 +319,7 @@ class ExchangeCoordinatorSuite extends SparkFunSuite with BeforeAndAfterAll { } } - withSQLContext(test, 1536, minNumPostShufflePartitions) + withSQLContext(test, 2000, minNumPostShufflePartitions) } test(s"determining the number of reducers: join operator$testNameNote") { @@ -422,7 +422,7 @@ class ExchangeCoordinatorSuite extends SparkFunSuite with BeforeAndAfterAll { } } - withSQLContext(test, 6144, minNumPostShufflePartitions) + withSQLContext(test, 6644, minNumPostShufflePartitions) } test(s"determining the number of reducers: complex query 2$testNameNote") { |