diff options
| author | Wenchen Fan <wenchen@databricks.com> | 2016-02-09 13:06:36 -0800 |
|---|---|---|
| committer | Davies Liu <davies.liu@gmail.com> | 2016-02-09 13:06:36 -0800 |
| commit | 7fe4fe630a3fc9755ebd0325bb595d76381633e8 (patch) | |
| tree | cbeebfe9ba88887e8e29f7e4b4aeaf73cbab4f8b /core | |
| parent | 2dbb9164405d6f595905c7d4b32e20177f0f669f (diff) | |
| download | spark-7fe4fe630a3fc9755ebd0325bb595d76381633e8.tar.gz spark-7fe4fe630a3fc9755ebd0325bb595d76381633e8.tar.bz2 spark-7fe4fe630a3fc9755ebd0325bb595d76381633e8.zip | |
[SPARK-12888] [SQL] [FOLLOW-UP] benchmark the new hash expression
Adds the benchmark results as comments.
The codegen version is slower than the interpreted version for `simple` case becasue of 3 reasons:
1. codegen version use a more complex hash algorithm than interpreted version, i.e. `Murmur3_x86_32.hashInt` vs [simple multiplication and addition](https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/rows.scala#L153).
2. codegen version will write the hash value to a row first and then read it out. I tried to create a `GenerateHasher` that can generate code to return hash value directly and got about 60% speed up for the `simple` case, does it worth?
3. the row in `simple` case only has one int field, so the runtime reflection may be removed because of branch prediction, which makes the interpreted version faster.
The `array` case is also slow for similar reasons, e.g. array elements are of same type, so interpreted version can probably get rid of runtime reflection by branch prediction.
Author: Wenchen Fan <wenchen@databricks.com>
Closes #10917 from cloud-fan/hash-benchmark.
Diffstat (limited to 'core')
| -rw-r--r-- | core/src/main/scala/org/apache/spark/util/Benchmark.scala | 4 |
1 files changed, 2 insertions, 2 deletions
diff --git a/core/src/main/scala/org/apache/spark/util/Benchmark.scala b/core/src/main/scala/org/apache/spark/util/Benchmark.scala index 1bf6f821e9..39d1829310 100644 --- a/core/src/main/scala/org/apache/spark/util/Benchmark.scala +++ b/core/src/main/scala/org/apache/spark/util/Benchmark.scala @@ -35,7 +35,8 @@ import org.apache.commons.lang3.SystemUtils * If outputPerIteration is true, the timing for each run will be printed to stdout. */ private[spark] class Benchmark( - name: String, valuesPerIteration: Long, + name: String, + valuesPerIteration: Long, iters: Int = 5, outputPerIteration: Boolean = false) { val benchmarks = mutable.ArrayBuffer.empty[Benchmark.Case] @@ -61,7 +62,6 @@ private[spark] class Benchmark( println val firstBest = results.head.bestMs - val firstAvg = results.head.avgMs // The results are going to be processor specific so it is useful to include that. println(Benchmark.getProcessorName()) printf("%-35s %16s %12s %13s %10s\n", name + ":", "Best/Avg Time(ms)", "Rate(M/s)", |