diff options
| author | Sean Owen <sowen@cloudera.com> | 2014-04-11 22:46:47 -0700 |
|---|---|---|
| committer | Patrick Wendell <pwendell@gmail.com> | 2014-04-11 22:46:47 -0700 |
| commit | 165e06a74c3d75e6b7341c120943add8b035b96a (patch) | |
| tree | 407aa7d706a64eb45f7e4d02f3f6db640b130b28 /project | |
| parent | aa8bb117a3ff98420ab751ba4ddbaad88ab57f9d (diff) | |
| download | spark-165e06a74c3d75e6b7341c120943add8b035b96a.tar.gz spark-165e06a74c3d75e6b7341c120943add8b035b96a.tar.bz2 spark-165e06a74c3d75e6b7341c120943add8b035b96a.zip | |
SPARK-1057 (alternative) Remove fastutil
(This is for discussion at this point -- I'm not suggesting this should be committed.)
This is what removing fastutil looks like. Much of it is straightforward, like using `java.io` buffered stream classes, and Guava for murmurhash3.
Uses of the `FastByteArrayOutputStream` were a little trickier. In only one case though do I think the change to use `java.io` actually entails an extra array copy.
The rest is using `OpenHashMap` and `OpenHashSet`. These are now written in terms of more scala-like operations.
`OpenHashMap` is where I made three non-trivial changes to make it work, and they need review:
- It is no longer private
- The key must be a `ClassTag`
- Unless a lot of other code changes, the key type can't enforce being a supertype of `Null`
It all works and tests pass, and I think there is reason to believe it's OK from a speed perspective.
But what about those last changes?
Author: Sean Owen <sowen@cloudera.com>
Closes #266 from srowen/SPARK-1057-alternate and squashes the following commits:
2601129 [Sean Owen] Fix Map return type error not previously caught
ec65502 [Sean Owen] Updates from matei's review
00bc81e [Sean Owen] Remove use of fastutil and replace with use of java.io, spark.util and Guava classes
Diffstat (limited to 'project')
| -rw-r--r-- | project/SparkBuild.scala | 1 |
1 files changed, 0 insertions, 1 deletions
diff --git a/project/SparkBuild.scala b/project/SparkBuild.scala index 21163760e6..a6058bba3d 100644 --- a/project/SparkBuild.scala +++ b/project/SparkBuild.scala @@ -331,7 +331,6 @@ object SparkBuild extends Build { "org.spark-project.akka" %% "akka-slf4j" % akkaVersion excludeAll(excludeNetty), "org.spark-project.akka" %% "akka-testkit" % akkaVersion % "test", "org.json4s" %% "json4s-jackson" % "3.2.6" excludeAll(excludeScalap), - "it.unimi.dsi" % "fastutil" % "6.4.4", "colt" % "colt" % "1.2.0", "org.apache.mesos" % "mesos" % "0.13.0", "commons-net" % "commons-net" % "2.2", |