diff options
author | wm624@hotmail.com <wm624@hotmail.com> | 2016-05-17 16:51:01 +0100 |
---|---|---|
committer | Sean Owen <sowen@cloudera.com> | 2016-05-17 16:51:01 +0100 |
commit | bebe5f9811f968db92c2d33e2b30c35cfb808a4a (patch) | |
tree | 298ccd77858b6080d308578d81d1d72bbe8bb76e | |
parent | 932d8002931d352dd2ec87184e6c84ec5fa859cd (diff) | |
download | spark-bebe5f9811f968db92c2d33e2b30c35cfb808a4a.tar.gz spark-bebe5f9811f968db92c2d33e2b30c35cfb808a4a.tar.bz2 spark-bebe5f9811f968db92c2d33e2b30c35cfb808a4a.zip |
[SPARK-15318][ML][EXAMPLE] spark.ml Collaborative Filtering example does not work in spark-shell
## What changes were proposed in this pull request?
(Please fill in changes proposed in this fix)
copy & paste example in ml-collaborative-filtering.html into spark-shell, we see the following errors.
scala> case class Rating(userId: Int, movieId: Int, rating: Float, timestamp: Long)
defined class Rating
scala> object Rating {
def parseRating(str: String): Rating = { | val fields = str.split("::") | assert(fields.size == 4) | Rating(fields(0).toInt, fields(1).toInt, fields(2).toFloat, fields(3).toLong) | }
}
<console>:29: error: Rating.type does not take parameters
Rating(fields(0).toInt, fields(1).toInt, fields(2).toFloat, fields(3).toLong)
^
In standard scala repl, it has the same error.
Scala/spark-shell repl has some quirks (e.g. packages are also not well supported).
The reason of errors is that scala/spark-shell repl discards previous definitions when we define the Object with the same class name. Solution: We can rename the Object Rating.
## How was this patch tested?
(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)
Manually test it: 1). ./bin/run-example ALSExample
2). copy & paste example in the generated document. It works fine.
Author: wm624@hotmail.com <wm624@hotmail.com>
Closes #13110 from wangmiao1981/repl.
-rw-r--r-- | examples/src/main/scala/org/apache/spark/examples/ml/ALSExample.scala | 19 |
1 files changed, 12 insertions, 7 deletions
diff --git a/examples/src/main/scala/org/apache/spark/examples/ml/ALSExample.scala b/examples/src/main/scala/org/apache/spark/examples/ml/ALSExample.scala index 6b151a622e..da19ea9f10 100644 --- a/examples/src/main/scala/org/apache/spark/examples/ml/ALSExample.scala +++ b/examples/src/main/scala/org/apache/spark/examples/ml/ALSExample.scala @@ -24,16 +24,21 @@ import org.apache.spark.ml.recommendation.ALS // $example off$ import org.apache.spark.sql.SparkSession +/** + * An example demonstrating ALS. + * Run with + * {{{ + * bin/run-example ml.ALSExample + * }}} + */ object ALSExample { // $example on$ case class Rating(userId: Int, movieId: Int, rating: Float, timestamp: Long) - object Rating { - def parseRating(str: String): Rating = { - val fields = str.split("::") - assert(fields.size == 4) - Rating(fields(0).toInt, fields(1).toInt, fields(2).toFloat, fields(3).toLong) - } + def parseRating(str: String): Rating = { + val fields = str.split("::") + assert(fields.size == 4) + Rating(fields(0).toInt, fields(1).toInt, fields(2).toFloat, fields(3).toLong) } // $example off$ @@ -46,7 +51,7 @@ object ALSExample { // $example on$ val ratings = spark.read.text("data/mllib/als/sample_movielens_ratings.txt") - .map(Rating.parseRating) + .map(parseRating) .toDF() val Array(training, test) = ratings.randomSplit(Array(0.8, 0.2)) |