aboutsummaryrefslogtreecommitdiff
path: root/examples/src
diff options
context:
space:
mode:
authorwm624@hotmail.com <wm624@hotmail.com>2016-05-17 16:51:01 +0100
committerSean Owen <sowen@cloudera.com>2016-05-17 16:51:01 +0100
commitbebe5f9811f968db92c2d33e2b30c35cfb808a4a (patch)
tree298ccd77858b6080d308578d81d1d72bbe8bb76e /examples/src
parent932d8002931d352dd2ec87184e6c84ec5fa859cd (diff)
downloadspark-bebe5f9811f968db92c2d33e2b30c35cfb808a4a.tar.gz
spark-bebe5f9811f968db92c2d33e2b30c35cfb808a4a.tar.bz2
spark-bebe5f9811f968db92c2d33e2b30c35cfb808a4a.zip
[SPARK-15318][ML][EXAMPLE] spark.ml Collaborative Filtering example does not work in spark-shell
## What changes were proposed in this pull request? (Please fill in changes proposed in this fix) copy & paste example in ml-collaborative-filtering.html into spark-shell, we see the following errors. scala> case class Rating(userId: Int, movieId: Int, rating: Float, timestamp: Long) defined class Rating scala> object Rating { def parseRating(str: String): Rating = { | val fields = str.split("::") | assert(fields.size == 4) | Rating(fields(0).toInt, fields(1).toInt, fields(2).toFloat, fields(3).toLong) | } } <console>:29: error: Rating.type does not take parameters Rating(fields(0).toInt, fields(1).toInt, fields(2).toFloat, fields(3).toLong) ^ In standard scala repl, it has the same error. Scala/spark-shell repl has some quirks (e.g. packages are also not well supported). The reason of errors is that scala/spark-shell repl discards previous definitions when we define the Object with the same class name. Solution: We can rename the Object Rating. ## How was this patch tested? (Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests) Manually test it: 1). ./bin/run-example ALSExample 2). copy & paste example in the generated document. It works fine. Author: wm624@hotmail.com <wm624@hotmail.com> Closes #13110 from wangmiao1981/repl.
Diffstat (limited to 'examples/src')
-rw-r--r--examples/src/main/scala/org/apache/spark/examples/ml/ALSExample.scala19
1 files changed, 12 insertions, 7 deletions
diff --git a/examples/src/main/scala/org/apache/spark/examples/ml/ALSExample.scala b/examples/src/main/scala/org/apache/spark/examples/ml/ALSExample.scala
index 6b151a622e..da19ea9f10 100644
--- a/examples/src/main/scala/org/apache/spark/examples/ml/ALSExample.scala
+++ b/examples/src/main/scala/org/apache/spark/examples/ml/ALSExample.scala
@@ -24,16 +24,21 @@ import org.apache.spark.ml.recommendation.ALS
// $example off$
import org.apache.spark.sql.SparkSession
+/**
+ * An example demonstrating ALS.
+ * Run with
+ * {{{
+ * bin/run-example ml.ALSExample
+ * }}}
+ */
object ALSExample {
// $example on$
case class Rating(userId: Int, movieId: Int, rating: Float, timestamp: Long)
- object Rating {
- def parseRating(str: String): Rating = {
- val fields = str.split("::")
- assert(fields.size == 4)
- Rating(fields(0).toInt, fields(1).toInt, fields(2).toFloat, fields(3).toLong)
- }
+ def parseRating(str: String): Rating = {
+ val fields = str.split("::")
+ assert(fields.size == 4)
+ Rating(fields(0).toInt, fields(1).toInt, fields(2).toFloat, fields(3).toLong)
}
// $example off$
@@ -46,7 +51,7 @@ object ALSExample {
// $example on$
val ratings = spark.read.text("data/mllib/als/sample_movielens_ratings.txt")
- .map(Rating.parseRating)
+ .map(parseRating)
.toDF()
val Array(training, test) = ratings.randomSplit(Array(0.8, 0.2))