aboutsummaryrefslogtreecommitdiff
Commit message (Collapse)AuthorAgeFilesLines
...
| | * | | Minor typo fix for yarn clientRaymond Liu2014-01-072-2/+2
| | | | |
| * | | | Merge pull request #322 from falaki/MLLibDocumentationImprovementPatrick Wendell2014-01-071-56/+274
| |\ \ \ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | SPARK-1009 Updated MLlib docs to show how to use it in Python In addition added detailed examples for regression, clustering and recommendation algorithms in a separate Scala section. Fixed a few minor issues with existing documentation.
| | * \ \ \ Fixed merge conflictHossein Falaki2014-01-07306-3327/+4228
| | |\ \ \ \
| | * | | | | Added proper evaluation example for collaborative filtering and fixed typoHossein Falaki2014-01-061-4/+8
| | | | | | |
| | * | | | | Added table of contents and minor fixesHossein Falaki2014-01-031-8/+16
| | | | | | |
| | * | | | | Commented the last part of collaborative filtering examples that lead to errorsHossein Falaki2014-01-021-5/+6
| | | | | | |
| | * | | | | Added Scala and Python examples for mllibHossein Falaki2014-01-021-52/+261
| | | | | | |
| * | | | | | Merge pull request #355 from ScrapCodes/patch-1Patrick Wendell2014-01-071-1/+1
| |\ \ \ \ \ \ | | |_|_|_|_|/ | |/| | | | | | | | | | | | | | | | | | | | | | | | | | Update README.md The link does not work otherwise.
| | * | | | | Update README.mdPrashant Sharma2014-01-081-1/+1
| | | |_|_|/ | | |/| | | | | | | | | The link does not work otherwise.
| * | | | | Merge pull request #313 from tdas/project-refactorPatrick Wendell2014-01-0751-739/+1907
| |\ \ \ \ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Refactored the streaming project to separate external libraries like Twitter, Kafka, Flume, etc. At a high level, these are the following changes. 1. All the external code was put in `SPARK_HOME/external/` as separate SBT projects and Maven modules. Their artifact names are `spark-streaming-twitter`, `spark-streaming-kafka`, etc. Both SparkBuild.scala and pom.xml files have been updated. References to external libraries and repositories have been removed from the settings of root and streaming projects/modules. 2. To avail the external functionality (say, creating a Twitter stream), the developer has to `import org.apache.spark.streaming.twitter._` . For Scala API, the developer has to call `TwitterUtils.createStream(streamingContext, ...)`. For the Java API, the developer has to call `TwitterUtils.createStream(javaStreamingContext, ...)`. 3. Each external project has its own scala and java unit tests. Note the unit tests of each external library use classes of the streaming unit tests (`TestSuiteBase`, `LocalJavaStreamingContext`, etc.). To enable this code sharing among test classes, `dependsOn(streaming % "compile->compile,test->test")` was used in the SparkBuild.scala . In the streaming/pom.xml, an additional `maven-jar-plugin` was necessary to capture this dependency (see comment inside the pom.xml for more information). 4. Jars of the external projects have been added to examples project but not to the assembly project. 5. In some files, imports have been rearrange to conform to the Spark coding guidelines.
| | * | | | | Fixed examples/pom.xml and run-example based on Patrick's suggestions.Tathagata Das2014-01-072-12/+2
| | | | | | |
| | * | | | | Removed XYZFunctions and added XYZUtils as a common Scala and Java interface ↵Tathagata Das2014-01-0735-646/+383
| | | | | | | | | | | | | | | | | | | | | | | | | | | | for creating XYZ streams.
| | * | | | | Merge remote-tracking branch 'apache/master' into project-refactorTathagata Das2014-01-06302-3381/+3981
| | |\ \ \ \ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Conflicts: examples/src/main/java/org/apache/spark/streaming/examples/JavaFlumeEventCount.java streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaStreamingContext.scala streaming/src/test/java/org/apache/spark/streaming/JavaAPISuite.java streaming/src/test/scala/org/apache/spark/streaming/InputStreamsSuite.scala streaming/src/test/scala/org/apache/spark/streaming/TestSuiteBase.scala
| | * | | | | | Changed JavaStreamingContextWith*** to ***Function in streaming.api.java.*** ↵Tathagata Das2014-01-0615-76/+67
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | package. Also fixed packages of Flume and MQTT tests.
| | * | | | | | Merge branch 'apache-master' into project-refactorTathagata Das2013-12-3114-67/+52
| | |\ \ \ \ \ \
| | * | | | | | | Removed extra empty lines.Tathagata Das2013-12-313-3/+0
| | | | | | | | |
| | * | | | | | | Removed unnecessary comments.Tathagata Das2013-12-313-55/+8
| | | | | | | | |
| | * | | | | | | Added pom.xml for external projects and removed unnecessary dependencies and ↵Tathagata Das2013-12-319-106/+548
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | repositoris from other poms and sbt.
| | * | | | | | | Refactored kafka, flume, zeromq, mqtt as separate external projects, with ↵Tathagata Das2013-12-3050-599/+1612
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | their own self-contained scala API, java API, scala unit tests and java unit tests. Updated examples to use the external projects.
| | * | | | | | | Refactored streaming project to separate out the twitter functionality.Tathagata Das2013-12-269-14/+64
| | | | | | | | |
| * | | | | | | | Merge pull request #336 from liancheng/akka-remote-lookupPatrick Wendell2014-01-078-38/+30
| |\ \ \ \ \ \ \ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Get rid of `Either[ActorRef, ActorSelection]' In this pull request, instead of returning an `Either[ActorRef, ActorSelection]`, `registerOrLookup` identifies the remote actor blockingly to obtain an `ActorRef`, or throws an exception if the remote actor doesn't exist or the lookup times out (configured by `spark.akka.lookupTimeout`). This function is only called when an `SparkEnv` is constructed (instantiating driver or executor), so the blocking call is considered acceptable. Executor side `ActorSelection`s/`ActorRef`s to driver side `MapOutputTrackerMasterActor` and `BlockManagerMasterActor` are affected by this pull request. `ActorSelection` is dangerous and should be used with care. It's only absolutely safe to send messages via an `ActorSelection` when the remote actor is stateless, so that actor incarnation is irrelevant. But as pointed by @ScrapCodes in the comments below, executor exits immediately once the connection to the driver lost, `ActorSelection`s are not harmful in this scenario. So this pull request is mostly a code style patch.
| | * | | | | | | | Fixed test suite compilation errorsLian, Cheng2014-01-061-3/+3
| | | | | | | | | |
| | * | | | | | | | Fixed several compilation errors in test suitesLian, Cheng2014-01-062-5/+8
| | | | | | | | | |
| | * | | | | | | | Get rid of `Either[ActorRef, ActorSelection]'Lian, Cheng2014-01-066-30/+19
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Although we can send messages via an ActorSelection, it would be better to identify the actor and obtain an ActorRef first, so that we can get informed earlier if the remote actor doesn't exist, and get rid of the annoying Either wrapper.
| * | | | | | | | | Merge pull request #327 from lucarosellini/masterMatei Zaharia2014-01-083-3/+73
| |\ \ \ \ \ \ \ \ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Added ‘-i’ command line option to Spark REPL We had to create a new implementation of both scala.tools.nsc.CompilerCommand and scala.tools.nsc.Settings, because using scala.tools.nsc.GenericRunnerSettings would bring in other options (-howtorun, -save and -execute) which don’t make sense in Spark. Any new Spark specific command line option could now be added to org.apache.spark.repl.SparkRunnerSettings class. Since the behavior of loading a script from the command line should be the same as loading it using the “:load” command inside the shell, the script should be loaded when the SparkContext is available, that’s why we had to move the call to ‘loadfiles(settings)’ _after_ the call to postInitialization(). This still doesn’t work if ‘isAsync = true’.
| | * | | | | | | | | Added license header and removed @author tagLuca Rosellini2014-01-072-4/+34
| | | | | | | | | | |
| | * | | | | | | | | Added ‘-i’ command line option to spark REPL.Luca Rosellini2014-01-033-3/+43
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | We had to create a new implementation of both scala.tools.nsc.CompilerCommand and scala.tools.nsc.Settings, because using scala.tools.nsc.GenericRunnerSettings would bring in other options (-howtorun, -save and -execute) which don’t make sense in Spark. Any new Spark specific command line option could now be added to org.apache.spark.repl.SparkRunnerSettings class. Since the behavior of loading a script from the command line should be the same as loading it using the “:load” command inside the shell, the script should be loaded when the SparkContext is available, that’s why we had to move the call to ‘loadfiles(settings)’ _after_ the call to postInitialization(). This still doesn’t work if ‘isAsync = true’.
| | * | | | | | | | | Merge pull request #1 from apache/masterLuca Rosellini2014-01-0352-1542/+785
| | |\ \ \ \ \ \ \ \ \ | | | | | | | | | | | | | | | | | | | | | | | | Merge latest Spark changes
| * | \ \ \ \ \ \ \ \ \ Merge pull request #354 from hsaputra/addasfheadertosbtMatei Zaharia2014-01-081-0/+18
| |\ \ \ \ \ \ \ \ \ \ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Add ASF header to the new sbt script. Add ASF header to the new sbt script.
| | * | | | | | | | | | | Add ASF header to the new sbt script.Henry Saputra2014-01-071-0/+18
| | | |_|_|_|_|_|/ / / / | | |/| | | | | | | | |
| * | | | | | | | | | | Merge pull request #350 from mateiz/standalone-limitMatei Zaharia2014-01-0813-20/+70
| |\ \ \ \ \ \ \ \ \ \ \ | | |/ / / / / / / / / / | |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Add way to limit default # of cores used by apps in standalone mode Also documents the spark.deploy.spreadOut option, and fixes a config option that had a dash in its name.
| | * | | | | | | | | | Address review commentsMatei Zaharia2014-01-078-8/+11
| | | | | | | | | | | |
| | * | | | | | | | | | Fix unit test compilationMatei Zaharia2014-01-071-1/+2
| | | | | | | | | | | |
| | * | | | | | | | | | Add way to limit default # of cores used by applications on standalone modeMatei Zaharia2014-01-078-14/+60
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Also documents the spark.deploy.spreadOut option.
| * | | | | | | | | | | Merge pull request #352 from markhamstra/oldArchPatrick Wendell2014-01-071-8/+2
| |\ \ \ \ \ \ \ \ \ \ \ | | |_|_|_|_|_|_|_|_|/ / | |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Don't leave os.arch unset after BlockManagerSuite Recent SparkConf changes meant that BlockManagerSuite was now leaving the os.arch System.property unset. That's a problem for any subsequent tests that rely upon having a valid os.arch. This is true for CompressionCodecSuite in the usual maven build test order, even though it isn't usually true for the sbt build.
| | * | | | | | | | | | Fix BlockManagerSuite#afterMark Hamstra2014-01-071-8/+2
| | | | | | | | | | | |
| * | | | | | | | | | | Merge pull request #328 from falaki/MatrixFactorizationModel-fixPatrick Wendell2014-01-075-4/+134
| |\ \ \ \ \ \ \ \ \ \ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | SPARK-1012: DAGScheduler Exception Fix Added a predict method to MatrixFactorizationModel to enable bulk prediction. This method takes and RDD[(Int, Int)] of users and products and return an RDD with a Rating element per each element in the input RDD. Also added python bindings to the new bulk prediction methods to address SPARK-1011 issue. This is ready to be merged now.
| | * | | | | | | | | | | Merge branch 'master' into MatrixFactorizationModel-fixHossein Falaki2014-01-07170-1931/+1342
| | |\| | | | | | | | | |
| | * | | | | | | | | | | Added predictAll python function to MatrixFactorizationModelHossein Falaki2014-01-061-4/+6
| | | | | | | | | | | | |
| | * | | | | | | | | | | Added Rating deserializerHossein Falaki2014-01-062-4/+26
| | | | | | | | | | | | |
| | * | | | | | | | | | | Added serializing method for Rating objectHossein Falaki2014-01-061-4/+16
| | | | | | | | | | | | |
| | * | | | | | | | | | | Added python binding for bulk recommendationHossein Falaki2014-01-044-2/+46
| | | | | | | | | | | | |
| | * | | | | | | | | | | Removed unnecessary blank lineHossein Falaki2014-01-031-1/+0
| | | | | | | | | | | | |
| | * | | | | | | | | | | Added unit tests for bulk prediction in MatrixFactorizationModelHossein Falaki2014-01-031-2/+31
| | | | | | | | | | | | |
| | * | | | | | | | | | | Added a method to enable bulk predictionHossein Falaki2014-01-031-1/+23
| | | | | | | | | | | | |
| * | | | | | | | | | | | Merge pull request #351 from pwendell/maven-fixPatrick Wendell2014-01-074-7/+12
| |\ \ \ \ \ \ \ \ \ \ \ \ | | |_|/ / / / / / / / / / | |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Add log4j exclusion rule to maven. To make this work I had to rename the defaults file. Otherwise maven's pattern matching rules included it when trying to match other log4j.properties files. I also fixed a bug in the existing maven build where two <transformers> tags were present in assembly/pom.xml such that one overwrote the other.
| | * | | | | | | | | | | Add log4j exclusion rule to maven.Patrick Wendell2014-01-074-7/+12
| | | |/ / / / / / / / / | | |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | To make this work I had to rename the defaults file. Otherwise maven's pattern matching rules included it when trying to match other log4j.properties files. I also fixed a bug in the existing maven build where two <transformers> tags were present in assembly/pom.xml such that one overwrote the other.
| * | | | | | | | | | | Merge pull request #337 from yinxusen/mllib-16-bugfixReynold Xin2014-01-072-2/+118
| |\ \ \ \ \ \ \ \ \ \ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Mllib 16 bugfix Bug fix: https://spark-project.atlassian.net/browse/MLLIB-16 Hi, I fixed the bug and added a test suite for `GradientDescent`. There are 2 checks in the test case. First, the final loss must be lower than the initial one. Second, the trend of loss sequence should be decreasing, i.e., at least 80% iterations have lower losses than their prior iterations. Thanks!
| | * | | | | | | | | | | Added GradientDescentSuiteXusen Yin2014-01-061-0/+116
| | | | | | | | | | | | |
| | * | | | | | | | | | | fix logistic loss bugXusen Yin2014-01-061-2/+2
| | | | | | | | | | | | |
generated by cgit v1.2.3 (git 2.39.1) at 2024-09-11 04:52:50 +0000