aboutsummaryrefslogtreecommitdiff
Commit message (Collapse)AuthorAgeFilesLines
* Fixed examples/pom.xml and run-example based on Patrick's suggestions.Tathagata Das2014-01-072-12/+2
|
* Removed XYZFunctions and added XYZUtils as a common Scala and Java interface ↵Tathagata Das2014-01-0735-646/+383
| | | | for creating XYZ streams.
* Merge remote-tracking branch 'apache/master' into project-refactorTathagata Das2014-01-06302-3381/+3981
|\ | | | | | | | | | | | | | | | | | | Conflicts: examples/src/main/java/org/apache/spark/streaming/examples/JavaFlumeEventCount.java streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaStreamingContext.scala streaming/src/test/java/org/apache/spark/streaming/JavaAPISuite.java streaming/src/test/scala/org/apache/spark/streaming/InputStreamsSuite.scala streaming/src/test/scala/org/apache/spark/streaming/TestSuiteBase.scala
| * Merge pull request #333 from pwendell/logging-silencePatrick Wendell2014-01-052-3/+25
| |\ | | | | | | | | | | | | | | | | | | | | | | | | | | | Quiet ERROR-level Akka Logs This fixes an issue I've seen where akka logs a bunch of things at ERROR level when connecting to a standalone cluster, even in the normal case. I noticed that even when lifecycle logging was disabled, the netty code inside of akka still logged away via akka's EndpointWriter class. There are also some other log streams that I think are new in akka 2.2.1 that I've disabled. Finally, I added some better logging to the standalone client. This makes it more clear when a connection failure occurs what is going on. Previously it never explicitly said if a connection attempt had failed. The commit messages here have some more detail.
| | * Responding to Aaron's reviewPatrick Wendell2014-01-051-0/+2
| | |
| | * Provide logging when attempts to connect to the master fail.Patrick Wendell2014-01-051-1/+11
| | | | | | | | | | | | | | | | | | | | | | | | Without these it's a bit less clear what's going on for the user. One thing I realize when doing this is that akka itself actually retries the initial association. So the retry we currently have is redundant with akka's.
| | * Quite akka when remote lifecycle logging is disabled.Patrick Wendell2014-01-051-2/+12
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | I noticed when connecting to a standalone cluster Spark gives a bunch of Akka ERROR logs that make it seem like something is failing. This patch does two things: 1. Akka dead letter logging is turned on/off according to the existing lifecycle spark property. 2. We explicitly silence akka's EndpointWriter log in log4j. This is necessary because for some reason that log doesn't pick up on the lifecycle logging settings. After a few hours of debugging this was the only solution I found that worked.
| * | Merge pull request #334 from pwendell/examples-fixReynold Xin2014-01-0547-54/+75
| |\ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | Removing SPARK_EXAMPLES_JAR in the code This re-writes all of the examples to use the `SparkContext.jarOfClass` mechanism for loading the examples jar. This necessary for environments like YARN and the Standalone mode where example programs will be submit from inside the cluster rather than at the client using `./spark-example`. This still leaves SPARK_EXAMPLES_JAR in place in the shell scripts for setting up the classpath if `./spark-example` is run.
| | * | Removing SPARK_EXAMPLES_JAR in the codePatrick Wendell2014-01-0547-54/+75
| | |/
| * | Merge pull request #335 from rxin/serReynold Xin2014-01-052-2/+16
| |\ \ | | |/ | |/| | | | | | | | | | Fall back to zero-arg constructor for Serializer initialization if there is no constructor that accepts SparkConf. This maintains backward compatibility with older serializers implemented by users.
| | * Fall back to zero-arg constructor for Serializer initialization if there is ↵Reynold Xin2014-01-052-2/+16
| |/ | | | | | | | | | | no constructor that accepts SparkConf. This maintains backward compatibility with older serializers implemented by users.
| * Merge pull request #292 from soulmachine/naive-bayesReynold Xin2014-01-042-0/+227
| |\ | | | | | | | | | | | | | | | standard Naive Bayes classifier Has implemented the standard Naive Bayes classifier. This is an updated version of #288, which is closed because of misoperations.
| | * Aggregated all sample points to driver without any shuffleLian, Cheng2014-01-022-53/+31
| | |
| | * Response to comments from Reynold, Ameet and EvanLian, Cheng2013-12-302-62/+90
| | | | | | | | | | | | | | | | | | | | | * Arguments renamed according to Ameet's suggestion * Using DoubleMatrix instead of Array[Double] in computation * Removed arguments C (kinds of label) and D (dimension of feature vector) from NaiveBayes.train() * Replaced reduceByKey with foldByKey to avoid modifying original input data
| | * Response to Reynold's commentsLian, Cheng2013-12-291-10/+16
| | |
| | * Added Apache license header to NaiveBayesSuiteLian, Cheng2013-12-271-0/+17
| | |
| | * Reformatted some lines commented by MateiLian, Cheng2013-12-271-2/+3
| | |
| | * Let reduceByKey to take care of local combineLian, Cheng2013-12-251-27/+16
| | | | | | | | | | | | Also refactored some heavy FP code to improve readability and reduce memory footprint.
| | * Refactored NaiveBayesLian, Cheng2013-12-252-28/+41
| | | | | | | | | | | | | | | * Minimized shuffle output with mapPartitions. * Reduced RDD actions from 3 to 1.
| | * standard Naive Bayes classifierFrank Dai2013-12-252-0/+195
| | |
| * | Merge pull request #329 from pwendell/remove-binariesPatrick Wendell2014-01-0333-210/+128
| |\ \ | | | | | | | | | | | | | | | | | | | | SPARK-1002: Remove Binaries from Spark Source This adds a few changes on top of the work by @scrapcodes.
| | * \ Merge remote-tracking branch 'apache-github/master' into remove-binariesPatrick Wendell2014-01-0398-1453/+436
| | |\ \ | | |/ / | |/| | | | | | | | | | | | | | Conflicts: core/src/test/scala/org/apache/spark/DriverSuite.scala docs/python-programming-guide.md
| * | | Merge pull request #325 from witgo/masterPatrick Wendell2014-01-0313-66/+92
| |\ \ \ | | | | | | | | | | | | | | | Modify spark on yarn to create SparkConf process
| | * \ \ merge upstream/masterliguoqiang2014-01-0336-1236/+198
| | |\ \ \
| | * | | | Modify spark on yarn to create SparkConf processliguoqiang2014-01-037-48/+65
| | | | | |
| | * | | | Modify spark on yarn to create SparkConf processliguoqiang2014-01-0315-46/+56
| | | | | |
| * | | | | Merge pull request #317 from ScrapCodes/spark-915-segregate-scriptsPatrick Wendell2014-01-0362-161/+155
| |\ \ \ \ \ | | |_|/ / / | |/| | | | | | | | | | Spark-915 segregate scripts
| | * | | | sbin/compute-classpath* bin/compute-classpath*Prashant Sharma2014-01-035-3/+3
| | | | | |
| | * | | | sbin/spark-class* -> bin/spark-class*Prashant Sharma2014-01-0314-15/+15
| | | | | |
| | * | | | a few left over document changePrashant Sharma2014-01-023-4/+4
| | | | | |
| | * | | | pyspark -> bin/pysparkPrashant Sharma2014-01-025-19/+19
| | | | | |
| | * | | | run-example -> bin/run-examplePrashant Sharma2014-01-0219-31/+31
| | | | | |
| | * | | | spark-shell -> bin/spark-shellPrashant Sharma2014-01-029-15/+15
| | | | | |
| | * | | | Merge branch 'scripts-reorg' of github.com:shane-huang/incubator-spark into ↵Prashant Sharma2014-01-0241-96/+90
| | |\ \ \ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | spark-915-segregate-scripts Conflicts: bin/spark-shell core/pom.xml core/src/main/scala/org/apache/spark/SparkContext.scala core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala core/src/main/scala/org/apache/spark/ui/UIWorkloadGenerator.scala core/src/test/scala/org/apache/spark/DriverSuite.scala python/run-tests sbin/compute-classpath.sh sbin/spark-class sbin/stop-slaves.sh
| | | * | | | deprecate "spark" script and SPAKR_CLASSPATH environment variableAndrew xia2013-10-127-99/+4
| | | | | | |
| | | * | | | refactor $FWD variableAndrew xia2013-09-295-7/+7
| | | | | | |
| | | * | | | Merge branch 'reorgscripts' into scripts-reorgshane-huang2013-09-2740-87/+175
| | | |\ \ \ \
| | | | * | | | rm bin/spark.cmd as we don't have windows test environment. Will added it ↵shane-huang2013-09-261-27/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | later if needed Signed-off-by: shane-huang <shengsheng.huang@intel.com>
| | | | * | | | fix paths and change spark to use APP_MEM as application driver memory ↵shane-huang2013-09-263-35/+10
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | instead of SPARK_MEM, user should add application jars to SPARK_CLASSPATH Signed-off-by: shane-huang <shengsheng.huang@intel.com>
| | | | * | | | fix pathshane-huang2013-09-261-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Signed-off-by: shane-huang <shengsheng.huang@intel.com>
| | | | * | | | add scripts in binshane-huang2013-09-2312-17/+163
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Signed-off-by: shane-huang <shengsheng.huang@intel.com>
| | | | * | | | moved user scripts to bin foldershane-huang2013-09-2311-0/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Signed-off-by: shane-huang <shengsheng.huang@intel.com>
| | | | * | | | add admin scripts to sbinshane-huang2013-09-2314-47/+47
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Signed-off-by: shane-huang <shengsheng.huang@intel.com>
| | | | * | | | added spark-class and spark-executor to sbinshane-huang2013-09-2314-22/+16
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Signed-off-by: shane-huang <shengsheng.huang@intel.com>
| * | | | | | | Merge pull request #285 from colorant/yarn-refactorPatrick Wendell2014-01-0236-1226/+189
| |\ \ \ \ \ \ \ | | |_|_|_|/ / / | |/| | | | | | | | | | | | | | Yarn refactor
| | * | | | | | fix docs for yarnRaymond Liu2014-01-032-5/+2
| | | | | | | |
| | * | | | | | minor fix for loginfoRaymond Liu2014-01-031-1/+1
| | | | | | | |
| | * | | | | | move duplicate pom config into parent pomRaymond Liu2014-01-033-179/+84
| | | | | | | |
| | * | | | | | Using name yarn-alpha/yarn instead of yarn-2.0/yarn-2.2Raymond Liu2014-01-0318-30/+30
| | | | | | | |
| | * | | | | | Add yarn/common/src/test dir in building scriptRaymond Liu2014-01-031-0/+7
| | | | | | | |
generated by cgit v1.2.3 (git 2.39.1) at 2024-07-10 12:19:58 +0000