| Commit message (Collapse) | Author | Age | Files | Lines |
| |
|
|\ |
|
| |\
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
Quiet ERROR-level Akka Logs
This fixes an issue I've seen where akka logs a bunch of things at ERROR level when connecting to a standalone cluster, even in the normal case. I noticed that even when lifecycle logging was disabled, the netty code inside of akka still logged away via akka's EndpointWriter class. There are also some other log streams that I think are new in akka 2.2.1 that I've disabled.
Finally, I added some better logging to the standalone client. This makes it more clear when a connection failure occurs what is going on. Previously it never explicitly said if a connection attempt had failed.
The commit messages here have some more detail.
|
| | | |
|
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
Without these it's a bit less clear what's going on for the user.
One thing I realize when doing this is that akka itself actually retries
the initial association. So the retry we currently have is redundant with
akka's.
|
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
I noticed when connecting to a standalone cluster Spark gives a bunch
of Akka ERROR logs that make it seem like something is failing.
This patch does two things:
1. Akka dead letter logging is turned on/off according to the existing
lifecycle spark property.
2. We explicitly silence akka's EndpointWriter log in log4j. This is necessary
because for some reason that log doesn't pick up on the lifecycle
logging settings. After a few hours of debugging this was the only solution
I found that worked.
|
| |\ \
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | | |
Removing SPARK_EXAMPLES_JAR in the code
This re-writes all of the examples to use the `SparkContext.jarOfClass` mechanism for loading the examples jar. This necessary for environments like YARN and the Standalone mode where example programs will be submit from inside the cluster rather than at the client using `./spark-example`.
This still leaves SPARK_EXAMPLES_JAR in place in the shell scripts for setting up the classpath if `./spark-example` is run.
|
| | |/ |
|
| |\ \
| | |/
| |/|
| | |
| | |
| | | |
Fall back to zero-arg constructor for Serializer initialization if there is no constructor that accepts SparkConf.
This maintains backward compatibility with older serializers implemented by users.
|
| |/
| |
| |
| |
| |
| | |
no constructor that accepts SparkConf.
This maintains backward compatibility with older serializers implemented by users.
|
| |\
| | |
| | |
| | |
| | |
| | | |
standard Naive Bayes classifier
Has implemented the standard Naive Bayes classifier. This is an updated version of #288, which is closed because of misoperations.
|
| | | |
|
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
* Arguments renamed according to Ameet's suggestion
* Using DoubleMatrix instead of Array[Double] in computation
* Removed arguments C (kinds of label) and D (dimension of feature vector) from NaiveBayes.train()
* Replaced reduceByKey with foldByKey to avoid modifying original input data
|
| | | |
|
| | | |
|
| | | |
|
| | |
| | |
| | |
| | | |
Also refactored some heavy FP code to improve readability and reduce memory footprint.
|
| | |
| | |
| | |
| | |
| | | |
* Minimized shuffle output with mapPartitions.
* Reduced RDD actions from 3 to 1.
|
| | | |
|
| |\ \
| | | |
| | | |
| | | |
| | | |
| | | | |
SPARK-1002: Remove Binaries from Spark Source
This adds a few changes on top of the work by @scrapcodes.
|
| | |\ \
| | |/ /
| |/| |
| | | |
| | | |
| | | | |
Conflicts:
core/src/test/scala/org/apache/spark/DriverSuite.scala
docs/python-programming-guide.md
|
| |\ \ \
| | | | |
| | | | |
| | | | | |
Modify spark on yarn to create SparkConf process
|
| | |\ \ \ |
|
| | | | | | |
|
| | | | | | |
|
| |\ \ \ \ \
| | |_|/ / /
| |/| | | |
| | | | | | |
Spark-915 segregate scripts
|
| | | | | | |
|
| | | | | | |
|
| | | | | | |
|
| | | | | | |
|
| | | | | | |
|
| | | | | | |
|
| | |\ \ \ \
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | | |
spark-915-segregate-scripts
Conflicts:
bin/spark-shell
core/pom.xml
core/src/main/scala/org/apache/spark/SparkContext.scala
core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala
core/src/main/scala/org/apache/spark/ui/UIWorkloadGenerator.scala
core/src/test/scala/org/apache/spark/DriverSuite.scala
python/run-tests
sbin/compute-classpath.sh
sbin/spark-class
sbin/stop-slaves.sh
|
| | | | | | | |
|
| | | | | | | |
|
| | | |\ \ \ \ |
|
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | | |
later if needed
Signed-off-by: shane-huang <shengsheng.huang@intel.com>
|
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | | |
instead of SPARK_MEM, user should add application jars to SPARK_CLASSPATH
Signed-off-by: shane-huang <shengsheng.huang@intel.com>
|
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | | |
Signed-off-by: shane-huang <shengsheng.huang@intel.com>
|
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | | |
Signed-off-by: shane-huang <shengsheng.huang@intel.com>
|
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | | |
Signed-off-by: shane-huang <shengsheng.huang@intel.com>
|
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | | |
Signed-off-by: shane-huang <shengsheng.huang@intel.com>
|
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | | |
Signed-off-by: shane-huang <shengsheng.huang@intel.com>
|
| |\ \ \ \ \ \ \
| | |_|_|_|/ / /
| |/| | | | | |
| | | | | | | | |
Yarn refactor
|
| | | | | | | | |
|
| | | | | | | | |
|
| | | | | | | | |
|
| | | | | | | | |
|
| | | | | | | | |
|
| | | | | | | | |
|