| Commit message (Collapse) | Author | Age | Files | Lines |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
- Got rid of global SparkContext.globalConf
- Pass SparkConf to serializers and compression codecs
- Made SparkConf public instead of private[spark]
- Improved API of SparkContext and SparkConf
- Switched executor environment vars to be passed through SparkConf
- Fixed some places that were still using system properties
- Fixed some tests, though others are still failing
This still fails several tests in core, repl and streaming, likely due
to properties not being set or cleared correctly (some of the tests run
fine in isolation).
|
| |
|
|\
| |
| |
| |
| |
| |
| |
| | |
DRY out the POMs with scala.binary.version
...instead of hard-coding 2.10 repeatedly.
As long as it's not a `<project>`-level `<artifactId>`, I think that we are okay parameterizing these.
|
| | |
|
|\ \
| |/
|/|
| |
| |
| | |
Fix the --name option for Spark on Yarn
Looks like the --name option accidentally got broken in one of the merges. The Client hangs if the --name option is used right now.
|
| | |
|
|\|
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Conflicts:
core/pom.xml
core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
pom.xml
project/SparkBuild.scala
streaming/pom.xml
yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala
|
| |\
| | |
| | |
| | |
| | | |
Conflicts:
yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala
|
| | | |
|
| |\ \
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | | |
Conflicts:
yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala
yarn/src/main/scala/org/apache/spark/deploy/yarn/WorkerRunnable.scala
yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala
|
| | | | |
|
| | | | |
|
| | | | |
|
| | | |
| | | |
| | | |
| | | | |
build/dep-resolution atleast a bit faster.
|
| | | | |
|
|\ \ \ \
| | |_|/
| |/| |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | | |
Conflicts:
core/src/main/scala/org/apache/spark/api/python/PythonRDD.scala
core/src/main/scala/org/apache/spark/rdd/MapPartitionsRDD.scala
core/src/main/scala/org/apache/spark/rdd/MapPartitionsWithContextRDD.scala
core/src/main/scala/org/apache/spark/rdd/RDD.scala
python/pyspark/rdd.py
|
| | |/
| |/|
| | |
| | |
| | |
| | |
| | | |
With this scheduler, the user application is launched locally,
While the executor will be launched by YARN on remote nodes.
This enables spark-shell to run upon YARN.
|
|\| |
| | |
| | |
| | |
| | |
| | | |
Conflicts:
core/src/main/scala/org/apache/spark/util/collection/PrimitiveVector.scala
streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaStreamingContext.scala
|
| |\ \ |
|
| | |/ |
|
| | |
| | |
| | |
| | | |
Passed the sbt/sbt compile and test
|
| |/
| |
| |
| |
| |
| |
| | |
Also remove unused imports as I found them along the way.
Remove return statements when returning value in the Scala code.
Passing compile and tests.
|
| |
| |
| |
| |
| |
| | |
-) Remove some of unused imports as I found them
-) Remove ";" in the imports statements
-) Remove () at the end of method call like size that does not have size effect.
|
|\| |
|
| |
| |
| |
| | |
log4j.properties to be put into hdfs.
|
|\| |
|
| |
| |
| |
| |
| |
| | |
be explicit
about inclusion of spark.jar and app.jar
|
| | |
|
| |\
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
Closes #11
Conflicts:
docs/running-on-yarn.md
yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala
|
| | |
| | |
| | |
| | |
| | |
| | | |
cleanup
the classpaths
|
| | |
| | |
| | |
| | | |
cleanup the staging directory on exit
|
|\| |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
Conflicts:
bagel/pom.xml
core/pom.xml
core/src/test/scala/org/apache/spark/ui/UISuite.scala
examples/pom.xml
mllib/pom.xml
pom.xml
project/SparkBuild.scala
repl/pom.xml
streaming/pom.xml
tools/pom.xml
In scala 2.10, a shorter representation is used for naming artifacts
so changed to shorter scala version for artifacts and made it a property in pom.
|
| | | |
|
|\| |
| | |
| | |
| | |
| | |
| | | |
Conflicts:
core/src/test/scala/org/apache/spark/DistributedSuite.scala
project/SparkBuild.scala
|
| | | |
|
| | | |
|
|\| |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
Conflicts:
core/src/main/scala/org/apache/spark/ui/jobs/JobProgressUI.scala
docs/_config.yml
project/SparkBuild.scala
repl/src/main/scala/org/apache/spark/repl/SparkILoop.scala
|
| | | |
|
| |/ |
|
| | |
|
|/ |
|
|
|
|
|
| |
This uses a seperate Hadoop version for YARN artifact. This means when people link against
spark-yarn, things will resolve correctly.
|
| |
|
| |
|
|\
| |
| |
| |
| |
| |
| |
| | |
Conflicts:
core/src/main/scala/org/apache/spark/ui/UIUtils.scala
core/src/main/scala/org/apache/spark/ui/jobs/PoolTable.scala
core/src/main/scala/org/apache/spark/ui/jobs/StageTable.scala
docs/running-on-yarn.md
|
| | |
|
| | |
|
| | |
|
|/ |
|
|\
| |
| | |
Spark on Yarn should use yarn approved directories for spark.local.dir and tmp
|