aboutsummaryrefslogtreecommitdiff
path: root/bin
Commit message (Collapse)AuthorAgeFilesLines
* Finish d1d2b6d9b6b5f9cc45047507368a816903722d9eAnkur Dave2014-01-101-1/+0
|
* graph -> graphx in bin/compute-classpath.shAnkur Dave2014-01-091-2/+2
|
* Merge remote-tracking branch 'spark-upstream/master' into HEADAnkur Dave2014-01-0824-612/+709
|\ | | | | | | | | | | | | | | | | | | | | Conflicts: README.md core/src/main/scala/org/apache/spark/util/collection/OpenHashMap.scala core/src/main/scala/org/apache/spark/util/collection/OpenHashSet.scala core/src/main/scala/org/apache/spark/util/collection/PrimitiveKeyOpenHashMap.scala pom.xml project/SparkBuild.scala repl/src/main/scala/org/apache/spark/repl/SparkILoop.scala
| * Merge pull request #313 from tdas/project-refactorPatrick Wendell2014-01-071-6/+1
| |\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Refactored the streaming project to separate external libraries like Twitter, Kafka, Flume, etc. At a high level, these are the following changes. 1. All the external code was put in `SPARK_HOME/external/` as separate SBT projects and Maven modules. Their artifact names are `spark-streaming-twitter`, `spark-streaming-kafka`, etc. Both SparkBuild.scala and pom.xml files have been updated. References to external libraries and repositories have been removed from the settings of root and streaming projects/modules. 2. To avail the external functionality (say, creating a Twitter stream), the developer has to `import org.apache.spark.streaming.twitter._` . For Scala API, the developer has to call `TwitterUtils.createStream(streamingContext, ...)`. For the Java API, the developer has to call `TwitterUtils.createStream(javaStreamingContext, ...)`. 3. Each external project has its own scala and java unit tests. Note the unit tests of each external library use classes of the streaming unit tests (`TestSuiteBase`, `LocalJavaStreamingContext`, etc.). To enable this code sharing among test classes, `dependsOn(streaming % "compile->compile,test->test")` was used in the SparkBuild.scala . In the streaming/pom.xml, an additional `maven-jar-plugin` was necessary to capture this dependency (see comment inside the pom.xml for more information). 4. Jars of the external projects have been added to examples project but not to the assembly project. 5. In some files, imports have been rearrange to conform to the Spark coding guidelines.
| | * Fixed examples/pom.xml and run-example based on Patrick's suggestions.Tathagata Das2014-01-071-6/+1
| | |
| * | CR feedback (sbt -> sbt/sbt and correct JAR path in script) :)Holden Karau2014-01-051-1/+1
| | |
| * | Finish documentation changesHolden Karau2014-01-052-2/+2
| |/
| * Merge remote-tracking branch 'apache-github/master' into remove-binariesPatrick Wendell2014-01-0324-610/+712
| |\ | | | | | | | | | | | | | | | Conflicts: core/src/test/scala/org/apache/spark/DriverSuite.scala docs/python-programming-guide.md
| | * sbin/compute-classpath* bin/compute-classpath*Prashant Sharma2014-01-034-2/+146
| | |
| | * sbin/spark-class* -> bin/spark-class*Prashant Sharma2014-01-036-4/+266
| | |
| | * run-example -> bin/run-examplePrashant Sharma2014-01-022-2/+2
| | |
| | * Merge branch 'scripts-reorg' of github.com:shane-huang/incubator-spark into ↵Prashant Sharma2014-01-0221-752/+448
| |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | spark-915-segregate-scripts Conflicts: bin/spark-shell core/pom.xml core/src/main/scala/org/apache/spark/SparkContext.scala core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala core/src/main/scala/org/apache/spark/ui/UIWorkloadGenerator.scala core/src/test/scala/org/apache/spark/DriverSuite.scala python/run-tests sbin/compute-classpath.sh sbin/spark-class sbin/stop-slaves.sh
| | * deprecate "spark" script and SPAKR_CLASSPATH environment variableAndrew xia2013-10-121-92/+0
| | |
| | * refactor $FWD variableAndrew xia2013-09-293-4/+4
| | |
| | * rm bin/spark.cmd as we don't have windows test environment. Will added it ↵shane-huang2013-09-261-27/+0
| | | | | | | | | | | | | | | | | | later if needed Signed-off-by: shane-huang <shengsheng.huang@intel.com>
| | * fix paths and change spark to use APP_MEM as application driver memory ↵shane-huang2013-09-261-33/+8
| | | | | | | | | | | | | | | | | | instead of SPARK_MEM, user should add application jars to SPARK_CLASSPATH Signed-off-by: shane-huang <shengsheng.huang@intel.com>
| | * add scripts in binshane-huang2013-09-238-10/+155
| | | | | | | | | | | | Signed-off-by: shane-huang <shengsheng.huang@intel.com>
| | * moved user scripts to bin foldershane-huang2013-09-238-0/+418
| | | | | | | | | | | | Signed-off-by: shane-huang <shengsheng.huang@intel.com>
| | * add admin scripts to sbinshane-huang2013-09-2313-704/+0
| | | | | | | | | | | | Signed-off-by: shane-huang <shengsheng.huang@intel.com>
| | * added spark-class and spark-executor to sbinshane-huang2013-09-231-1/+1
| | | | | | | | | | | | Signed-off-by: shane-huang <shengsheng.huang@intel.com>
| * | Merge branch 'master' into scala-2.10Raymond Liu2013-11-135-9/+57
| |\ \
| * | | version changed 2.9.3 -> 2.10 in shell script.Prashant Sharma2013-09-151-1/+1
| | | |
| * | | Merged with masterPrashant Sharma2013-09-0613-121/+251
| |\ \ \ | | | |/ | | |/|
| * | | Merge branch 'master' of github.com:mesos/spark into scala-2.10Prashant Sharma2013-07-152-49/+65
| |\ \ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Conflicts: core/src/main/scala/spark/Utils.scala core/src/test/scala/spark/ui/UISuite.scala project/SparkBuild.scala run
| * \ \ \ Merge branch 'master' into master-mergePrashant Sharma2013-07-122-0/+4
| |\ \ \ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Conflicts: README.md core/pom.xml core/src/main/scala/spark/deploy/JsonProtocol.scala core/src/main/scala/spark/deploy/LocalSparkCluster.scala core/src/main/scala/spark/deploy/master/Master.scala core/src/main/scala/spark/deploy/master/MasterWebUI.scala core/src/main/scala/spark/deploy/worker/Worker.scala core/src/main/scala/spark/deploy/worker/WorkerWebUI.scala core/src/main/scala/spark/storage/BlockManagerUI.scala core/src/main/scala/spark/util/AkkaUtils.scala pom.xml project/SparkBuild.scala streaming/src/main/scala/spark/streaming/receivers/ActorReceiver.scala
| * | | | | Removed some unnecessary code and fixed dependenciesPrashant Sharma2013-07-111-1/+1
| | | | | |
* | | | | | Added GraphX to classpath.Reynold Xin2013-11-071-0/+1
| | | | | |
* | | | | | Merge remote-tracking branch 'spark-upstream/master'Ankur Dave2013-10-301-4/+18
|\ \ \ \ \ \ | | |_|_|_|/ | |/| | | | | | | | | | | | | | | | Conflicts: project/SparkBuild.scala
| * | | | | Merge pull request #66 from shivaram/sbt-assembly-depsMatei Zaharia2013-10-181-4/+18
| |\ \ \ \ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Add SBT target to assemble dependencies This pull request is an attempt to address the long assembly build times during development. Instead of rebuilding the assembly jar for every Spark change, this pull request adds a new SBT target `spark` that packages all the Spark modules and builds an assembly of the dependencies. So the work flow that should work now would be something like ``` ./sbt/sbt spark # Doing this once should suffice ## Make changes ./sbt/sbt compile ./sbt/sbt test or ./spark-shell ```
| | * | | | | Exclude assembly jar from classpath if using depsShivaram Venkataraman2013-10-161-10/+18
| | | | | | |
| | * | | | | Merge branch 'master' of https://github.com/apache/incubator-spark into ↵Shivaram Venkataraman2013-10-151-2/+0
| | |\ \ \ \ \ | | | | | | | | | | | | | | | | | | | | | | | | sbt-assembly-deps
| | * | | | | | Add new SBT target for dependency assemblyShivaram Venkataraman2013-10-091-0/+6
| | | |_|_|_|/ | | |/| | | |
* | | | | | | Merge branch 'master' of https://github.com/apache/incubator-spark into ↵Joseph E. Gonzalez2013-10-183-3/+39
|\| | | | | | | | | | | | | | | | | | | | | | | | | | | indexedrdd_graphx
| * | | | | | SPARK-627 , Implementing --config arguments in the scriptsKarthikTunga2013-10-161-1/+1
| | | | | | |
| * | | | | | SPARK-627 , Implementing --config arguments in the scriptsKarthikTunga2013-10-162-2/+2
| | | | | | |
| * | | | | | Implementing --config argument in the scriptsKarthikTunga2013-10-162-7/+10
| | | | | | |
| * | | | | | Merge branch 'master' of https://github.com/apache/incubator-sparkKarthikTunga2013-10-151-2/+0
| |\ \ \ \ \ \ | | | |/ / / / | | |/| | | | | | | | | | | Updating local branch
| * | | | | | SPARK-627 - reading --config argumentKarthikTunga2013-10-152-0/+33
| | |/ / / / | |/| | | |
* | | | | | merged with upstream changesJoseph E. Gonzalez2013-10-141-2/+0
|\ \ \ \ \ \ | | |/ / / / | |/| | | |
| * | | | | Address Matei's commentsAaron Davidson2013-10-051-2/+0
| | | | | |
| * | | | | Standalone Scheduler fault recoveryAaron Davidson2013-09-261-1/+1
| |/ / / / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Implements a basic form of Standalone Scheduler fault recovery. In particular, this allows faults to be manually recovered from by means of restarting the Master process on the same machine. This is the majority of the code necessary for general fault tolerance, which will first elect a leader and then recover the Master state. In order to enable fault recovery, the Master will persist a small amount of state related to the registration of Workers and Applications to disk. If the Master is started and sees that this state is still around, it will enter Recovery mode, during which time it will not schedule any new Executors on Workers (but it does accept the registration of new Clients and Workers). At this point, the Master attempts to reconnect to all Workers and Client applications that were registered at the time of failure. After confirming either the existence or nonexistence of all such nodes (within a certain timeout), the Master will exit Recovery mode and resume normal scheduling.
* | | | | Merging latest changes from spark main branchJoseph E. Gonzalez2013-09-1710-106/+59
|\| | | |
| * | | | Run script fixes for Windows after package & assembly changeMatei Zaharia2013-09-011-26/+20
| | | | |
| * | | | Initial work to rename package to org.apache.sparkMatei Zaharia2013-09-014-5/+5
| | | | |
| * | | | Print output from spark-daemon only when it fails to launchMatei Zaharia2013-08-314-8/+15
| | | | |
| * | | | Delete some code that was added back in a merge and print less info inMatei Zaharia2013-08-311-3/+0
| | | | | | | | | | | | | | | | | | | | spark-daemon
| * | | | Fix finding of assembly JAR, as well as some pointers to ./runMatei Zaharia2013-08-291-1/+1
| | | | |
| * | | | Change build and run instructions to use assembliesMatei Zaharia2013-08-292-72/+18
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This commit makes Spark invocation saner by using an assembly JAR to find all of Spark's dependencies instead of adding all the JARs in lib_managed. It also packages the examples into an assembly and uses that as SPARK_EXAMPLES_JAR. Finally, it replaces the old "run" script with two better-named scripts: "run-examples" for examples, and "spark-class" for Spark internal classes (e.g. REPL, master, etc). This is also designed to minimize the confusion people have in trying to use "run" to run their own classes; it's not meant to do that, but now at least if they look at it, they can modify run-examples to do a decent job for them. As part of this, Bagel's examples are also now properly moved to the examples package instead of bagel.
| * | | | Don't assume spark-examples JAR always existsJey Kottalam2013-08-181-2/+3
| | | | |
| * | | | Maven build now also works with YARNJey Kottalam2013-08-161-1/+1
| | | | |
generated by cgit v1.2.3 (git 2.39.1) at 2024-07-03 23:59:33 +0000