| Commit message (Collapse) | Author | Age | Files | Lines | |
|---|---|---|---|---|---|
| * | Also add getConf to NewHadoopRDD | Mikhail Bautin | 2013-08-30 | 1 | -0/+3 |
| | | |||||
| * | Make HadoopRDD's configuration accessible | Mikhail Bautin | 2013-08-30 | 1 | -1/+3 |
| | | |||||
| * | Merge pull request #857 from mateiz/assembly | Matei Zaharia | 2013-08-29 | 4 | -4/+5 |
| |\ | | | | | Change build and run instructions to use assemblies | ||||
| | * | Update Maven build to create assemblies expected by new scripts | Matei Zaharia | 2013-08-29 | 3 | -28/+0 |
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This includes the following changes: - The "assembly" package now builds in Maven by default, and creates an assembly containing both hadoop-client and Spark, unlike the old BigTop distribution assembly that skipped hadoop-client - There is now a bigtop-dist package to build the old BigTop assembly - The repl-bin package is no longer built by default since the scripts don't reply on it; instead it can be enabled with -Prepl-bin - Py4J is now included in the assembly/lib folder as a local Maven repo, so that the Maven package can link to it - run-example now adds the original Spark classpath as well because the Maven examples assembly lists spark-core and such as provided - The various Maven projects add a spark-yarn dependency correctly | ||||
| | * | Fix finding of assembly JAR, as well as some pointers to ./run | Matei Zaharia | 2013-08-29 | 3 | -2/+3 |
| | | | |||||
| | * | Fix PySpark for assembly run and include it in dist | Matei Zaharia | 2013-08-29 | 3 | -0/+28 |
| | | | |||||
| | * | Change build and run instructions to use assemblies | Matei Zaharia | 2013-08-29 | 2 | -2/+2 |
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This commit makes Spark invocation saner by using an assembly JAR to find all of Spark's dependencies instead of adding all the JARs in lib_managed. It also packages the examples into an assembly and uses that as SPARK_EXAMPLES_JAR. Finally, it replaces the old "run" script with two better-named scripts: "run-examples" for examples, and "spark-class" for Spark internal classes (e.g. REPL, master, etc). This is also designed to minimize the confusion people have in trying to use "run" to run their own classes; it's not meant to do that, but now at least if they look at it, they can modify run-examples to do a decent job for them. As part of this, Bagel's examples are also now properly moved to the examples package instead of bagel. | ||||
| * | | Fix removed block zero size log reporting | jerryshao | 2013-08-30 | 1 | -2/+2 |
| |/ | |||||
| * | Merge pull request #871 from pwendell/expose-local | Patrick Wendell | 2013-08-28 | 1 | -1/+1 |
| |\ | | | | | Expose `isLocal` in SparkContext. | ||||
| | * | Make local variable public | Patrick Wendell | 2013-08-28 | 1 | -1/+1 |
| | | | |||||
| * | | Merge pull request #865 from tgravescs/fixtmpdir | Matei Zaharia | 2013-08-28 | 1 | -0/+22 |
| |\ \ | | | | | | | Spark on Yarn should use yarn approved directories for spark.local.dir and tmp | ||||
| | * | | Change Executor to only look at the env variable SPARK_YARN_MODE | Y.CORP.YAHOO.COM\tgraves | 2013-08-28 | 1 | -1/+1 |
| | | | | |||||
| | * | | Updated based on review comments. | Y.CORP.YAHOO.COM\tgraves | 2013-08-27 | 1 | -9/+6 |
| | | | | |||||
| | * | | Allow for Executors to have different directories then the Spark Master for Yarn | Y.CORP.YAHOO.COM\tgraves | 2013-08-27 | 1 | -0/+25 |
| | |/ | |||||
| * | | Added worker state to the cluster master JSON ui. | Reynold Xin | 2013-08-26 | 1 | -1/+2 |
| | | | |||||
| * | | Revert "Merge pull request #841 from rxin/json" | Reynold Xin | 2013-08-26 | 6 | -64/+70 |
| |/ | | | | | This reverts commit 1fb1b0992838c8cdd57eec45793e67a0490f1a52, reversing changes made to c69c48947d5102c81a9425cb380d861c3903685c. | ||||
| * | Merge pull request #854 from markhamstra/pomUpdate | Matei Zaharia | 2013-08-22 | 1 | -0/+4 |
| |\ | | | | | Synced sbt and maven builds to use the same dependencies, etc. | ||||
| | * | Synced sbt and maven builds | Mark Hamstra | 2013-08-21 | 1 | -0/+4 |
| | | | |||||
| * | | Merge pull request #832 from alig/coalesce | Matei Zaharia | 2013-08-22 | 4 | -46/+389 |
| |\ \ | |/ |/| | Coalesced RDD with locality | ||||
| | * | Merged in from upstream to use TaskLocation instead of strings | Ali Ghodsi | 2013-08-20 | 2 | -8/+11 |
| | | | |||||
| | * | added curly braces to make the code more consistent | Ali Ghodsi | 2013-08-20 | 1 | -1/+2 |
| | | | |||||
| | * | indent | Ali Ghodsi | 2013-08-20 | 1 | -1/+1 |
| | | | |||||
| | * | Bug in test fixed | Ali Ghodsi | 2013-08-20 | 1 | -3/+3 |
| | | | |||||
| | * | Added a test to make sure no locality preferences are ignored | Ali Ghodsi | 2013-08-20 | 1 | -0/+5 |
| | | | |||||
| | * | Simpler code | Ali Ghodsi | 2013-08-20 | 2 | -5/+4 |
| | | | |||||
| | * | simpler code | Ali Ghodsi | 2013-08-20 | 1 | -16/+7 |
| | | | |||||
| | * | Fixed almost all of Matei's feedback | Ali Ghodsi | 2013-08-20 | 2 | -31/+26 |
| | | | |||||
| | * | fixed Matei's comments | Ali Ghodsi | 2013-08-20 | 3 | -73/+99 |
| | | | |||||
| | * | making CoalescedRDDPartition public | Ali Ghodsi | 2013-08-20 | 1 | -2/+1 |
| | | | |||||
| | * | comment in the test to make it more understandable | Ali Ghodsi | 2013-08-20 | 1 | -1/+1 |
| | | | |||||
| | * | Coalescer now uses current preferred locations for derived RDDs. Made run() ↵ | Ali Ghodsi | 2013-08-20 | 4 | -34/+59 |
| | | | | | | | | | in DAGScheduler thread safe and added a method to be able to ask it for preferred locations. Added a similar method that wraps the former inside SparkContext. | ||||
| | * | added one test that will test a future functionality | Ali Ghodsi | 2013-08-20 | 1 | -1/+10 |
| | | | |||||
| | * | Added error messages to the tests to make failed tests less cryptic | Ali Ghodsi | 2013-08-20 | 1 | -7/+7 |
| | | | |||||
| | * | fixed matei's comments | Ali Ghodsi | 2013-08-20 | 1 | -15/+16 |
| | | | |||||
| | * | Made a function object that returns the coalesced groups | Ali Ghodsi | 2013-08-20 | 1 | -30/+35 |
| | | | |||||
| | * | several of Reynold's suggestions implemented | Ali Ghodsi | 2013-08-20 | 1 | -15/+14 |
| | | | |||||
| | * | space removed | Ali Ghodsi | 2013-08-20 | 1 | -1/+1 |
| | | | |||||
| | * | use count rather than foreach | Ali Ghodsi | 2013-08-20 | 1 | -2/+1 |
| | | | |||||
| | * | made preferredLocation a val of the surrounding case class | Ali Ghodsi | 2013-08-20 | 1 | -10/+3 |
| | | | |||||
| | * | Fix bug in tests | Ali Ghodsi | 2013-08-20 | 2 | -6/+6 |
| | | | |||||
| | * | Renamed split to partition | Ali Ghodsi | 2013-08-20 | 1 | -11/+11 |
| | | | |||||
| | * | word wrap before 100 chars per line | Ali Ghodsi | 2013-08-20 | 2 | -41/+51 |
| | | | |||||
| | * | added goals inline as comment | Ali Ghodsi | 2013-08-20 | 1 | -0/+21 |
| | | | |||||
| | * | Large scale load and locality tests for the coalesced partitions added | Ali Ghodsi | 2013-08-20 | 2 | -63/+118 |
| | | | |||||
| | * | Bug, should compute slack wrt parent partition size, not number of bins | Ali Ghodsi | 2013-08-20 | 1 | -2/+2 |
| | | | |||||
| | * | load balancing coalescer | Ali Ghodsi | 2013-08-20 | 2 | -11/+218 |
| | | | |||||
| * | | Removed meaningless types | Mark Hamstra | 2013-08-20 | 1 | -1/+1 |
| |/ | |||||
| * | Merge remote-tracking branch 'jey/hadoop-agnostic' | Matei Zaharia | 2013-08-20 | 29 | -2255/+178 |
| |\ | | | | | | | | | Conflicts: core/src/main/scala/spark/PairRDDFunctions.scala | ||||
| | * | Fix Maven build with Hadoop 0.23.9 | Jey Kottalam | 2013-08-18 | 1 | -0/+8 |
| | | | |||||
| | * | Maven build now also works with YARN | Jey Kottalam | 2013-08-16 | 1 | -70/+0 |
| | | | |||||
 |
index : spark | |
| Mirror of Apache Spark |
| aboutsummaryrefslogtreecommitdiff |