aboutsummaryrefslogtreecommitdiff
path: root/core
Commit message (Collapse)AuthorAgeFilesLines
* Update build version in masterPatrick Wendell2013-09-241-1/+1
|
* Merge branch 'master' of github.com:markhamstra/incubator-sparkReynold Xin2013-09-231-1/+0
|\
| * Removed repetative import; fixes hidden definition compiler warning.Mark Hamstra2013-09-031-1/+0
| |
* | Change Exception to NoSuchElementException and minor style fixjerryshao2013-09-221-6/+7
| |
* | Remove infix style and othersjerryshao2013-09-221-10/+8
| |
* | Refactor FairSchedulableBuilder:jerryshao2013-09-221-39/+53
| | | | | | | | | | 1. Configuration can be read from classpath if not set explicitly. 2. Add missing close handler.
* | Merge pull request #937 from jerryshao/localProperties-fixReynold Xin2013-09-212-2/+50
|\ \ | | | | | | Fix PR926 local properties issues in Spark Streaming like scenarios
| * | Add barrier for local properties unit test and fix some stylesjerryshao2013-09-222-3/+11
| | |
| * | Fix issue when local properties pass from parent to child threadjerryshao2013-09-182-2/+42
| | |
* | | After unit tests, clear port properties unconditionallyAnkur Dave2013-09-192-9/+7
|/ / | | | | | | | | | | | | | | | | | | | | | | In MapOutputTrackerSuite, the "remote fetch" test sets spark.driver.port and spark.hostPort, assuming that they will be cleared by LocalSparkContext. However, the test never sets sc, so it remains null, causing LocalSparkContext to skip clearing these properties. Subsequent tests therefore fail with java.net.BindException: "Address already in use". This commit makes LocalSparkContext clear the properties even if sc is null.
* | Changed localProperties to use ThreadLocal (not DynamicVariable).Kay Ousterhout2013-09-111-9/+9
| | | | | | | | | | | | The fact that DynamicVariable uses an InheritableThreadLocal can cause problems where the properties end up being shared across threads in certain circumstances.
* | Merge pull request #919 from mateiz/jets3tPatrick Wendell2013-09-111-0/+5
|\ \ | | | | | | Add explicit jets3t dependency, which is excluded in hadoop-client
| * | Add explicit jets3t dependency, which is excluded in hadoop-clientMatei Zaharia2013-09-101-0/+5
| | |
* | | Merge pull request #922 from pwendell/port-changePatrick Wendell2013-09-112-2/+2
|\ \ \ | | | | | | | | Change default port number from 3030 to 4030.
| * | | Change port from 3030 to 4040Patrick Wendell2013-09-112-2/+2
| |/ /
* / / SPARK-894 - Not all WebUI fields delivered VIA JSONDavid McCauley2013-09-111-1/+3
|/ /
* | Merge pull request #915 from ooyala/masterMatei Zaharia2013-09-091-1/+9
|\ \ | | | | | | Get rid of / improve ugly NPE when Utils.deleteRecursively() fails
| * | Style fix: put body of if within curly bracesEvan Chan2013-09-091-1/+3
| | |
| * | Print out more friendly error if listFiles() failsEvan Chan2013-09-091-1/+7
| | | | | | | | | | | | listFiles() could return null if the I/O fails, and this currently results in an ugly NPE which is hard to diagnose.
* | | Merge pull request #907 from stephenh/document_coalesce_shuffleMatei Zaharia2013-09-092-4/+27
|\ \ \ | | | | | | | | Add better docs for coalesce.
| * | | Use a set since shuffle could change order.Stephen Haberman2013-09-091-1/+1
| | | |
| * | | Reword 'evenly distributed' to 'distributed with a hash partitioner.Stephen Haberman2013-09-091-2/+2
| | | |
| * | | Add better docs for coalesce.Stephen Haberman2013-09-082-4/+27
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Include the useful tip that if shuffle=true, coalesce can actually increase the number of partitions. This makes coalesce more like a generic `RDD.repartition` operation. (Ideally this `RDD.repartition` could automatically choose either a coalesce or a shuffle if numPartitions was either less than or greater than, respectively, the current number of partitions.)
* | | | Add metrics-ganglia to core pom fileY.CORP.YAHOO.COM\tgraves2013-09-091-0/+4
| | | |
* | | | Merge pull request #890 from mridulm/masterMatei Zaharia2013-09-083-2/+17
|\ \ \ \ | | | | | | | | | | Fix hash bug
| * | | | Address review comments - rename toHash to nonNegativeHashMridul Muralidharan2013-09-043-3/+3
| | | | |
| * | | | Fix hash bug - caused failure after 35k stages, sighMridul Muralidharan2013-09-043-2/+17
| | | | |
* | | | | Merge pull request #909 from mateiz/exec-id-fixReynold Xin2013-09-082-7/+7
|\ \ \ \ \ | | | | | | | | | | | | Fix an instance where full standalone mode executor IDs were passed to
| * | | | | Fix an instance where full standalone mode executor IDs were passed toMatei Zaharia2013-09-082-7/+7
| | |/ / / | |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | StandaloneSchedulerBackend instead of the smaller IDs used within Spark (that lack the application name). This was reported by ClearStory in https://github.com/clearstorydata/spark/pull/9. Also fixed some messages that said slave instead of executor.
* | | | | Merge pull request #905 from mateiz/docs2Matei Zaharia2013-09-088-18/+19
|\ \ \ \ \ | | | | | | | | | | | | Job scheduling and cluster mode docs
| * | | | | Fix unit test failure due to changed defaultMatei Zaharia2013-09-081-1/+1
| | | | | |
| * | | | | More fair scheduler docs and property names.Matei Zaharia2013-09-086-12/+13
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Also changed uses of "job" terminology to "application" when they referred to an entire Spark program, to avoid confusion.
| * | | | | Work in progress:Matei Zaharia2013-09-084-5/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - Add job scheduling docs - Rename some fair scheduler properties - Organize intro page better - Link to Apache wiki for "contributing to Spark"
* | | | | | Merge pull request #906 from pwendell/ganglia-sinkPatrick Wendell2013-09-089-28/+114
|\ \ \ \ \ \ | |_|/ / / / |/| | | | | Clean-up of Metrics Code/Docs and Add Ganglia Sink
| * | | | | Adding sc name in metrics sourcePatrick Wendell2013-09-085-9/+14
| | | | | |
| * | | | | Adding more docs and some code cleanupPatrick Wendell2013-09-083-19/+18
| | | | | |
| * | | | | Ganglia sinkPatrick Wendell2013-09-081-0/+82
| | |/ / / | |/| | |
* | | | | Merge pull request #898 from ilikerps/660Matei Zaharia2013-09-081-3/+3
|\ \ \ \ \ | |_|/ / / |/| | | | SPARK-660: Add StorageLevel support in Python
| * | | | Export StorageLevel and refactorAaron Davidson2013-09-071-3/+3
| | | | |
| * | | | Remove reflection, hard-code StorageLevelsAaron Davidson2013-09-071-11/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The sc.StorageLevel -> StorageLevel pathway is a bit janky, but otherwise the shell would have to call a private method of SparkContext. Having StorageLevel available in sc also doesn't seem like the end of the world. There may be a better solution, though. As for creating the StorageLevel object itself, this seems to be the best way in Python 2 for creating singleton, enum-like objects: http://stackoverflow.com/questions/36932/how-can-i-represent-an-enum-in-python
| * | | | Memoize StorageLevels read from JVMAaron Davidson2013-09-061-1/+1
| | | | |
| * | | | SPARK-660: Add StorageLevel support in PythonAaron Davidson2013-09-051-0/+11
| | | | | | | | | | | | | | | | | | | | | | | | | It uses reflection... I am not proud of that fact, but it at least ensures compatibility (sans refactoring of the StorageLevel stuff).
* | | | | Fixed the bug that ResultTask was not properly deserializing outputId.Reynold Xin2013-09-071-2/+2
| |_|/ / |/| | |
* | | | Hot fix to resolve the compilation error caused by SPARK-821.Reynold Xin2013-09-061-1/+1
| | | |
* | | | Merge pull request #895 from ilikerps/821Patrick Wendell2013-09-057-7/+102
|\ \ \ \ | | | | | | | | | | SPARK-821: Don't cache results when action run locally on driver
| * | | | Reynold's second round of commentsAaron Davidson2013-09-052-17/+19
| | | | |
| * | | | Add unit test and address commentsAaron Davidson2013-09-055-6/+98
| | | | |
| * | | | SPARK-821: Don't cache results when action run locally on driverAaron Davidson2013-09-054-4/+5
| |/ / / | | | | | | | | | | | | | | | | | | | | Caching the results of local actions (e.g., rdd.first()) causes the driver to store entire partitions in its own memory, which may be highly constrained. This patch simply makes the CacheManager avoid caching the result of all locally-run computations.
* | | | Merge pull request #891 from xiajunluan/SPARK-864Matei Zaharia2013-09-051-1/+8
|\ \ \ \ | | | | | | | | | | [SPARK-864]DAGScheduler Exception if we delete Worker and StandaloneExecutorBackend then add Worker
| * | | | Fix bug SPARK-864Andrew xia2013-09-051-1/+8
| | |_|/ | |/| |
generated by cgit v1.2.3 (git 2.39.1) at 2024-09-08 10:24:26 +0000