aboutsummaryrefslogtreecommitdiff
Commit message (Collapse)AuthorAgeFilesLines
...
| | | * | | | Merge branch 'reorgscripts' into scripts-reorgshane-huang2013-09-2740-87/+175
| | | |\ \ \ \
| | | | * | | | rm bin/spark.cmd as we don't have windows test environment. Will added it ↵shane-huang2013-09-261-27/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | later if needed Signed-off-by: shane-huang <shengsheng.huang@intel.com>
| | | | * | | | fix paths and change spark to use APP_MEM as application driver memory ↵shane-huang2013-09-263-35/+10
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | instead of SPARK_MEM, user should add application jars to SPARK_CLASSPATH Signed-off-by: shane-huang <shengsheng.huang@intel.com>
| | | | * | | | fix pathshane-huang2013-09-261-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Signed-off-by: shane-huang <shengsheng.huang@intel.com>
| | | | * | | | add scripts in binshane-huang2013-09-2312-17/+163
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Signed-off-by: shane-huang <shengsheng.huang@intel.com>
| | | | * | | | moved user scripts to bin foldershane-huang2013-09-2311-0/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Signed-off-by: shane-huang <shengsheng.huang@intel.com>
| | | | * | | | add admin scripts to sbinshane-huang2013-09-2314-47/+47
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Signed-off-by: shane-huang <shengsheng.huang@intel.com>
| | | | * | | | added spark-class and spark-executor to sbinshane-huang2013-09-2314-22/+16
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Signed-off-by: shane-huang <shengsheng.huang@intel.com>
| * | | | | | | Merge pull request #285 from colorant/yarn-refactorPatrick Wendell2014-01-0236-1226/+189
| |\ \ \ \ \ \ \ | | |_|_|_|/ / / | |/| | | | | | | | | | | | | | Yarn refactor
| | * | | | | | fix docs for yarnRaymond Liu2014-01-032-5/+2
| | | | | | | |
| | * | | | | | minor fix for loginfoRaymond Liu2014-01-031-1/+1
| | | | | | | |
| | * | | | | | move duplicate pom config into parent pomRaymond Liu2014-01-033-179/+84
| | | | | | | |
| | * | | | | | Using name yarn-alpha/yarn instead of yarn-2.0/yarn-2.2Raymond Liu2014-01-0318-30/+30
| | | | | | | |
| | * | | | | | Add yarn/common/src/test dir in building scriptRaymond Liu2014-01-031-0/+7
| | | | | | | |
| | * | | | | | Fix yarn/README.mdRaymond Liu2014-01-031-6/+4
| | | | | | | |
| | * | | | | | Clean up unused files for yarnRaymond Liu2014-01-034-311/+0
| | | | | | | |
| | * | | | | | Fix pom for build yarn/2.x with yarn/common into one jarRaymond Liu2014-01-034-36/+202
| | | | | | | |
| | * | | | | | Use unmanaged source dir to include common yarn codeRaymond Liu2014-01-031-11/+15
| | | | | | | |
| | * | | | | | merge yarn/scheduler yarn/common code into one directoryRaymond Liu2014-01-033-0/+0
| | | | | | | |
| | * | | | | | Need to send dummy hello message to actually estabilish akka connection.Raymond Liu2014-01-032-0/+4
| | | | | | | |
| | * | | | | | A few clean up for yarn 2.0 codeRaymond Liu2014-01-032-8/+7
| | | | | | | |
| | * | | | | | Update maven build documentationRaymond Liu2014-01-032-8/+4
| | | | | | | |
| | * | | | | | Fix yarn/README.md and update docs/running-on-yarn.mdRaymond Liu2014-01-032-3/+1
| | | | | | | |
| | * | | | | | Add README for yarn modulesRaymond Liu2014-01-031-0/+16
| | | | | | | |
| | * | | | | | some code clean up for Yarn 2.2Raymond Liu2014-01-032-3/+3
| | | | | | | |
| | * | | | | | Fix pom file for scala binary versionRaymond Liu2014-01-036-8/+8
| | | | | | | |
| | * | | | | | Fix yarn/assemble pom fileRaymond Liu2014-01-032-0/+75
| | | | | | | |
| | * | | | | | Change profile name new-yarn to hadoop2.2-yarnRaymond Liu2014-01-034-4/+4
| | | | | | | |
| | * | | | | | Fix pom for yarn code reorgnaize commitRaymond Liu2014-01-038-535/+264
| | | | | | | |
| | * | | | | | Reorganize yarn related codes into sub projects to remove duplicate files.Raymond Liu2014-01-0330-957/+337
| |/ / / / / /
| | | | | * | Changes on top of Prashant's patch.Patrick Wendell2014-01-0310-72/+42
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Closes #316
| | | | | * | Restored the previously removed testPrashant Sharma2014-01-031-1/+12
| | | | | | |
| | | | | * | fixed review commentsPrashant Sharma2014-01-039-21/+44
| | | | | | |
| | | | | * | Merge branch 'master' into spark-1002-remove-jarsPrashant Sharma2014-01-03149-1221/+2291
| | | | | |\ \ | | |_|_|_|/ / | |/| | | | |
| * | | | | | Merge pull request #323 from tgravescs/sparkconf_yarn_fixPatrick Wendell2014-01-0210-113/+101
| |\ \ \ \ \ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | fix spark on yarn after the sparkConf changes This fixes it so that spark on yarn now compiles and works after the sparkConf changes. There are also other issues I discovered along the way that are broken: - mvn builds for yarn don't assemble correctly - unset SPARK_EXAMPLES_JAR isn't handled properly anymore - I'm pretty sure spark.conf doesn't actually work as its not distributed with yarn those things can be fixed in separate pr unless others disagree.
| | * | | | | | fix yarn-clientThomas Graves2014-01-022-8/+10
| | | | | | | |
| | * | | | | | Fix yarn build after sparkConf changesThomas Graves2014-01-0210-109/+95
| | |/ / / / /
| * | | | | | Merge pull request #320 from kayousterhout/erroneous_failed_msgReynold Xin2014-01-022-12/+15
| |\ \ \ \ \ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Remove erroneous FAILED state for killed tasks. Currently, when tasks are killed, the Executor first sends a status update for the task with a "KILLED" state, and then sends a second status update with a "FAILED" state saying that the task failed due to an exception. The second FAILED state is misleading/unncessary, and occurs due to a NonLocalReturnControl Exception that gets thrown due to the way we kill tasks. This commit eliminates that problem. I'm not at all sure that this is the best way to fix this problem, so alternate suggestions welcome. @rxin guessing you're the right person to look at this.
| | * | | | | | Remove erroneous FAILED state for killed tasks.Kay Ousterhout2014-01-022-12/+15
| | |/ / / / / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Currently, when tasks are killed, the Executor first sends a status update for the task with a "KILLED" state, and then sends a second status update with a "FAILED" state saying that the task failed due to an exception. The second FAILED state is misleading/unncessary, and occurs due to a NonLocalReturnControl Exception that gets thrown due to the way we kill tasks. This commit eliminates that problem.
| * | | | | | Merge pull request #297 from tdas/window-improvementPatrick Wendell2014-01-029-172/+388
| |\ \ \ \ \ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Improvements to DStream window ops and refactoring of Spark's CheckpointSuite - Added a new RDD - PartitionerAwareUnionRDD. Using this RDD, one can take multiple RDDs partitioned by the same partitioner and unify them into a single RDD while preserving the partitioner. So m RDDs with p partitions each will be unified to a single RDD with p partitions and the same partitioner. The preferred location for each partition of the unified RDD will be the most common preferred location of the corresponding partitions of the parent RDDs. For example, location of partition 0 of the unified RDD will be where most of partition 0 of the parent RDDs are located. - Improved the performance of DStream's reduceByKeyAndWindow and groupByKeyAndWindow. Both these operations work by doing per-batch reduceByKey/groupByKey and then using PartitionerAwareUnionRDD to union the RDDs across the window. This eliminates a shuffle related to the window operation, which can reduce batch processing time by 30-40% for simple workloads. - Fixed bugs and simplified Spark's CheckpointSuite. Some of the tests were incorrect and unreliable. Added missing tests for ZippedRDD. I can go into greater detail if necessary. - Added mapSideCombine option to combineByKeyAndWindow.
| | * | | | | | Added Apache boilerplate and class docs to PartitionerAwareUnionRDD.Tathagata Das2013-12-261-3/+33
| | | | | | | |
| | * | | | | | Removed unncessary options from WindowedDStream.Tathagata Das2013-12-261-5/+3
| | | | | | | |
| | * | | | | | Merge branch 'apache-master' into window-improvementTathagata Das2013-12-2650-1548/+1841
| | |\ \ \ \ \ \
| | * \ \ \ \ \ \ Merge branch 'master' into window-improvementTathagata Das2013-12-2637-123/+465
| | |\ \ \ \ \ \ \
| | * | | | | | | | Updated groupByKeyAndWindow to be computed incrementally, and added ↵Tathagata Das2013-12-265-12/+34
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | mapSideCombine to combineByKeyAndWindow.
| | * | | | | | | | Fixed bug in PartitionAwareUnionRDDTathagata Das2013-12-261-6/+9
| | | | | | | | | |
| | * | | | | | | | Merge branch 'scheduler-update' into window-improvementTathagata Das2013-12-234-5/+32
| | |\ \ \ \ \ \ \ \
| | * | | | | | | | | Added tests for PartitionerAwareUnionRDD in the CheckpointSuite. Refactored ↵Tathagata Das2013-12-203-170/+231
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | CheckpointSuite to make the tests simpler and more reliable. Added missing test for ZippedRDD.
| | * | | | | | | | | Merge branch 'scheduler-update' into window-improvementTathagata Das2013-12-19306-4277/+10714
| | |\ \ \ \ \ \ \ \ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Conflicts: streaming/src/main/scala/org/apache/spark/streaming/dstream/WindowedDStream.scala
| | * | | | | | | | | | Added flag in window operation to use partition awaare union.Tathagata Das2013-11-211-1/+3
| | | | | | | | | | | |
generated by cgit v1.2.3 (git 2.39.1) at 2024-09-18 01:17:45 +0000