aboutsummaryrefslogtreecommitdiff
path: root/core
Commit message (Collapse)AuthorAgeFilesLines
* re-enable YARN supportJey Kottalam2013-08-151-1/+13
|
* SparkEnv isn't available this early, and not needed anywayJey Kottalam2013-08-152-25/+0
|
* make SparkHadoopUtil a member of SparkEnvJey Kottalam2013-08-158-26/+31
|
* rename HadoopMapRedUtil => SparkHadoopMapRedUtil, HadoopMapReduceUtil => ↵Jey Kottalam2013-08-155-6/+7
| | | | SparkHadoopMapReduceUtil
* add commentJey Kottalam2013-08-151-4/+4
|
* dynamically detect hadoop versionJey Kottalam2013-08-152-8/+48
|
* remove core/src/hadoop{1,2} dirsJey Kottalam2013-08-156-104/+0
|
* move yarn to its own directoryJey Kottalam2013-08-1510-1864/+0
|
* More minor UI changes including code review feedback.Reynold Xin2013-08-156-16/+39
|
* Various UI improvements.Reynold Xin2013-08-1412-88/+83
|
* Renamed setCurrentJobDescription to setJobDescription.Reynold Xin2013-08-141-1/+1
|
* A few small scheduler / job description changes.Reynold Xin2013-08-144-70/+74
| | | | | | | | 1. Renamed SparkContext.addLocalProperty to setLocalProperty. And allow this function to unset a property. 2. Renamed SparkContext.setDescription to setCurrentJobDescription. 3. Throw an exception if the fair scheduler allocation file is invalid.
* Merge pull request #822 from pwendell/ui-featuresMatei Zaharia2013-08-146-27/+54
|\ | | | | Adding GC Stats to TaskMetrics (and three small fixes)
| * Style cleanup based on Matei feedbackPatrick Wendell2013-08-143-5/+4
| |
| * Small style clean-upPatrick Wendell2013-08-132-2/+2
| |
| * Correcting terminology in RDD pagePatrick Wendell2013-08-131-1/+1
| |
| * Correct sorting order for stagesPatrick Wendell2013-08-132-10/+6
| |
| * Capturing GC detials in TaskMetricsPatrick Wendell2013-08-134-10/+37
| |
| * Bug fix for display of shuffle read/write metrics.Patrick Wendell2013-08-131-6/+11
| | | | | | | | | | This fixes an error where empty cells are missing if a given task has no shuffle read/write.
* | Fixed 2 bugs in executor UI.Kay Ousterhout2013-08-131-12/+10
| | | | | | | | | | | | 1) UI crashed if the executor UI was loaded before any tasks started. 2) The total tasks was incorrectly reported due to using string (rather than int) arithmetic.
* | Merge pull request #821 from pwendell/print-launch-commandMatei Zaharia2013-08-131-1/+1
|\ \ | | | | | | Print run command to stderr rather than stdout
| * | Print run command to stderr rather than stdoutPatrick Wendell2013-08-131-1/+1
| | |
* | | Reuse the set of failed states rather than creating a new object each timeKay Ousterhout2013-08-131-1/+3
| | |
* | | Properly account for killed tasks.Kay Ousterhout2013-08-131-1/+1
| |/ |/| | | | | | | | | | | The TaskState class's isFinished() method didn't return true for KILLED tasks, which means some resources are never reclaimed for tasks that are killed. This also made it inconsistent with the isFinished() method used by CoarseMesosSchedulerBackend.
* | Slight change to pr-784Patrick Wendell2013-08-135-9/+10
| |
* | Merge pull request #784 from jerryshao/dev-metrics-servletPatrick Wendell2013-08-1314-35/+157
|\ \ | | | | | | Add MetricsServlet for Spark metrics system
| * | MetricsServlet code refactor according to commentsjerryshao2013-08-1211-43/+35
| | |
| * | Add MetricsServlet for Spark metrics systemjerryshao2013-08-1212-28/+158
| | |
* | | Merge pull request #807 from JoshRosen/guava-optionalMatei Zaharia2013-08-125-17/+87
|\ \ \ | |/ / |/| | Change scala.Option to Guava Optional in Java APIs
| * | Fix import organization.Josh Rosen2013-08-121-2/+1
| | |
| * | Change scala.Option to Guava Optional in Java APIs.Josh Rosen2013-08-115-17/+88
| | |
* | | Merge pull request #808 from pwendell/ui_compressed_bytesReynold Xin2013-08-111-1/+2
|\ \ \ | | | | | | | | Report compressed bytes read when calculating TaskMetrics
| * | | Report compressed bytes read when calculating TaskMetricsPatrick Wendell2013-08-111-1/+2
| | | |
* | | | Merge pull request #805 from woggle/hadoop-rdd-jobconfMatei Zaharia2013-08-111-2/+2
|\ \ \ \ | |_|/ / |/| | | Use new Configuration() instead of slower new JobConf() in SerializableWritable
| * | | Use new Configuration() instead of new JobConf() for ObjectWritable.Charles Reiss2013-08-101-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | JobConf's constructor loads default config files in some verisons of Hadoop, which is quite slow, and we only need the Configuration object to pass the correct ClassLoader.
* | | | Merge pull request #795 from mridulm/masterMatei Zaharia2013-08-102-7/+42
|\ \ \ \ | | | | | | | | | | Fix bug reported in PR 791 : a race condition in ConnectionManager and Connection
| * | | | Change line sizeMridul Muralidharan2013-08-081-5/+9
| | | | |
| * | | | Attempt to fix bug reported in PR 791 : a race condition in ↵Mridul Muralidharan2013-08-082-7/+38
| | | | | | | | | | | | | | | | | | | | ConnectionManager and Connection
* | | | | Merge remote-tracking branch 'origin/pr/792'Matei Zaharia2013-08-1010-259/+279
|\ \ \ \ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Conflicts: core/src/main/scala/spark/ui/jobs/IndexPage.scala core/src/main/scala/spark/ui/jobs/StagePage.scala
| * | | | | Shortened names, as per Matei's suggestionKay Ousterhout2013-08-102-14/+13
| | | | | |
| * | | | | Only print event queue full error message onceKay Ousterhout2013-08-091-1/+3
| | | | | |
| * | | | | Style fix: removing unnecessary return typeKay Ousterhout2013-08-091-6/+6
| | | | | |
| * | | | | Style fixes based on code reviewKay Ousterhout2013-08-093-132/+110
| | | | | |
| * | | | | Refactored SparkListener to process all events asynchronously.Kay Ousterhout2013-08-099-165/+196
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This commit fixes issues where SparkListeners that take a while to process events slow the DAGScheduler. This commit also fixes a bug in the UI where if a user goes to a web page of a stage that does not exist, they can create a memory leak (granted, this is not an issue at small scale -- probably only an issue if someone actively tried to DOS the UI).
| * | | | | Fixed 3 bugs that caused UI to crash (including SPARK-810).Kay Ousterhout2013-08-072-169/+191
| |/ / / / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | One bug caused the UI to crash if you try to look at a job's status before any of the tasks have finished. The second bug was a concurrency issue where two different threads (the scheduling thread and a UI thread) could be reading/updating the data structures in JobProgressListener concurrently. The third bug mis-used an Option, also causing the UI to crash under certain conditions.
* | | | | Merge pull request #785 from anfeng/masterPatrick Wendell2013-08-101-0/+25
|\ \ \ \ \ | | | | | | | | | | | | expose HDFS file system stats via Executor metrics
| * | | | | reformat registerFileSystemStat()anfeng2013-08-061-1/+3
| | | | | |
| * | | | | expose HDFS file system stats via Executor metricsanfeng2013-08-061-0/+23
| | | | | |
* | | | | | Merge pull request #786 from shivaram/mllib-javaMatei Zaharia2013-08-091-0/+5
|\ \ \ \ \ \ | |_|_|_|/ / |/| | | | | Java fixes, tests and examples for ALS, KMeans
| * | | | | Java examples, tests for KMeans and ALSShivaram Venkataraman2013-08-061-0/+5
| |/ / / / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - Changes ALS to accept RDD[Rating] instead of (Int, Int, Double) making it easier to call from Java - Renames class methods from `train` to `run` to enable static methods to be called from Java. - Add unit tests which check if both static / class methods can be called. - Also add examples which port the main() function in ALS, KMeans to the examples project. Couple of minor changes to existing code: - Add a toJavaRDD method in RDD to convert scala RDD to java RDD easily - Workaround a bug where using double[] from Java leads to class cast exception in KMeans init
generated by cgit v1.2.3 (git 2.39.1) at 2024-07-28 22:09:39 +0000