spark - Mirror of Apache Spark

	Commit message (Collapse)	Author	Age	Files	Lines
...
\| \| * \| \| \| \| \| \| \| \| \|	Add warning message for spilling	Andrew Or	2013-12-31	1	-5/+9
\| \| \| \| \| \| \| \| \| \| \| \|
\| \| * \| \| \| \| \| \| \| \| \|	Address Aaron's and Jerry's comments	Andrew Or	2013-12-31	2	-5/+6
\| \| \|/ / / / / / / / /
\| \| * \| \| \| \| \| \| \| \|	Fix CheckpointSuite test fail	Andrew Or	2013-12-30	1	-2/+2
\| \| \| \| \| \| \| \| \| \| \|
\| \| * \| \| \| \| \| \| \| \|	Simplify merge logic based on the invariant that all spills contain unique keys	Andrew Or	2013-12-30	1	-37/+22
\| \| \| \| \| \| \| \| \| \| \|
\| \| * \| \| \| \| \| \| \| \|	Merge pull request from aarondav: Utilize DiskBlockManager pathway for temp ↵	Andrew Or	2013-12-30	3	-16/+49
\| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \|	file writing This gives us a couple advantages: - Uses spark.local.dir and randomly selects a directory/disk. - Ensure files are deleted on normal DiskBlockManager cleanup. - Availability of same stats as usual DiskBlockObjectWriter (currenty unused). Also enable basic cleanup when iterator is fully drained. Still requires cleanup for operations that fail or don't go through all elements.
\| \| * \| \| \| \| \| \| \| \|	Merge branch 'master' of github.com:andrewor14/incubator-spark	Andrew Or	2013-12-29	2	-9/+11
\| \| \|\ \ \ \ \ \ \ \ \
\| \| \| * \| \| \| \| \| \| \| \|	Use Comparator instead of Ordering	Aaron Davidson	2013-12-29	2	-9/+11
\| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \|	lower object creation costs
\| \| * \| \| \| \| \| \| \| \| \|	Add test suite for ExternalAppendOnlyMap	Andrew Or	2013-12-29	1	-0/+217
\| \| \| \| \| \| \| \| \| \| \| \|
\| \| * \| \| \| \| \| \| \| \| \|	Make serializer a parameter to ExternalAppendOnlyMap	Andrew Or	2013-12-29	2	-4/+4
\| \| \|/ / / / / / / / /
\| \| * \| \| \| \| \| \| \| \|	Address Aaron's comments	Andrew Or	2013-12-29	5	-88/+188
\| \| \| \| \| \| \| \| \| \| \|
\| \| * \| \| \| \| \| \| \| \|	Add Apache headers	Aaron Davidson	2013-12-27	3	-4/+54
\| \| \| \| \| \| \| \| \| \| \|
\| \| * \| \| \| \| \| \| \| \|	Rename spark.shuffle.buffer variables	Andrew Or	2013-12-27	1	-2/+2
\| \| \| \| \| \| \| \| \| \| \|
\| \| * \| \| \| \| \| \| \| \|	Final cleanup	Andrew Or	2013-12-26	4	-25/+28
\| \| \| \| \| \| \| \| \| \| \|
\| \| * \| \| \| \| \| \| \| \|	Use real serializer & manual ordering	Aaron Davidson	2013-12-26	1	-11/+27
\| \| \| \| \| \| \| \| \| \| \|
\| \| * \| \| \| \| \| \| \| \|	Return efficient iterator if no spillage happened	Aaron Davidson	2013-12-26	3	-9/+20
\| \| \| \| \| \| \| \| \| \| \|
\| \| * \| \| \| \| \| \| \| \|	Sort AppendOnlyMap in-place	Andrew Or	2013-12-26	2	-14/+51
\| \| \| \| \| \| \| \| \| \| \|
\| \| * \| \| \| \| \| \| \| \|	Fix streaming JavaAPISuite again	Andrew Or	2013-12-26	1	-8/+12
\| \| \| \| \| \| \| \| \| \| \|
\| \| * \| \| \| \| \| \| \| \|	Fix streaming JavaAPISuite that depended on order	Aaron Davidson	2013-12-26	1	-11/+16
\| \| \| \| \| \| \| \| \| \| \|
\| \| * \| \| \| \| \| \| \| \|	Allow Product2 rather than just tuple kv pairs	Aaron Davidson	2013-12-26	2	-8/+8
\| \| \| \| \| \| \| \| \| \| \|
\| \| * \| \| \| \| \| \| \| \|	Move maps to util, and refactor more	Andrew Or	2013-12-26	7	-23/+22
\| \| \| \| \| \| \| \| \| \| \|
\| \| * \| \| \| \| \| \| \| \|	SamplingSizeTracker + Map + test suite	Aaron Davidson	2013-12-26	5	-11/+204
\| \| \| \| \| \| \| \| \| \| \|
\| \| * \| \| \| \| \| \| \| \|	New minor edits	Andrew Or	2013-12-26	3	-54/+49
\| \| \| \| \| \| \| \| \| \| \|
\| \| * \| \| \| \| \| \| \| \|	Minor cleanup for Scala style	Aaron Davidson	2013-12-26	3	-55/+55
\| \| \| \| \| \| \| \| \| \| \|
\| \| * \| \| \| \| \| \| \| \|	Add toggle for ExternalAppendOnlyMap in Aggregator and CoGroupedRDD	Andrew Or	2013-12-26	3	-24/+65
\| \| \| \| \| \| \| \| \| \| \|
\| \| * \| \| \| \| \| \| \| \|	Provide for cases when mergeCombiners is not specified in ExternalAppendOnlyMap	Andrew Or	2013-12-26	2	-68/+121
\| \| \| \| \| \| \| \| \| \| \|
\| \| * \| \| \| \| \| \| \| \|	Refactor ExternalAppendOnlyMap to take in KVC instead of just KV	Andrew Or	2013-12-26	3	-76/+78
\| \| \| \| \| \| \| \| \| \| \|
\| \| * \| \| \| \| \| \| \| \|	Working ExternalAppendOnlyMap for both CoGroupedRDDs and Aggregator	Andrew Or	2013-12-26	4	-66/+61
\| \| \| \| \| \| \| \| \| \| \|
\| \| * \| \| \| \| \| \| \| \|	Working ExternalAppendOnlyMap for Aggregator, but not for CoGroupedRDD	Andrew Or	2013-12-26	4	-21/+182
\| \| \| \| \| \| \| \| \| \| \|
\| * \| \| \| \| \| \| \| \| \|	Merge pull request #369 from pillis/master	Reynold Xin	2014-01-10	2	-0/+52
\| \|\ \ \ \ \ \ \ \ \ \ \| \| \|_\|_\|_\|_\|_\|/ / / / \| \|/\| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \|	SPARK-961 Add a Vector.random() method Added method and testcases
\| \| * \| \| \| \| \| \| \| \|	SPARK-961. Add a Vector.random() method - update 1	Pillis	2014-01-10	2	-10/+12
\| \| \| \| \| \| \| \| \| \| \|
\| \| * \| \| \| \| \| \| \| \|	SPARK-961 Add a Vector.random() method	Pillis	2014-01-09	2	-0/+50
\| \| \| \| \| \| \| \| \| \| \|
* \| \| \| \| \| \| \| \| \| \|	Added waitForStop and stop to JavaStreamingContext.	Tathagata Das	2014-01-11	2	-3/+23
\| \| \| \| \| \| \| \| \| \| \|
* \| \| \| \| \| \| \| \| \| \|	Converted JobScheduler to use actors for event handling. Changed ↵	Tathagata Das	2014-01-11	17	-185/+485
\| \|_\|_\|_\|_\|/ / / / / \|/\| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \|	protected[streaming] to private[streaming] in StreamingContext and DStream. Added waitForStop to StreamingContext, and StreamingContextSuite.
* \| \| \| \| \| \| \| \| \|	Merge remote-tracking branch 'apache/master' into driver-test	Tathagata Das	2014-01-10	37	-105/+205
\|\\| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \|	Conflicts: streaming/src/main/scala/org/apache/spark/streaming/DStreamGraph.scala
\| * \| \| \| \| \| \| \| \|	Merge pull request #371 from tgravescs/yarn_client_addjar_misc_fixes	Thomas Graves	2014-01-10	8	-51/+105
\| \|\ \ \ \ \ \ \ \ \ \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \|	Yarn client addjar and misc fixes Fix the addJar functionality in yarn-client mode, add support for the other options supported in yarn-standalone mode, set the application type on yarn in hadoop 2.X, add documentation, change heartbeat interval to be same code as the yarn-standalone so it doesn't take so long to get containers and exit.
\| \| * \| \| \| \| \| \| \| \|	Update based on review comments	Thomas Graves	2014-01-09	4	-27/+23
\| \| \| \| \| \| \| \| \| \| \|
\| \| * \| \| \| \| \| \| \| \|	add application type	Thomas Graves	2014-01-09	1	-0/+1
\| \| \| \| \| \| \| \| \| \| \|
\| \| * \| \| \| \| \| \| \| \|	yarn-client addJar fix and misc other	Thomas Graves	2014-01-09	5	-37/+94
\| \| \| \| \| \| \| \| \| \| \|
\| * \| \| \| \| \| \| \| \| \|	Merge pull request #384 from pwendell/debug-logs	Patrick Wendell	2014-01-10	3	-4/+8
\| \|\ \ \ \ \ \ \ \ \ \ \| \| \|_\|_\|_\|_\|_\|/ / / / \| \|/\| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \|	Make DEBUG-level logs consummable. Removes two things that caused issues with the debug logs: (a) Internal polling in the DAGScheduler was polluting the logs. (b) The Scala REPL logs were really noisy.
\| \| * \| \| \| \| \| \| \| \|	Make DEBUG-level logs consummable.	Patrick Wendell	2014-01-10	3	-4/+8
\| \| \| \|_\|_\|_\|_\|/ / / \| \| \|/\| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \|	Removes two things that caused issues with the debug logs: (a) Internal polling in the DAGScheduler was polluting the logs. (b) The Scala REPL logs were really noisy.
\| * \| \| \| \| \| \| \| \|	Merge pull request #375 from mateiz/option-fix	Matei Zaharia	2014-01-09	1	-2/+2
\| \|\ \ \ \ \ \ \ \ \ \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \|	Fix bug added when we changed AppDescription.maxCores to an Option The Scala compiler warned about this -- we were comparing an Option against an integer now.
\| \| * \| \| \| \| \| \| \| \|	Fix bug added when we changed AppDescription.maxCores to an Option	Matei Zaharia	2014-01-09	1	-2/+2
\| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \|	The Scala compiler warned about this -- we were comparing an Option against an integer now.
\| * \| \| \| \| \| \| \| \| \|	Merge pull request #378 from pwendell/consolidate_on	Patrick Wendell	2014-01-09	2	-2/+2
\| \|\ \ \ \ \ \ \ \ \ \ \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \|	Enable shuffle consolidation by default. Bump this to being enabled for 0.9.0.
\| \| * \| \| \| \| \| \| \| \| \|	Enable shuffle consolidation by default.	Patrick Wendell	2014-01-09	2	-2/+2
\| \| \| \|/ / / / / / / / \| \| \|/\| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \|	Bump this to being enabled for 0.9.0.
\| * \| \| \| \| \| \| \| \| \|	Merge pull request #363 from pwendell/streaming-logs	Patrick Wendell	2014-01-09	21	-35/+81
\| \|\ \ \ \ \ \ \ \ \ \ \| \| \|_\|_\|_\|_\|_\|_\|/ / / \| \|/\| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \|	Set default logging to WARN for Spark streaming examples. This programatically sets the log level to WARN by default for streaming tests. If the user has already specified a log4j.properties file, the user's file will take precedence over this default.
\| \| * \| \| \| \| \| \| \| \|	Minor clean-up	Patrick Wendell	2014-01-09	1	-1/+1
\| \| \| \| \| \| \| \| \| \| \|
\| \| * \| \| \| \| \| \| \| \|	Set default logging to WARN for Spark streaming examples.	Patrick Wendell	2014-01-09	21	-34/+80
\| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \|	This programatically sets the log level to WARN by default for streaming tests. If the user has already specified a log4j.properties file, the user's file will take precedence over this default.
\| * \| \| \| \| \| \| \| \| \|	Merge pull request #353 from pwendell/ipython-simplify	Patrick Wendell	2014-01-09	2	-8/+4
\| \|\ \ \ \ \ \ \ \ \ \ \| \| \|_\|/ / / / / / / / \| \|/\| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \|	Simplify and fix pyspark script. This patch removes compatibility for IPython < 1.0 but fixes the launch script and makes it much simpler. I tested this using the three commands in the PySpark documentation page: 1. IPYTHON=1 ./pyspark 2. IPYTHON_OPTS="notebook" ./pyspark 3. IPYTHON_OPTS="notebook --pylab inline" ./pyspark There are two changes: - We rely on PYTHONSTARTUP env var to start PySpark - Removed the quotes around $IPYTHON_OPTS... having quotes gloms them together as a single argument passed to `exec` which seemed to cause ipython to fail (it instead expects them as multiple arguments).
\| \| * \| \| \| \| \| \| \| \|	Small fix suggested by josh	Patrick Wendell	2014-01-09	1	-0/+1
\| \| \| \| \| \| \| \| \| \| \|
\| \| * \| \| \| \| \| \| \| \|	Simplify and fix pyspark script.	Patrick Wendell	2014-01-07	2	-9/+4
\| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \|	This patch removes compatibility for IPython < 1.0 but fixes the launch script and makes it much simpler. I tested this using the three commands in the PySpark documentation page: 1. IPYTHON=1 ./pyspark 2. IPYTHON_OPTS="notebook" ./pyspark 3. IPYTHON_OPTS="notebook --pylab inline" ./pyspark There are two changes: - We rely on PYTHONSTARTUP env var to start PySpark - Removed the quotes around $IPYTHON_OPTS... having quotes gloms them together as a single argument passed to `exec` which seemed to cause ipython to fail (it instead expects them as multiple arguments).