aboutsummaryrefslogtreecommitdiff
path: root/python
Commit message (Collapse)AuthorAgeFilesLines
* Merge branch 'master' of git://github.com/mesos/spark into scala-2.10Prashant Sharma2013-09-155-1/+78
|\ | | | | | | | | | | Conflicts: core/src/main/scala/org/apache/spark/SparkContext.scala project/SparkBuild.scala
| * Whoopsy daisyAaron Davidson2013-09-081-1/+0
| |
| * Export StorageLevel and refactorAaron Davidson2013-09-075-26/+62
| |
| * Remove reflection, hard-code StorageLevelsAaron Davidson2013-09-072-24/+26
| | | | | | | | | | | | | | | | | | | | | | The sc.StorageLevel -> StorageLevel pathway is a bit janky, but otherwise the shell would have to call a private method of SparkContext. Having StorageLevel available in sc also doesn't seem like the end of the world. There may be a better solution, though. As for creating the StorageLevel object itself, this seems to be the best way in Python 2 for creating singleton, enum-like objects: http://stackoverflow.com/questions/36932/how-can-i-represent-an-enum-in-python
| * Memoize StorageLevels read from JVMAaron Davidson2013-09-061-2/+9
| |
| * SPARK-660: Add StorageLevel support in PythonAaron Davidson2013-09-053-1/+34
| | | | | | | | | | It uses reflection... I am not proud of that fact, but it at least ensures compatibility (sans refactoring of the StorageLevel stuff).
* | Merged with masterPrashant Sharma2013-09-0625-98/+948
|\|
| * Add missing license headers found with RATMatei Zaharia2013-09-021-1/+18
| |
| * Exclude some private modules in epydocMatei Zaharia2013-09-021-0/+1
| |
| * Further fixes to get PySpark to work on WindowsMatei Zaharia2013-09-021-5/+12
| |
| * Allow PySpark to launch worker.py directly on WindowsMatei Zaharia2013-09-011-4/+7
| |
| * Move some classes to more appropriate packages:Matei Zaharia2013-09-011-2/+2
| | | | | | | | | | | | * RDD, *RDDFunctions -> org.apache.spark.rdd * Utils, ClosureCleaner, SizeEstimator -> org.apache.spark.util * JavaSerializer, KryoSerializer -> org.apache.spark.serializer
| * Add banner to PySpark and make wordcount output nicerMatei Zaharia2013-09-012-1/+14
| |
| * Initial work to rename package to org.apache.sparkMatei Zaharia2013-09-013-5/+5
| |
| * Merge pull request #861 from AndreSchumacher/pyspark_sampling_functionMatei Zaharia2013-08-312-7/+167
| |\ | | | | | | Pyspark sampling function
| | * RDD sample() and takeSample() prototypes for PySparkAndre Schumacher2013-08-282-7/+167
| | |
| * | Merge pull request #870 from JoshRosen/spark-885Matei Zaharia2013-08-311-1/+5
| |\ \ | | | | | | | | Don't send SIGINT / ctrl-c to Py4J gateway subprocess
| | * | Don't send SIGINT to Py4J gateway subprocess.Josh Rosen2013-08-281-1/+5
| | |/ | | | | | | | | | | | | | | | | | | | | | | | | This addresses SPARK-885, a usability issue where PySpark's Java gateway process would be killed if the user hit ctrl-c. Note that SIGINT still won't cancel the running s This fix is based on http://stackoverflow.com/questions/5045771
| * | Merge pull request #869 from AndreSchumacher/subtractMatei Zaharia2013-08-301-0/+37
| |\ \ | | | | | | | | PySpark: implementing subtractByKey(), subtract() and keyBy()
| | * | PySpark: implementing subtractByKey(), subtract() and keyBy()Andre Schumacher2013-08-281-0/+37
| | |/
| * | Fix PySpark for assembly run and include it in distMatei Zaharia2013-08-291-0/+0
| | |
| * | Change build and run instructions to use assembliesMatei Zaharia2013-08-291-1/+1
| |/ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This commit makes Spark invocation saner by using an assembly JAR to find all of Spark's dependencies instead of adding all the JARs in lib_managed. It also packages the examples into an assembly and uses that as SPARK_EXAMPLES_JAR. Finally, it replaces the old "run" script with two better-named scripts: "run-examples" for examples, and "spark-class" for Spark internal classes (e.g. REPL, master, etc). This is also designed to minimize the confusion people have in trying to use "run" to run their own classes; it's not meant to do that, but now at least if they look at it, they can modify run-examples to do a decent job for them. As part of this, Bagel's examples are also now properly moved to the examples package instead of bagel.
| * Implementing SPARK-838: Add DoubleRDDFunctions methods to PySparkAndre Schumacher2013-08-212-1/+168
| |
| * Implementing SPARK-878 for PySpark: adding zip and egg files to context and ↵Andre Schumacher2013-08-165-5/+37
| | | | | | | | passing it down to workers which add these to their sys.path
| * Fix PySpark unit tests on Python 2.6.Josh Rosen2013-08-142-19/+20
| |
| * Merge pull request #802 from stayhf/SPARK-760-PythonMatei Zaharia2013-08-121-0/+70
| |\ | | | | | | Simple PageRank algorithm implementation in Python for SPARK-760
| | * Code update for Matei's suggestionsstayhf2013-08-111-7/+9
| | |
| | * Simple PageRank algorithm implementation in Python for SPARK-760stayhf2013-08-101-0/+68
| | |
| * | Merge pull request #813 from AndreSchumacher/add_files_pysparkMatei Zaharia2013-08-121-1/+6
| |\ \ | | | | | | | | Implementing SPARK-865: Add the equivalent of ADD_JARS to PySpark
| | * | Implementing SPARK-865: Add the equivalent of ADD_JARS to PySparkAndre Schumacher2013-08-121-1/+6
| | | | | | | | | | | | | | | | Now ADD_FILES uses a comma as file name separator.
| * | | Merge pull request #747 from mateiz/improved-lrMatei Zaharia2013-08-061-27/+26
| |\ \ \ | | | | | | | | | | Update the Python logistic regression example
| | * | | Fix string parsing and style in LRMatei Zaharia2013-07-311-1/+1
| | | | |
| | * | | Update the Python logistic regression example to read from a file andMatei Zaharia2013-07-291-27/+26
| | | | | | | | | | | | | | | | | | | | batch input records for more efficient NumPy computations
| * | | | Do not inherit master's PYTHONPATH on workers.Josh Rosen2013-07-291-3/+2
| |/ / / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This fixes SPARK-832, an issue where PySpark would not work when the master and workers used different SPARK_HOME paths. This change may potentially break code that relied on the master's PYTHONPATH being used on workers. To have custom PYTHONPATH additions used on the workers, users should set a custom PYTHONPATH in spark-env.sh rather than setting it in the shell.
| * | | Merge branch 'master' of github.com:mesos/sparkMatei Zaharia2013-07-296-15/+9
| |\ \ \
| | * | | Some fixes to Python examples (style and package name for LR)Matei Zaharia2013-07-276-15/+9
| | | |/ | | |/|
| * | | SPARK-815. Python parallelize() should split lists before batchingMatei Zaharia2013-07-291-2/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | One unfortunate consequence of this fix is that we materialize any collections that are given to us as generators, but this seems necessary to get reasonable behavior on small collections. We could add a batchSize parameter later to bypass auto-computation of batch size if this becomes a problem (e.g. if users really want to parallelize big generators nicely)
| * | | Use None instead of empty string as it's slightly smaller/fasterMatei Zaharia2013-07-291-1/+1
| | | |
| * | | Allow python/run-tests to run from any directoryMatei Zaharia2013-07-291-0/+3
| | | |
| * | | Optimize Python foreach() to not return as many objectsMatei Zaharia2013-07-291-1/+5
| | | |
| * | | Optimize Python take() to not compute entire first partitionMatei Zaharia2013-07-291-6/+9
| |/ /
| * | Add Apache license headers and LICENSE and NOTICE filesMatei Zaharia2013-07-1619-1/+325
| | |
* | | PySpark: replacing class manifest by class tag for Scala 2.10.2 inside rdd.pyAndre Schumacher2013-08-301-2/+2
|/ /
* | Fixed PySpark perf regression by not using socket.makefile(), and improvedroot2013-07-011-18/+24
| | | | | | | | | | | | | | debuggability by letting "print" statements show up in the executor's stderr Conflicts: core/src/main/scala/spark/api/python/PythonRDD.scala
* | Fix reporting of PySpark exceptionsJey Kottalam2013-06-212-5/+19
| |
* | PySpark daemon: fix deadlock, improve error handlingJey Kottalam2013-06-211-17/+50
| |
* | Add tests and fixes for Python daemon shutdownJey Kottalam2013-06-213-22/+69
| |
* | Prefork Python worker processesJey Kottalam2013-06-212-32/+138
| |
* | Add Python timing instrumentationJey Kottalam2013-06-212-1/+19
| |
* | Fix Python saveAsTextFile doctest to not expect order to be preservedJey Kottalam2013-04-021-1/+1
| |
generated by cgit v1.2.3 (git 2.39.1) at 2024-08-08 13:35:33 +0000