aboutsummaryrefslogtreecommitdiff
path: root/python
Commit message (Collapse)AuthorAgeFilesLines
* Fix PySpark unit tests on Python 2.6.Josh Rosen2013-08-142-19/+20
|
* Merge pull request #802 from stayhf/SPARK-760-PythonMatei Zaharia2013-08-121-0/+70
|\ | | | | Simple PageRank algorithm implementation in Python for SPARK-760
| * Code update for Matei's suggestionsstayhf2013-08-111-7/+9
| |
| * Simple PageRank algorithm implementation in Python for SPARK-760stayhf2013-08-101-0/+68
| |
* | Merge pull request #813 from AndreSchumacher/add_files_pysparkMatei Zaharia2013-08-121-1/+6
|\ \ | | | | | | Implementing SPARK-865: Add the equivalent of ADD_JARS to PySpark
| * | Implementing SPARK-865: Add the equivalent of ADD_JARS to PySparkAndre Schumacher2013-08-121-1/+6
| | | | | | | | | | | | Now ADD_FILES uses a comma as file name separator.
* | | Merge pull request #747 from mateiz/improved-lrMatei Zaharia2013-08-061-27/+26
|\ \ \ | | | | | | | | Update the Python logistic regression example
| * | | Fix string parsing and style in LRMatei Zaharia2013-07-311-1/+1
| | | |
| * | | Update the Python logistic regression example to read from a file andMatei Zaharia2013-07-291-27/+26
| | | | | | | | | | | | | | | | batch input records for more efficient NumPy computations
* | | | Do not inherit master's PYTHONPATH on workers.Josh Rosen2013-07-291-3/+2
|/ / / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This fixes SPARK-832, an issue where PySpark would not work when the master and workers used different SPARK_HOME paths. This change may potentially break code that relied on the master's PYTHONPATH being used on workers. To have custom PYTHONPATH additions used on the workers, users should set a custom PYTHONPATH in spark-env.sh rather than setting it in the shell.
* | | Merge branch 'master' of github.com:mesos/sparkMatei Zaharia2013-07-296-15/+9
|\ \ \
| * | | Some fixes to Python examples (style and package name for LR)Matei Zaharia2013-07-276-15/+9
| | |/ | |/|
* | | SPARK-815. Python parallelize() should split lists before batchingMatei Zaharia2013-07-291-2/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | One unfortunate consequence of this fix is that we materialize any collections that are given to us as generators, but this seems necessary to get reasonable behavior on small collections. We could add a batchSize parameter later to bypass auto-computation of batch size if this becomes a problem (e.g. if users really want to parallelize big generators nicely)
* | | Use None instead of empty string as it's slightly smaller/fasterMatei Zaharia2013-07-291-1/+1
| | |
* | | Allow python/run-tests to run from any directoryMatei Zaharia2013-07-291-0/+3
| | |
* | | Optimize Python foreach() to not return as many objectsMatei Zaharia2013-07-291-1/+5
| | |
* | | Optimize Python take() to not compute entire first partitionMatei Zaharia2013-07-291-6/+9
|/ /
* | Add Apache license headers and LICENSE and NOTICE filesMatei Zaharia2013-07-1619-1/+325
| |
* | Fixed PySpark perf regression by not using socket.makefile(), and improvedroot2013-07-011-18/+24
| | | | | | | | | | | | | | debuggability by letting "print" statements show up in the executor's stderr Conflicts: core/src/main/scala/spark/api/python/PythonRDD.scala
* | Fix reporting of PySpark exceptionsJey Kottalam2013-06-212-5/+19
| |
* | PySpark daemon: fix deadlock, improve error handlingJey Kottalam2013-06-211-17/+50
| |
* | Add tests and fixes for Python daemon shutdownJey Kottalam2013-06-213-22/+69
| |
* | Prefork Python worker processesJey Kottalam2013-06-212-32/+138
| |
* | Add Python timing instrumentationJey Kottalam2013-06-212-1/+19
| |
* | Fix Python saveAsTextFile doctest to not expect order to be preservedJey Kottalam2013-04-021-1/+1
| |
* | Fix argv handling in Python transitive closure exampleJey Kottalam2013-04-021-1/+1
| |
* | Change numSplits to numPartitions in PySpark.Josh Rosen2013-02-242-38/+38
| |
* | Add commutative requirement for 'reduce' to Python docstring.Mark Hamstra2013-02-091-2/+2
|/
* Remove unnecessary doctest __main__ methods.Josh Rosen2013-02-032-18/+0
|
* Fetch fewer objects in PySpark's take() method.Josh Rosen2013-02-031-0/+4
|
* Fix reporting of PySpark doctest failures.Josh Rosen2013-02-032-2/+6
|
* Use spark.local.dir for PySpark temp files (SPARK-580).Josh Rosen2013-02-012-10/+9
|
* Do not launch JavaGateways on workers (SPARK-674).Josh Rosen2013-02-014-18/+25
| | | | | | | | | | | The problem was that the gateway was being initialized whenever the pyspark.context module was loaded. The fix uses lazy initialization that occurs only when SparkContext instances are actually constructed. I also made the gateway and jvm variables private. This change results in ~3-4x performance improvement when running the PySpark unit tests.
* Fix stdout redirection in PySpark.Josh Rosen2013-02-012-2/+12
|
* SPARK-673: Capture and re-throw Python exceptionsPatrick Wendell2013-01-311-2/+8
| | | | | This patch alters the Python <-> executor protocol to pass on exception data when they occur in user Python code.
* Merge pull request #430 from pwendell/pyspark-guideMatei Zaharia2013-01-301-0/+1
|\ | | | | Minor improvements to PySpark docs
| * Make module help available in python shell.Patrick Wendell2013-01-301-0/+1
| | | | | | | | Also, adds a line in doc explaining how to use.
* | Replace old 'master' term with 'driver'.Stephen Haberman2013-01-251-1/+1
| |
* | Merge pull request #396 from JoshRosen/spark-653Matei Zaharia2013-01-242-14/+29
|\ \ | | | | | | Make PySpark AccumulatorParam an abstract base class
| * | Remove use of abc.ABCMeta due to cloudpickle issue.Josh Rosen2013-01-231-7/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | cloudpickle runs into issues while pickling subclasses of AccumulatorParam, which may be related to this Python issue: http://bugs.python.org/issue7689 This seems hard to fix and the ABCMeta wasn't necessary, so I removed it.
| * | Make AccumulatorParam an abstract base class.Josh Rosen2013-01-212-13/+31
| | |
* | | Allow PySpark's SparkFiles to be used from driverJosh Rosen2013-01-235-9/+63
| | | | | | | | | | | | Fix minor documentation formatting issues.
* | | Fix sys.path bug in PySpark SparkContext.addPyFileJosh Rosen2013-01-224-7/+41
| | |
* | | Don't download files to master's working directory.Josh Rosen2013-01-215-5/+70
|/ / | | | | | | | | | | | | This should avoid exceptions caused by existing files with different contents. I also removed some unused code.
* | Merge pull request #389 from JoshRosen/python_rdd_checkpointingMatei Zaharia2013-01-205-3/+116
|\ \ | | | | | | Add checkpointing to the Python API
| * | Clean up setup code in PySpark checkpointing testsJosh Rosen2013-01-202-16/+6
| | |
| * | Update checkpointing API docs in Python/Java.Josh Rosen2013-01-202-16/+12
| | |
| * | Add checkpointFile() and more tests to PySpark.Josh Rosen2013-01-203-2/+37
| | |
| * | Add RDD checkpointing to Python API.Josh Rosen2013-01-205-1/+93
| | |
* | | Fix PythonPartitioner equality; see SPARK-654.Josh Rosen2013-01-201-6/+11
|/ / | | | | | | | | | | PythonPartitioner did not take the Python-side partitioning function into account when checking for equality, which might cause problems in the future.
generated by cgit v1.2.3 (git 2.39.1) at 2024-07-28 19:18:50 +0000