| Commit message (Collapse) | Author | Age | Files | Lines |
|
|
|
|
|
|
|
| |
Current schema inference for local python collections halts as soon as there are no NullTypes. This is different than when we specify a sampling ratio of 1.0 on a distributed collection. This could result in incomplete schema information.
Author: Holden Karau <holden@us.ibm.com>
Closes #10275 from holdenk/SPARK-12300-fix-schmea-inferance-on-local-collections.
|
|
|
|
|
|
|
|
|
|
| |
Python API
The semantics of Python countByValue is different from Scala API, it is more like countDistinctValue, so here change to make it consistent with Scala/Java API.
Author: jerryshao <sshao@hortonworks.com>
Closes #10350 from jerryshao/SPARK-12353.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
After reading the JIRA https://issues.apache.org/jira/browse/SPARK-12520, I double checked the code.
For example, users can do the Equi-Join like
```df.join(df2, 'name', 'outer').select('name', 'height').collect()```
- There exists a bug in 1.5 and 1.4. The code just ignores the third parameter (join type) users pass. However, the join type we called is `Inner`, even if the user-specified type is the other type (e.g., `Outer`).
- After a PR: https://github.com/apache/spark/pull/8600, the 1.6 does not have such an issue, but the description has not been updated.
Plan to submit another PR to fix 1.5 and issue an error message if users specify a non-inner join type when using Equi-Join.
Author: gatorsmile <gatorsmile@gmail.com>
Closes #10477 from gatorsmile/pyOuterJoin.
|
|
|
|
|
|
|
|
|
|
| |
scaler model
Some methods are missing, such as ways to access the std, mean, etc. This PR is for feature parity for pyspark.mllib.feature.StandardScaler & StandardScalerModel.
Author: Holden Karau <holden@us.ibm.com>
Closes #10298 from holdenk/SPARK-12296-feature-parity-pyspark-mllib-StandardScalerModel.
|
|
|
|
|
|
| |
Author: pshearer <pshearer@massmutual.com>
Closes #10414 from pshearer/patch-1.
|
|
|
|
|
|
|
|
|
|
| |
No jira is created since this is a trivial change.
davies Please help review it
Author: Jeff Zhang <zjffdu@apache.org>
Closes #10143 from zjffdu/pyspark_typo.
|
|
|
|
|
|
|
|
| |
Added catch for casting Long to Int exception when PySpark ALS Ratings are serialized. It is easy to accidentally use Long IDs for user/product and before, it would fail with a somewhat cryptic "ClassCastException: java.lang.Long cannot be cast to java.lang.Integer." Now if this is done, a more descriptive error is shown, e.g. "PickleException: Ratings id 1205640308657491975 exceeds max integer value of 2147483647."
Author: Bryan Cutler <bjcutler@us.ibm.com>
Closes #9361 from BryanCutler/als-pyspark-long-id-error-SPARK-10158.
|
|
|
|
|
|
|
|
| |
Fix mistake doc of join type for ```dataframe.join```.
Author: Yanbo Liang <ybliang8@gmail.com>
Closes #10378 from yanboliang/leftsemi.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The current default storage level of Python persist API is MEMORY_ONLY_SER. This is different from the default level MEMORY_ONLY in the official document and RDD APIs.
davies Is this inconsistency intentional? Thanks!
Updates: Since the data is always serialized on the Python side, the storage levels of JAVA-specific deserialization are not removed, such as MEMORY_ONLY.
Updates: Based on the reviewers' feedback. In Python, stored objects will always be serialized with the [Pickle](https://docs.python.org/2/library/pickle.html) library, so it does not matter whether you choose a serialized level. The available storage levels in Python include `MEMORY_ONLY`, `MEMORY_ONLY_2`, `MEMORY_AND_DISK`, `MEMORY_AND_DISK_2`, `DISK_ONLY`, `DISK_ONLY_2` and `OFF_HEAP`.
Author: gatorsmile <gatorsmile@gmail.com>
Closes #10092 from gatorsmile/persistStorageLevel.
|
|
|
|
|
|
|
|
| |
Since we rename the column name from ```text``` to ```value``` for DataFrame load by ```SQLContext.read.text```, we need to update doc.
Author: Yanbo Liang <ybliang8@gmail.com>
Closes #10349 from yanboliang/text-value.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
when invFunc is None
when invFunc is None, `reduceByKeyAndWindow(func, None, winsize, slidesize)` is equivalent to
reduceByKey(func).window(winsize, slidesize).reduceByKey(winsize, slidesize)
and no checkpoint is necessary. The corresponding Scala code does exactly that, but Python code always creates a windowed stream with obligatory checkpointing. The patch fixes this.
I do not know how to unit-test this.
Author: David Tolpin <david.tolpin@gmail.com>
Closes #9888 from dtolpin/master.
|
|
|
|
|
|
|
|
| |
MLlib should use SQLContext.getOrCreate() instead of creating new SQLContext.
Author: Davies Liu <davies@databricks.com>
Closes #10338 from davies/create_context.
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Extend CrossValidator with HasSeed in PySpark.
This PR replaces [https://github.com/apache/spark/pull/7997]
CC: yanboliang thunterdb mmenestret Would one of you mind taking a look? Thanks!
Author: Joseph K. Bradley <joseph@databricks.com>
Author: Martin MENESTRET <mmenestret@ippon.fr>
Closes #10268 from jkbradley/pyspark-cv-seed.
|
|
|
|
|
|
|
|
|
|
| |
tests
Although this patch still doesn't solve the issue why the return code is 0 (see JIRA description), it resolves the issue of python version mismatch.
Author: Jeff Zhang <zjffdu@apache.org>
Closes #10322 from zjffdu/SPARK-12361.
|
|
|
|
|
|
|
|
|
|
| |
JIRA: https://issues.apache.org/jira/browse/SPARK-12016
We should not directly use Word2VecModel in pyspark. We need to wrap it in a Word2VecModelWrapper when loading it in pyspark.
Author: Liang-Chi Hsieh <viirya@appier.com>
Closes #10100 from viirya/fix-load-py-wordvecmodel.
|
|
|
|
|
|
|
|
| |
Adding ability to define an initial state RDD for use with updateStateByKey PySpark. Added unit test and changed stateful_network_wordcount example to use initial RDD.
Author: Bryan Cutler <bjcutler@us.ibm.com>
Closes #10082 from BryanCutler/initial-rdd-updateStateByKey-SPARK-11713.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
visualizing SQL query plan
This PR adds a `private[sql]` method `metadata` to `SparkPlan`, which can be used to describe detail information about a physical plan during visualization. Specifically, this PR uses this method to provide details of `PhysicalRDD`s translated from a data source relation. For example, a `ParquetRelation` converted from Hive metastore table `default.psrc` is now shown as the following screenshot:
![image](https://cloud.githubusercontent.com/assets/230655/11526657/e10cb7e6-9916-11e5-9afa-f108932ec890.png)
And here is the screenshot for a regular `ParquetRelation` (not converted from Hive metastore table) loaded from a really long path:
![output](https://cloud.githubusercontent.com/assets/230655/11680582/37c66460-9e94-11e5-8f50-842db5309d5a.png)
Author: Cheng Lian <lian@databricks.com>
Closes #10004 from liancheng/spark-12012.physical-rdd-metadata.
|
|
|
|
|
|
|
|
| |
In SPARK-11946 the API for pivot was changed a bit and got updated doc, the doc changes were not made for the python api though. This PR updates the python doc to be consistent.
Author: Andrew Ray <ray.andrew@gmail.com>
Closes #10176 from aray/sql-pivot-python-doc.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Currently, the current line is not cleared by Cltr-C
After this patch
```
>>> asdfasdf^C
Traceback (most recent call last):
File "~/spark/python/pyspark/context.py", line 225, in signal_handler
raise KeyboardInterrupt()
KeyboardInterrupt
```
It's still worse than 1.5 (and before).
Author: Davies Liu <davies@databricks.com>
Closes #10134 from davies/fix_cltrc.
|
|
|
|
|
|
|
|
|
|
|
|
| |
Python tests require access to the `KinesisTestUtils` file. When this file exists under src/test, python can't access it, since it is not available in the assembly jar.
However, if we move KinesisTestUtils to src/main, we need to add the KinesisProducerLibrary as a dependency. In order to avoid this, I moved KinesisTestUtils to src/main, and extended it with ExtendedKinesisTestUtils which is under src/test that adds support for the KPL.
cc zsxwing tdas
Author: Burak Yavuz <brkyvz@gmail.com>
Closes #10050 from brkyvz/kinesis-py.
|
|
|
|
|
|
|
|
|
| |
Use ```coefficients``` replace ```weights```, I wish they are the last two.
mengxr
Author: Yanbo Liang <ybliang8@gmail.com>
Closes #10065 from yanboliang/coefficients.
|
|
|
|
|
|
| |
Author: Davies Liu <davies@databricks.com>
Closes #10090 from davies/fix_coalesce.
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
recovery issue
Fixed a minor race condition in #10017
Closes #10017
Author: jerryshao <sshao@hortonworks.com>
Author: Shixiong Zhu <shixiong@databricks.com>
Closes #10074 from zsxwing/review-pr10017.
|
|
|
|
|
|
|
|
|
|
|
|
| |
KinesisStreamTests in test.py is broken because of #9403. See https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46896/testReport/(root)/KinesisStreamTests/test_kinesis_stream/
Because Streaming Python didn’t work when merging https://github.com/apache/spark/pull/9403, the PR build didn’t report the Python test failure actually.
This PR just disabled the test to unblock #10039
Author: Shixiong Zhu <shixiong@databricks.com>
Closes #10047 from zsxwing/disable-python-kinesis-test.
|
|
|
|
|
|
| |
Author: Jeff Zhang <zjffdu@apache.org>
Closes #9903 from zjffdu/SPARK-11917.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Added Python test cases for the function `isnan`, `isnull`, `nanvl` and `json_tuple`.
Fixed a bug in the function `json_tuple`
rxin , could you help me review my changes? Please let me know anything is missing.
Thank you! Have a good Thanksgiving day!
Author: gatorsmile <gatorsmile@gmail.com>
Closes #9977 from gatorsmile/json_tuple.
|
|
|
|
|
|
|
|
|
|
|
|
| |
TransformFunctionSerializer to Java
The Python exception track in TransformFunction and TransformFunctionSerializer is not sent back to Java. Py4j just throws a very general exception, which is hard to debug.
This PRs adds `getFailure` method to get the failure message in Java side.
Author: Shixiong Zhu <shixiong@databricks.com>
Closes #9922 from zsxwing/SPARK-11935.
|
|
|
|
|
|
|
|
|
|
| |
Currently, we does not have visualization for SQL query from Python, this PR fix that.
cc zsxwing
Author: Davies Liu <davies@databricks.com>
Closes #9949 from davies/pyspark_sql_ui.
|
|
|
|
|
|
| |
Author: felixcheung <felixcheung_m@hotmail.com>
Closes #9967 from felixcheung/pypivotdoc.
|
|
|
|
|
|
|
|
|
|
| |
…for registerFunction [Python]
Straightforward change on the python doc
Author: Jeff Zhang <zjffdu@apache.org>
Closes #9901 from zjffdu/SPARK-11860.
|
|
|
|
|
|
| |
Author: Reynold Xin <rxin@databricks.com>
Closes #9948 from rxin/SPARK-10621.
|
|
|
|
|
|
|
|
|
|
|
|
| |
DataFrameReader
This patch makes it consistent to use varargs in all DataFrameReader methods, including Parquet, JSON, text, and the generic load function.
Also added a few more API tests for the Java API.
Author: Reynold Xin <rxin@databricks.com>
Closes #9945 from rxin/SPARK-11967.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Currently pivot's signature looks like
```scala
scala.annotation.varargs
def pivot(pivotColumn: Column, values: Column*): GroupedData
scala.annotation.varargs
def pivot(pivotColumn: String, values: Any*): GroupedData
```
I think we can remove the one that takes "Column" types, since callers should always be passing in literals. It'd also be more clear if the values are not varargs, but rather Seq or java.util.List.
I also made similar changes for Python.
Author: Reynold Xin <rxin@databricks.com>
Closes #9929 from rxin/SPARK-11946.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Python API equal to Scala one
This is to bring the API documentation of StreamingLogisticReressionWithSGD and StreamingLinearRegressionWithSGC in line with the Scala versions.
-Fixed the algorithm descriptions
-Added default values to parameter descriptions
-Changed StreamingLogisticRegressionWithSGD regParam to default to 0, as in the Scala version
Author: Bryan Cutler <bjcutler@us.ibm.com>
Closes #9141 from BryanCutler/StreamingLogisticRegressionWithSGD-python-api-sync.
|
|
|
|
|
|
|
|
| |
They should use the existing SQLContext.
Author: Davies Liu <davies@databricks.com>
Closes #9914 from davies/create_udf.
|
|
|
|
|
|
|
|
|
|
| |
TransformFunction and TransformFunctionSerializer
TransformFunction and TransformFunctionSerializer don't rethrow the exception, so when any exception happens, it just return None. This will cause some weird NPE and confuse people.
Author: Shixiong Zhu <shixiong@databricks.com>
Closes #9847 from zsxwing/pyspark-streaming-exception.
|
|
|
|
|
|
|
|
|
| |
* Update doc for PySpark ```HasCheckpointInterval``` that users can understand how to disable checkpoint.
* Update doc for PySpark ```cacheNodeIds``` of ```DecisionTreeParams``` to notify the relationship between ```cacheNodeIds``` and ```checkpointInterval```.
Author: Yanbo Liang <ybliang8@gmail.com>
Closes #9856 from yanboliang/spark-11875.
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
reduceByKeyAndWindow
invFunc is optional and can be None. Instead of invFunc (the parameter) invReduceFunc (a local function) was checked for trueness (that is, not None, in this context). A local function is never None,
thus the case of invFunc=None (a common one when inverse reduction is not defined) was treated incorrectly, resulting in loss of data.
In addition, the docstring used wrong parameter names, also fixed.
Author: David Tolpin <david.tolpin@gmail.com>
Closes #9775 from dtolpin/master.
|
|
|
|
|
|
|
|
| |
[SPARK-7685](https://issues.apache.org/jira/browse/SPARK-7685) and [SPARK-9642](https://issues.apache.org/jira/browse/SPARK-9642) have already supported setting weight column for ```LogisticRegression``` and ```LinearRegression```. It's a very important feature, PySpark should also support. mengxr
Author: Yanbo Liang <ybliang8@gmail.com>
Closes #9811 from yanboliang/spark-11820.
|
|
|
|
|
|
|
|
| |
return Double.NaN for mean/average when count == 0 for all numeric types that is converted to Double, Decimal type continue to return null.
Author: JihongMa <linlin200605@gmail.com>
Closes #9705 from JihongMA/SPARK-11720.
|
|
|
|
|
|
|
|
| |
…ion in PySpark
Author: Jeff Zhang <zjffdu@apache.org>
Closes #9791 from zjffdu/SPARK-11804.
|
|
|
|
|
|
|
|
|
|
|
|
| |
Fixed the merge conflicts in #7410
Closes #7410
Author: Shixiong Zhu <shixiong@databricks.com>
Author: jerryshao <saisai.shao@intel.com>
Author: jerryshao <sshao@hortonworks.com>
Closes #9742 from zsxwing/pr7410.
|
|
|
|
|
|
|
|
| |
We will do checkpoint when generating a batch and completing a batch. When the processing time of a batch is greater than the batch interval, checkpointing for completing an old batch may run after checkpointing for generating a new batch. If this happens, checkpoint of an old batch actually has the latest information, so we want to recovery from it. This PR will use the latest checkpoint time as the file name, so that we can always recovery from the latest checkpoint file.
Author: Shixiong Zhu <shixiong@databricks.com>
Closes #9707 from zsxwing/fix-checkpoint.
|
|
|
|
|
|
| |
Author: Daniel Jalova <djalova@us.ibm.com>
Closes #9186 from djalova/SPARK-6328.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This patch adds the following options to the JSON data source, for dealing with non-standard JSON files:
* `allowComments` (default `false`): ignores Java/C++ style comment in JSON records
* `allowUnquotedFieldNames` (default `false`): allows unquoted JSON field names
* `allowSingleQuotes` (default `true`): allows single quotes in addition to double quotes
* `allowNumericLeadingZeros` (default `false`): allows leading zeros in numbers (e.g. 00012)
To avoid passing a lot of options throughout the json package, I introduced a new JSONOptions case class to define all JSON config options.
Also updated documentation to explain these options.
Scala
![screen shot 2015-11-15 at 6 12 12 pm](https://cloud.githubusercontent.com/assets/323388/11172965/e3ace6ec-8bc4-11e5-805e-2d78f80d0ed6.png)
Python
![screen shot 2015-11-15 at 6 11 28 pm](https://cloud.githubusercontent.com/assets/323388/11172964/e23ed6ee-8bc4-11e5-8216-312f5983acd5.png)
Author: Reynold Xin <rxin@databricks.com>
Closes #9724 from rxin/SPARK-11745.
|
|
|
|
|
|
|
|
| |
This PR adds pivot to the python api of GroupedData with the same syntax as Scala/Java.
Author: Andrew Ray <ray.andrew@gmail.com>
Closes #9653 from aray/sql-pivot-python.
|
|
|
|
|
|
|
|
|
|
| |
report failures
This PR just checks the test results and returns 1 if the test fails, so that `run-tests.py` can mark it fail.
Author: Shixiong Zhu <shixiong@databricks.com>
Closes #9669 from zsxwing/streaming-python-tests.
|
|
|
|
|
|
| |
Author: Chris Snow <chsnow123@gmail.com>
Closes #9640 from snowch/patch-3.
|
|
|
|
|
|
|
|
| |
Example for sqlContext.createDataDrame from pandas.DataFrame has a typo
Author: Chris Snow <chsnow123@gmail.com>
Closes #9639 from snowch/patch-2.
|
|
|
|
|
|
|
|
| |
switched stddev support from DeclarativeAggregate to ImperativeAggregate.
Author: JihongMa <linlin200605@gmail.com>
Closes #9380 from JihongMA/SPARK-11420.
|