| Commit message (Collapse) | Author | Age | Files | Lines |
|
|
|
|
|
|
|
|
|
|
|
|
| |
recovery issue
Fixed a minor race condition in #10017
Closes #10017
Author: jerryshao <sshao@hortonworks.com>
Author: Shixiong Zhu <shixiong@databricks.com>
Closes #10074 from zsxwing/review-pr10017.
|
|
|
|
|
|
|
|
|
|
|
|
| |
KinesisStreamTests in test.py is broken because of #9403. See https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46896/testReport/(root)/KinesisStreamTests/test_kinesis_stream/
Because Streaming Python didn’t work when merging https://github.com/apache/spark/pull/9403, the PR build didn’t report the Python test failure actually.
This PR just disabled the test to unblock #10039
Author: Shixiong Zhu <shixiong@databricks.com>
Closes #10047 from zsxwing/disable-python-kinesis-test.
|
|
|
|
|
|
| |
Author: Jeff Zhang <zjffdu@apache.org>
Closes #9903 from zjffdu/SPARK-11917.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Added Python test cases for the function `isnan`, `isnull`, `nanvl` and `json_tuple`.
Fixed a bug in the function `json_tuple`
rxin , could you help me review my changes? Please let me know anything is missing.
Thank you! Have a good Thanksgiving day!
Author: gatorsmile <gatorsmile@gmail.com>
Closes #9977 from gatorsmile/json_tuple.
|
|
|
|
|
|
|
|
|
|
|
|
| |
TransformFunctionSerializer to Java
The Python exception track in TransformFunction and TransformFunctionSerializer is not sent back to Java. Py4j just throws a very general exception, which is hard to debug.
This PRs adds `getFailure` method to get the failure message in Java side.
Author: Shixiong Zhu <shixiong@databricks.com>
Closes #9922 from zsxwing/SPARK-11935.
|
|
|
|
|
|
|
|
|
|
| |
Currently, we does not have visualization for SQL query from Python, this PR fix that.
cc zsxwing
Author: Davies Liu <davies@databricks.com>
Closes #9949 from davies/pyspark_sql_ui.
|
|
|
|
|
|
| |
Author: felixcheung <felixcheung_m@hotmail.com>
Closes #9967 from felixcheung/pypivotdoc.
|
|
|
|
|
|
|
|
|
|
| |
…for registerFunction [Python]
Straightforward change on the python doc
Author: Jeff Zhang <zjffdu@apache.org>
Closes #9901 from zjffdu/SPARK-11860.
|
|
|
|
|
|
| |
Author: Reynold Xin <rxin@databricks.com>
Closes #9948 from rxin/SPARK-10621.
|
|
|
|
|
|
|
|
|
|
|
|
| |
DataFrameReader
This patch makes it consistent to use varargs in all DataFrameReader methods, including Parquet, JSON, text, and the generic load function.
Also added a few more API tests for the Java API.
Author: Reynold Xin <rxin@databricks.com>
Closes #9945 from rxin/SPARK-11967.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Currently pivot's signature looks like
```scala
scala.annotation.varargs
def pivot(pivotColumn: Column, values: Column*): GroupedData
scala.annotation.varargs
def pivot(pivotColumn: String, values: Any*): GroupedData
```
I think we can remove the one that takes "Column" types, since callers should always be passing in literals. It'd also be more clear if the values are not varargs, but rather Seq or java.util.List.
I also made similar changes for Python.
Author: Reynold Xin <rxin@databricks.com>
Closes #9929 from rxin/SPARK-11946.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Python API equal to Scala one
This is to bring the API documentation of StreamingLogisticReressionWithSGD and StreamingLinearRegressionWithSGC in line with the Scala versions.
-Fixed the algorithm descriptions
-Added default values to parameter descriptions
-Changed StreamingLogisticRegressionWithSGD regParam to default to 0, as in the Scala version
Author: Bryan Cutler <bjcutler@us.ibm.com>
Closes #9141 from BryanCutler/StreamingLogisticRegressionWithSGD-python-api-sync.
|
|
|
|
|
|
|
|
| |
They should use the existing SQLContext.
Author: Davies Liu <davies@databricks.com>
Closes #9914 from davies/create_udf.
|
|
|
|
|
|
|
|
|
|
| |
TransformFunction and TransformFunctionSerializer
TransformFunction and TransformFunctionSerializer don't rethrow the exception, so when any exception happens, it just return None. This will cause some weird NPE and confuse people.
Author: Shixiong Zhu <shixiong@databricks.com>
Closes #9847 from zsxwing/pyspark-streaming-exception.
|
|
|
|
|
|
|
|
|
| |
* Update doc for PySpark ```HasCheckpointInterval``` that users can understand how to disable checkpoint.
* Update doc for PySpark ```cacheNodeIds``` of ```DecisionTreeParams``` to notify the relationship between ```cacheNodeIds``` and ```checkpointInterval```.
Author: Yanbo Liang <ybliang8@gmail.com>
Closes #9856 from yanboliang/spark-11875.
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
reduceByKeyAndWindow
invFunc is optional and can be None. Instead of invFunc (the parameter) invReduceFunc (a local function) was checked for trueness (that is, not None, in this context). A local function is never None,
thus the case of invFunc=None (a common one when inverse reduction is not defined) was treated incorrectly, resulting in loss of data.
In addition, the docstring used wrong parameter names, also fixed.
Author: David Tolpin <david.tolpin@gmail.com>
Closes #9775 from dtolpin/master.
|
|
|
|
|
|
|
|
| |
[SPARK-7685](https://issues.apache.org/jira/browse/SPARK-7685) and [SPARK-9642](https://issues.apache.org/jira/browse/SPARK-9642) have already supported setting weight column for ```LogisticRegression``` and ```LinearRegression```. It's a very important feature, PySpark should also support. mengxr
Author: Yanbo Liang <ybliang8@gmail.com>
Closes #9811 from yanboliang/spark-11820.
|
|
|
|
|
|
|
|
| |
return Double.NaN for mean/average when count == 0 for all numeric types that is converted to Double, Decimal type continue to return null.
Author: JihongMa <linlin200605@gmail.com>
Closes #9705 from JihongMA/SPARK-11720.
|
|
|
|
|
|
|
|
| |
…ion in PySpark
Author: Jeff Zhang <zjffdu@apache.org>
Closes #9791 from zjffdu/SPARK-11804.
|
|
|
|
|
|
|
|
|
|
|
|
| |
Fixed the merge conflicts in #7410
Closes #7410
Author: Shixiong Zhu <shixiong@databricks.com>
Author: jerryshao <saisai.shao@intel.com>
Author: jerryshao <sshao@hortonworks.com>
Closes #9742 from zsxwing/pr7410.
|
|
|
|
|
|
|
|
| |
We will do checkpoint when generating a batch and completing a batch. When the processing time of a batch is greater than the batch interval, checkpointing for completing an old batch may run after checkpointing for generating a new batch. If this happens, checkpoint of an old batch actually has the latest information, so we want to recovery from it. This PR will use the latest checkpoint time as the file name, so that we can always recovery from the latest checkpoint file.
Author: Shixiong Zhu <shixiong@databricks.com>
Closes #9707 from zsxwing/fix-checkpoint.
|
|
|
|
|
|
| |
Author: Daniel Jalova <djalova@us.ibm.com>
Closes #9186 from djalova/SPARK-6328.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This patch adds the following options to the JSON data source, for dealing with non-standard JSON files:
* `allowComments` (default `false`): ignores Java/C++ style comment in JSON records
* `allowUnquotedFieldNames` (default `false`): allows unquoted JSON field names
* `allowSingleQuotes` (default `true`): allows single quotes in addition to double quotes
* `allowNumericLeadingZeros` (default `false`): allows leading zeros in numbers (e.g. 00012)
To avoid passing a lot of options throughout the json package, I introduced a new JSONOptions case class to define all JSON config options.
Also updated documentation to explain these options.
Scala
![screen shot 2015-11-15 at 6 12 12 pm](https://cloud.githubusercontent.com/assets/323388/11172965/e3ace6ec-8bc4-11e5-805e-2d78f80d0ed6.png)
Python
![screen shot 2015-11-15 at 6 11 28 pm](https://cloud.githubusercontent.com/assets/323388/11172964/e23ed6ee-8bc4-11e5-8216-312f5983acd5.png)
Author: Reynold Xin <rxin@databricks.com>
Closes #9724 from rxin/SPARK-11745.
|
|
|
|
|
|
|
|
| |
This PR adds pivot to the python api of GroupedData with the same syntax as Scala/Java.
Author: Andrew Ray <ray.andrew@gmail.com>
Closes #9653 from aray/sql-pivot-python.
|
|
|
|
|
|
|
|
|
|
| |
report failures
This PR just checks the test results and returns 1 if the test fails, so that `run-tests.py` can mark it fail.
Author: Shixiong Zhu <shixiong@databricks.com>
Closes #9669 from zsxwing/streaming-python-tests.
|
|
|
|
|
|
| |
Author: Chris Snow <chsnow123@gmail.com>
Closes #9640 from snowch/patch-3.
|
|
|
|
|
|
|
|
| |
Example for sqlContext.createDataDrame from pandas.DataFrame has a typo
Author: Chris Snow <chsnow123@gmail.com>
Closes #9639 from snowch/patch-2.
|
|
|
|
|
|
|
|
| |
switched stddev support from DeclarativeAggregate to ImperativeAggregate.
Author: JihongMa <linlin200605@gmail.com>
Closes #9380 from JihongMA/SPARK-11420.
|
|
|
|
|
|
|
|
| |
Only install signal in main thread, or it will fail to create context in not-main thread.
Author: Davies Liu <davies@databricks.com>
Closes #9574 from davies/python_signal.
|
|
|
|
|
|
|
|
|
|
| |
Python
cc jkbradley
Author: Yu ISHIKAWA <yuu.ishikawa@gmail.com>
Closes #9534 from yu-iskw/SPARK-11566.
|
|
|
|
|
|
|
|
|
|
| |
like `df.agg(corr("col1", "col2")`
davies
Author: felixcheung <felixcheung_m@hotmail.com>
Closes #9536 from felixcheung/pyfunc.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
evaluate AggregateExpression1s
https://issues.apache.org/jira/browse/SPARK-9830
This PR contains the following main changes.
* Removing `AggregateExpression1`.
* Removing `Aggregate` operator, which is used to evaluate `AggregateExpression1`.
* Removing planner rule used to plan `Aggregate`.
* Linking `MultipleDistinctRewriter` to analyzer.
* Renaming `AggregateExpression2` to `AggregateExpression` and `AggregateFunction2` to `AggregateFunction`.
* Updating places where we create aggregate expression. The way to create aggregate expressions is `AggregateExpression(aggregateFunction, mode, isDistinct)`.
* Changing `val`s in `DeclarativeAggregate`s that touch children of this function to `lazy val`s (when we create aggregate expression in DataFrame API, children of an aggregate function can be unresolved).
Author: Yin Huai <yhuai@databricks.com>
Closes #9556 from yhuai/removeAgg1.
|
|
|
|
|
|
|
|
|
|
| |
in Python more specific
cc jkbradley
Author: Yu ISHIKAWA <yuu.ishikawa@gmail.com>
Closes #9577 from yu-iskw/SPARK-11610.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
For now they are thin wrappers around the corresponding Hive UDAFs.
One limitation with these in Hive 0.13.0 is they only support aggregating primitive types.
I chose snake_case here instead of camelCase because it seems to be used in the majority of the multi-word fns.
Do we also want to add these to `functions.py`?
This approach was recommended here: https://github.com/apache/spark/pull/8592#issuecomment-154247089
marmbrus rxin
Author: Nick Buroojy <nick.buroojy@civitaslearning.com>
Closes #9526 from nburoojy/nick/udaf-alias.
(cherry picked from commit a6ee4f989d020420dd08b97abb24802200ff23b2)
Signed-off-by: Michael Armbrust <michael@databricks.com>
|
|
|
|
|
|
|
|
| |
pyspark.ml.classification
Author: Yu ISHIKAWA <yuu.ishikawa@gmail.com>
Closes #8690 from yu-iskw/SPARK-10280.
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Could jkbradley and davies review it?
- Create a wrapper class: `LDAModelWrapper` for `LDAModel`. Because we can't deal with the return value of`describeTopics` in Scala from pyspark directly. `Array[(Array[Int], Array[Double])]` is too complicated to convert it.
- Add `loadLDAModel` in `PythonMLlibAPI`. Since `LDAModel` in Scala is an abstract class and we need to call `load` of `DistributedLDAModel`.
[[SPARK-8467] Add LDAModel.describeTopics() in Python - ASF JIRA](https://issues.apache.org/jira/browse/SPARK-8467)
Author: Yu ISHIKAWA <yuu.ishikawa@gmail.com>
Closes #8643 from yu-iskw/SPARK-8467-2.
|
|
|
|
|
|
|
|
| |
#9527 missed updating the python tests.
Author: Michael Armbrust <michael@databricks.com>
Closes #9533 from marmbrus/hotfixTextValue.
|
|
|
|
|
|
|
|
| |
…ithinPartitions.
Author: Nong Li <nong@databricks.com>
Closes #9504 from nongli/spark-11410.
|
|
|
|
|
|
|
|
|
|
|
|
| |
https://issues.apache.org/jira/browse/SPARK-10116
This is really trivial, just happened to notice it -- if `XORShiftRandom.hashSeed` is really supposed to have random bits throughout (as the comment implies), it needs to do something for the conversion to `long`.
mengxr mkolod
Author: Imran Rashid <irashid@cloudera.com>
Closes #8314 from squito/SPARK-10116.
|
|
|
|
|
|
|
|
|
|
| |
normal equation solver
Follow up [SPARK-9836](https://issues.apache.org/jira/browse/SPARK-9836), we should also support summary statistics for ```intercept```.
Author: Yanbo Liang <ybliang8@gmail.com>
Closes #9485 from yanboliang/spark-11473.
|
|
|
|
|
|
|
|
|
|
| |
coefficients/intercept/scale
PySpark ```AFTSurvivalRegressionModel``` should expose coefficients/intercept/scale. mengxr vectorijk
Author: Yanbo Liang <ybliang8@gmail.com>
Closes #9492 from yanboliang/spark-11527.
|
|
|
|
|
|
|
|
|
|
|
|
| |
return properly
This adds a failing test checking that `awaitTerminationOrTimeout` returns the expected value, and then fixes that failing test with the addition of a `return`.
tdas zsxwing
Author: Nick Evans <me@nicolasevans.org>
Closes #9336 from manygrams/fix_await_termination_or_timeout.
|
|
|
|
|
|
| |
Author: Yu ISHIKAWA <yuu.ishikawa@gmail.com>
Closes #9469 from yu-iskw/SPARK-10028.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
We added a bunch of higher order statistics such as skewness and kurtosis to GroupedData. I don't think they are common enough to justify being listed, since users can always use the normal statistics aggregate functions.
That is to say, after this change, we won't support
```scala
df.groupBy("key").kurtosis("colA", "colB")
```
However, we will still support
```scala
df.groupBy("key").agg(kurtosis(col("colA")), kurtosis(col("colB")))
```
Author: Reynold Xin <rxin@databricks.com>
Closes #9446 from rxin/SPARK-11489.
|
|
|
|
|
|
|
|
| |
Add Python API for stddev/stddev_pop/stddev_samp/variance/var_pop/var_samp/skewness/kurtosis
Author: Davies Liu <davies@databricks.com>
Closes #9424 from davies/py_var.
|
|
|
|
|
|
|
|
|
|
| |
in ML models
Deprecated in `LogisticRegression` and `LinearRegression`
Author: vectorijk <jiangkai@gmail.com>
Closes #9311 from vectorijk/spark-10592.
|
|
|
|
|
|
|
|
| |
and pyspark.ml.*
Author: lihao <lihaowhu@gmail.com>
Closes #9275 from lidinghao/SPARK-10286.
|
|
|
|
|
|
|
|
|
|
| |
This PR deprecates `runs` in k-means. `runs` introduces extra complexity and overhead in MLlib's k-means implementation. I haven't seen much usage with `runs` not equal to `1`. We don't have a unit test for it either. We can deprecate this method in 1.6, and void it in 1.7. It helps us simplify the implementation.
cc: srowen
Author: Xiangrui Meng <meng@databricks.com>
Closes #9322 from mengxr/SPARK-11358.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
provided schema
When creating a DataFrame from an RDD in PySpark, `createDataFrame` calls `.take(10)` to verify the first 10 rows of the RDD match the provided schema. Similar to https://issues.apache.org/jira/browse/SPARK-8070, but that issue affected cases where a schema was not provided.
Verifying the first 10 rows is of limited utility and causes the DAG to be executed non-lazily. If necessary, I believe this verification should be done lazily on all rows. However, since the caller is providing a schema to follow, I think it's acceptable to simply fail if the schema is incorrect.
marmbrus We chatted about this at SparkSummitEU. davies you made a similar change for the infer-schema path in https://github.com/apache/spark/pull/6606
Author: Jason White <jason.white@shopify.com>
Closes #9392 from JasonMWhite/createDataFrame_without_take.
|
|
|
|
|
|
|
|
|
|
| |
JIRA: https://issues.apache.org/jira/browse/SPARK-11322
As reported by JoshRosen in [databricks/spark-redshift/issues/89](https://github.com/databricks/spark-redshift/issues/89#issuecomment-149828308), the exception-masking behavior sometimes makes debugging harder. To deal with this issue, we should keep full stack trace in the captured exception.
Author: Liang-Chi Hsieh <viirya@appier.com>
Closes #9283 from viirya/py-exception-stacktrace.
|