| Commit message (Collapse) | Author | Age | Files | Lines |
|
|
|
|
|
|
|
|
|
|
|
| |
examples and user guide doc
ML ```LinearRegression``` use ```data/mllib/sample_libsvm_data.txt``` as dataset in examples and user guide doc, but it's actually classification dataset rather than regression dataset. We should use ```data/mllib/sample_linear_regression_data.txt``` instead.
The deeper causes is that ```LinearRegression``` with "normal" solver can not solve this dataset correctly, may be due to the ill condition and unreasonable label. This issue has been reported at [SPARK-11918](https://issues.apache.org/jira/browse/SPARK-11918).
It will confuse users if they run the example code but get exception, so we should make this change which can clearly illustrate the usage of ```LinearRegression``` algorithm.
Author: Yanbo Liang <ybliang8@gmail.com>
Closes #9905 from yanboliang/spark-11920.
|
|
|
|
|
|
|
|
|
|
|
| |
requests.
This way the timeout handling code can correctly close "hung" channels that are
processing streams.
Author: Marcelo Vanzin <vanzin@cloudera.com>
Closes #9747 from vanzin/SPARK-11762.
|
|
|
|
|
|
|
|
|
|
| |
Add label expression support for AM to restrict it runs on the specific set of nodes. I tested it locally and works fine.
sryza and vanzin please help to review, thanks a lot.
Author: jerryshao <sshao@hortonworks.com>
Closes #9800 from jerryshao/SPARK-7173.
|
|
|
|
|
|
| |
Author: Wenchen Fan <wenchen@databricks.com>
Closes #9898 from cloud-fan/agg.
|
|
|
|
|
|
| |
Author: Wenchen Fan <wenchen@databricks.com>
Closes #9906 from cloud-fan/nullable.
|
|
|
|
|
|
|
|
|
|
| |
We should use `InternalRow.isNullAt` to check if the field is null before calling `InternalRow.getXXX`
Thanks gatorsmile who discovered this bug.
Author: Wenchen Fan <wenchen@databricks.com>
Closes #9904 from cloud-fan/null.
|
|
|
|
|
|
|
|
|
| |
Can someone review my code to make sure I'm not missing anything? Thanks!
Author: Xiu Guo <xguo27@gmail.com>
Author: Xiu Guo <guoxi@us.ibm.com>
Closes #9612 from xguo27/SPARK-11628.
|
|
|
|
|
|
|
|
|
|
|
|
| |
There is an unhandled case in the transform method of VectorAssembler if one of the input columns doesn't have one of the supported type DoubleType, NumericType, BooleanType or VectorUDT.
So, if you try to transform a column of StringType you get a cryptic "scala.MatchError: StringType".
This PR aims to fix this, throwing a SparkException when dealing with an unknown column type.
Author: BenFradet <benjamin.fradet@gmail.com>
Closes #9885 from BenFradet/SPARK-11902.
|
|
|
|
|
|
|
|
| |
Like [SPARK-11852](https://issues.apache.org/jira/browse/SPARK-11852), ```k``` is params and we should save it under ```metadata/``` rather than both under ```data/``` and ```metadata/```. Refactor the constructor of ```ml.feature.PCAModel``` to take only ```pc``` but construct ```mllib.feature.PCAModel``` inside ```transform```.
Author: Yanbo Liang <ybliang8@gmail.com>
Closes #9897 from yanboliang/spark-11912.
|
|
|
|
|
|
|
|
|
|
| |
This PR adds a sidebar menu when browsing the user guide of MLlib. It uses a YAML file to describe the structure of the documentation. It should be trivial to adapt this to the other projects.
![screen shot 2015-11-18 at 4 46 12 pm](https://cloud.githubusercontent.com/assets/7594753/11259591/a55173f4-8e17-11e5-9340-0aed79d66262.png)
Author: Timothy Hunter <timhunter@databricks.com>
Closes #9826 from thunterdb/spark-11835.
|
|
|
|
|
|
|
|
|
|
|
|
| |
I believe this works for general estimators within CrossValidator, including compound estimators. (See the complex unit test.)
Added read/write for all 3 Evaluators as well.
CC: mengxr yanboliang
Author: Joseph K. Bradley <joseph@databricks.com>
Closes #9848 from jkbradley/cv-io.
|
|
|
|
|
|
|
|
|
|
| |
We used the name `Dataset` to refer to `SchemaRDD` in 1.2 in ML pipelines and created this example file. Since `Dataset` has a new meaning in Spark 1.6, we should rename it to avoid confusion. This PR also removes support for dense format to simplify the example code.
cc: yinxusen
Author: Xiangrui Meng <meng@databricks.com>
Closes #9873 from mengxr/SPARK-11895.
|
|
|
|
|
|
|
|
|
|
| |
JIRA: https://issues.apache.org/jira/browse/SPARK-11908
We should add NullType support to RowEncoder.
Author: Liang-Chi Hsieh <viirya@appier.com>
Closes #9891 from viirya/rowencoder-nulltype.
|
|
|
|
|
|
|
|
|
|
|
|
| |
1. Renamed map to mapGroup, flatMap to flatMapGroup.
2. Renamed asKey -> keyAs.
3. Added more documentation.
4. Changed type parameter T to V on GroupedDataset.
5. Added since versions for all functions.
Author: Reynold Xin <rxin@databricks.com>
Closes #9880 from rxin/SPARK-11899.
|
|
|
|
|
|
| |
Author: Reynold Xin <rxin@databricks.com>
Closes #9882 from rxin/SPARK-11901.
|
|
|
|
|
|
| |
Author: Reynold Xin <rxin@databricks.com>
Closes #9881 from rxin/SPARK-11900.
|
|
|
|
|
|
|
|
| |
seems scala 2.11 doesn't support: define private methods in `trait xxx` and use it in `object xxx extend xxx`.
Author: Wenchen Fan <wenchen@databricks.com>
Closes #9879 from cloud-fan/follow.
|
|
|
|
|
|
| |
spark.ml"
This reverts commit e359d5dcf5bd300213054ebeae9fe75c4f7eb9e7.
|
| |
|
|
|
|
|
|
| |
Author: Michael Armbrust <michael@databricks.com>
Closes #9871 from marmbrus/scala211-break.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
In this PR I delete a method that breaks type inference for aggregators (only in the REPL)
The error when this method is present is:
```
<console>:38: error: missing parameter type for expanded function ((x$2) => x$2._2)
ds.groupBy(_._1).agg(sum(_._2), sum(_._3)).collect()
```
Author: Michael Armbrust <michael@databricks.com>
Closes #9870 from marmbrus/dataset-repl-agg.
|
|
|
|
|
|
|
|
|
|
| |
This mainly moves SqlNewHadoopRDD to the sql package. There is some state that is
shared between core and I've left that in core. This allows some other associated
minor cleanup.
Author: Nong Li <nong@databricks.com>
Closes #9845 from nongli/spark-11787.
|
|
|
|
|
|
|
|
| |
using include_example
Author: Vikas Nelamangala <vikasnelamangala@Vikass-MacBook-Pro.local>
Closes #9689 from vikasnp/master.
|
|
|
|
|
|
|
|
|
| |
#theScaryParts (i.e. changes to the repl, executor classloaders and codegen)...
Author: Michael Armbrust <michael@databricks.com>
Author: Yin Huai <yhuai@databricks.com>
Closes #9825 from marmbrus/dataset-replClasses2.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
information for SparkR:::summary correctly
Fix use of aliases and changes uses of rdname and seealso
`aliases` is the hint for `?` - it should not be linked to some other name - those should be seealso
https://cran.r-project.org/web/packages/roxygen2/vignettes/rd.html
Clean up usage on family, as multiple use of family with the same rdname is causing duplicated See Also html blocks (like http://spark.apache.org/docs/latest/api/R/count.html)
Also changing some rdname for dplyr-like variant for better R user visibility in R doc, eg. rbind, summary, mutate, summarize
shivaram yanboliang
Author: felixcheung <felixcheung_m@hotmail.com>
Closes #9750 from felixcheung/rdocaliases.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
re-creating the UserDefinedFunction
https://issues.apache.org/jira/browse/SPARK-11716
This is one is #9739 and a regression test. When commit it, please make sure the author is jbonofre.
You can find the original PR at https://github.com/apache/spark/pull/9739
closes #9739
Author: Jean-Baptiste Onofré <jbonofre@apache.org>
Author: Yin Huai <yhuai@databricks.com>
Closes #9868 from yhuai/SPARK-11716.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
In PersistenceEngineSuite, we do not call `close()` on the PersistenceEngine at the end of the test. For the ZooKeeperPersistenceEngine, this causes us to leak a ZooKeeper client, causing the logs of unrelated tests to be periodically spammed with connection error messages from that client:
```
15/11/20 05:13:35.789 pool-1-thread-1-ScalaTest-running-PersistenceEngineSuite-SendThread(localhost:15741) INFO ClientCnxn: Opening socket connection to server localhost/127.0.0.1:15741. Will not attempt to authenticate using SASL (unknown error)
15/11/20 05:13:35.790 pool-1-thread-1-ScalaTest-running-PersistenceEngineSuite-SendThread(localhost:15741) WARN ClientCnxn: Session 0x15124ff48dd0000 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
```
This patch fixes this by using a `finally` block.
Author: Josh Rosen <joshrosen@databricks.com>
Closes #9864 from JoshRosen/close-zookeeper-client-in-tests.
|
|
|
|
|
|
|
|
|
|
| |
TransformFunction and TransformFunctionSerializer
TransformFunction and TransformFunctionSerializer don't rethrow the exception, so when any exception happens, it just return None. This will cause some weird NPE and confuse people.
Author: Shixiong Zhu <shixiong@databricks.com>
Closes #9847 from zsxwing/pyspark-streaming-exception.
|
|
|
|
|
|
|
|
|
|
|
|
| |
treat int in seconds.
Hive has since changed this behavior as well. https://issues.apache.org/jira/browse/HIVE-3454
Author: Nong Li <nong@databricks.com>
Author: Nong Li <nongli@gmail.com>
Author: Yin Huai <yhuai@databricks.com>
Closes #9685 from nongli/spark-11724.
|
|
|
|
|
|
|
|
| |
This patch reduces some RPC timeouts in order to speed up the slow "AkkaUtilsSuite.remote fetch ssl on - untrusted server", which used to take two minutes to run.
Author: Josh Rosen <joshrosen@databricks.com>
Closes #9869 from JoshRosen/SPARK-11650.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
before this PR, when users try to get an encoder for an un-supported class, they will only get a very simple error message like `Encoder for type xxx is not supported`.
After this PR, the error message become more friendly, for example:
```
No Encoder found for abc.xyz.NonEncodable
- array element class: "abc.xyz.NonEncodable"
- field (class: "scala.Array", name: "arrayField")
- root class: "abc.xyz.AnotherClass"
```
Author: Wenchen Fan <wenchen@databricks.com>
Closes #9810 from cloud-fan/error-message.
|
|
|
|
|
|
|
|
|
|
| |
JIRA: https://issues.apache.org/jira/browse/SPARK-11817
Instead of return None, we should truncate the fractional seconds to prevent inserting NULL.
Author: Liang-Chi Hsieh <viirya@appier.com>
Closes #9834 from viirya/truncate-fractional-sec.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
DataSet APIs look great! However, I am lost when doing multiple level joins. For example,
```
val ds1 = Seq(("a", 1), ("b", 2)).toDS().as("a")
val ds2 = Seq(("a", 1), ("b", 2)).toDS().as("b")
val ds3 = Seq(("a", 1), ("b", 2)).toDS().as("c")
ds1.joinWith(ds2, $"a._2" === $"b._2").as("ab").joinWith(ds3, $"ab._1._2" === $"c._2").printSchema()
```
The printed schema is like
```
root
|-- _1: struct (nullable = true)
| |-- _1: struct (nullable = true)
| | |-- _1: string (nullable = true)
| | |-- _2: integer (nullable = true)
| |-- _2: struct (nullable = true)
| | |-- _1: string (nullable = true)
| | |-- _2: integer (nullable = true)
|-- _2: struct (nullable = true)
| |-- _1: string (nullable = true)
| |-- _2: integer (nullable = true)
```
Personally, I think we need the printSchema function. Sometimes, I do not know how to specify the column, especially when their data types are mixed. For example, if I want to write the following select for the above multi-level join, I have to know the schema:
```
newDS.select(expr("_1._2._2 + 1").as[Int]).collect()
```
marmbrus rxin cloud-fan Do you have the same feeling?
Author: gatorsmile <gatorsmile@gmail.com>
Closes #9855 from gatorsmile/printSchemaDataSet.
|
|
|
|
|
|
|
|
|
|
| |
jira: https://issues.apache.org/jira/browse/SPARK-11689
Add simple user guide for LDA under spark.ml and example code under examples/. Use include_example to include example code in the user guide markdown. Check SPARK-11606 for instructions.
Author: Yuhao Yang <hhbyyh@gmail.com>
Closes #9722 from hhbyyh/ldaMLExample.
|
|
|
|
|
|
|
|
| |
```withStd``` and ```withMean``` should be params of ```StandardScaler``` and ```StandardScalerModel```.
Author: Yanbo Liang <ybliang8@gmail.com>
Closes #9839 from yanboliang/standardScaler-refactor.
|
|
|
|
|
|
|
|
|
|
| |
This patch fixes an issue where the `spark.sql.TungstenAggregate.testFallbackStartsAt` SQLConf setting was not properly reset / cleared at the end of `TungstenAggregationQueryWithControlledFallbackSuite`. This ended up causing test failures in HiveCompatibilitySuite in Maven builds by causing spilling to occur way too frequently.
This configuration leak was inadvertently introduced during test cleanup in #9618.
Author: Josh Rosen <joshrosen@databricks.com>
Closes #9857 from JoshRosen/clear-fallback-prop-in-test-teardown.
|
|
|
|
|
|
|
|
| |
https://issues.apache.org/jira/browse/SPARK-11867
Author: Xusen Yin <yinxusen@gmail.com>
Closes #9849 from yinxusen/SPARK-11867.
|
|
|
|
|
|
|
|
|
|
|
|
| |
Need to remove parent directory (```className```) rather than just tempDir (```className/random_name```)
I tested this with IDFSuite, which has 2 read/write tests, and it fixes the problem.
CC: mengxr Can you confirm this is fine? I believe it is since the same ```random_name``` is used for all tests in a suite; we basically have an extra unneeded level of nesting.
Author: Joseph K. Bradley <joseph@databricks.com>
Closes #9851 from jkbradley/tempdir-cleanup.
|
|
|
|
|
|
|
|
|
| |
* Update doc for PySpark ```HasCheckpointInterval``` that users can understand how to disable checkpoint.
* Update doc for PySpark ```cacheNodeIds``` of ```DecisionTreeParams``` to notify the relationship between ```cacheNodeIds``` and ```checkpointInterval```.
Author: Yanbo Liang <ybliang8@gmail.com>
Closes #9856 from yanboliang/spark-11875.
|
|
|
|
|
|
|
|
|
|
|
|
| |
Add read/write support to the following estimators under spark.ml:
* ChiSqSelector
* PCA
* VectorIndexer
* Word2Vec
Author: Yanbo Liang <ybliang8@gmail.com>
Closes #9838 from yanboliang/spark-11829.
|
|
|
|
|
|
|
|
|
|
| |
https://issues.apache.org/jira/browse/SPARK-11846
mengxr
Author: Xusen Yin <yinxusen@gmail.com>
Closes #9836 from yinxusen/SPARK-11846.
|
|
|
|
|
|
|
|
| |
Apply the user supplied pathfilter while retrieving the files from fs.
Author: Dilip Biswal <dbiswal@us.ibm.com>
Closes #9830 from dilipbiswal/spark-11544.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This PR has the following optimization:
1) The greatest/least already does the null-check, so the `If` and `IsNull` are not necessary.
2) In greatest/least, it should initialize the result using the first child (removing one block).
3) For primitive types, the generated greater expression is too complicated (`a > b ? 1 : (a < b) ? -1 : 0) > 0`), should be as simple as `a > b`
Combine these optimization, this could improve the performance of `ss_max` query by 30%.
Author: Davies Liu <davies@databricks.com>
Closes #9846 from davies/improve_max.
|
|
|
|
|
|
|
|
|
|
| |
correctly checkpointed
To make sure that all lineage is correctly truncated for TrackStateRDD when checkpointed.
Author: Tathagata Das <tathagata.das1565@gmail.com>
Closes #9831 from tdas/SPARK-11845.
|
|
|
|
|
|
|
|
|
| |
Don't log ERROR messages when executors are explicitly killed or when
the exit reason is not yet known.
Author: Marcelo Vanzin <vanzin@cloudera.com>
Closes #9780 from vanzin/SPARK-11789.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Fixes bug with grouping sets (including cube/rollup) where aggregates that included grouping expressions would return the wrong (null) result.
Also simplifies the analyzer rule a bit and leaves column pruning to the optimizer.
Added multiple unit tests to DataFrameAggregateSuite and verified it passes hive compatibility suite:
```
build/sbt -Phive -Dspark.hive.whitelist='groupby.*_grouping.*' 'test-only org.apache.spark.sql.hive.execution.HiveCompatibilitySuite'
```
This is an alternative to pr https://github.com/apache/spark/pull/9419 but I think its better as it simplifies the analyzer rule instead of adding another special case to it.
Author: Andrew Ray <ray.andrew@gmail.com>
Closes #9815 from aray/groupingset-agg-fix.
|
|
|
|
|
|
|
|
| |
a small change
Author: hushan <hushan@xiaomi.com>
Closes #9691 from suyanNone/unify-getDependency.
|
|
|
|
|
|
| |
Author: Marcelo Vanzin <vanzin@cloudera.com>
Closes #9820 from vanzin/SPARK-11828.
|
|
|
|
|
|
|
|
|
|
| |
…xceptions are thrown during executor shutdown
This commit will make sure that when uncaught exceptions are prepended with [Container in shutdown] when JVM is shutting down.
Author: Srinivasa Reddy Vundela <vsr@cloudera.com>
Closes #9809 from vundela/master_11799.
|
|
|
|
|
|
|
|
| |
Use port 0 to fix port-contention-related flakiness
Author: Shixiong Zhu <shixiong@databricks.com>
Closes #9841 from zsxwing/SPARK-11831.
|