diff options
| author | sethah <seth.hendrickson16@gmail.com> | 2016-03-23 11:20:44 -0700 |
|---|---|---|
| committer | Joseph K. Bradley <joseph@databricks.com> | 2016-03-23 11:20:44 -0700 |
| commit | 30bdb5cbd9aec191cf15cdc83c3fee375c04c2b2 (patch) | |
| tree | 4d48b42ebe347fc40d5deeb3a77996db0c30eea1 /core | |
| parent | 48ee16d8012602c75d50aa2a85e26b7de3c48944 (diff) | |
| download | spark-30bdb5cbd9aec191cf15cdc83c3fee375c04c2b2.tar.gz spark-30bdb5cbd9aec191cf15cdc83c3fee375c04c2b2.tar.bz2 spark-30bdb5cbd9aec191cf15cdc83c3fee375c04c2b2.zip | |
[SPARK-13068][PYSPARK][ML] Type conversion for Pyspark params
## What changes were proposed in this pull request?
This patch adds type conversion functionality for parameters in Pyspark. A `typeConverter` field is added to the constructor of `Param` class. This argument is a function which converts values passed to this param to the appropriate type if possible. This is beneficial so that the params can fail at set time if they are given inappropriate values, but even more so because coherent error messages are now provided when Py4J cannot cast the python type to the appropriate Java type.
This patch also adds a `TypeConverters` class with factory methods for common type conversions. Most of the changes involve adding these factory type converters to existing params. The previous solution to this issue, `expectedType`, is deprecated and can be removed in 2.1.0 as discussed on the Jira.
## How was this patch tested?
Unit tests were added in python/pyspark/ml/tests.py to test parameter type conversion. These tests check that values that should be convertible are converted correctly, and that the appropriate errors are thrown when invalid values are provided.
Author: sethah <seth.hendrickson16@gmail.com>
Closes #11663 from sethah/SPARK-13068-tc.
Diffstat (limited to 'core')
0 files changed, 0 insertions, 0 deletions