diff options
| author | VinceShieh <vincent.xie@intel.com> | 2017-03-07 11:24:20 -0800 |
|---|---|---|
| committer | Joseph K. Bradley <joseph@databricks.com> | 2017-03-07 11:24:20 -0800 |
| commit | 4a9034b17374cf19c77cb74e36c86cd085d59602 (patch) | |
| tree | 1bb589f4efd445295897173aff2146ecfec425f8 /python/pyspark | |
| parent | c05baabf10dd4c808929b4ae7a6d118aba6dd665 (diff) | |
| download | spark-4a9034b17374cf19c77cb74e36c86cd085d59602.tar.gz spark-4a9034b17374cf19c77cb74e36c86cd085d59602.tar.bz2 spark-4a9034b17374cf19c77cb74e36c86cd085d59602.zip | |
[SPARK-17498][ML] StringIndexer enhancement for handling unseen labels
## What changes were proposed in this pull request?
This PR is an enhancement to ML StringIndexer.
Before this PR, String Indexer only supports "skip"/"error" options to deal with unseen records.
But those unseen records might still be useful and user would like to keep the unseen labels in
certain use cases, This PR enables StringIndexer to support keeping unseen labels as
indices [numLabels].
'''Before
StringIndexer().setHandleInvalid("skip")
StringIndexer().setHandleInvalid("error")
'''After
support the third option "keep"
StringIndexer().setHandleInvalid("keep")
## How was this patch tested?
Test added in StringIndexerSuite
Signed-off-by: VinceShieh <vincent.xieintel.com>
(Please fill in changes proposed in this fix)
Author: VinceShieh <vincent.xie@intel.com>
Closes #16883 from VinceShieh/spark-17498.
Diffstat (limited to 'python/pyspark')
0 files changed, 0 insertions, 0 deletions