<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>pyspark.ml package — PySpark master documentation</title>
<link rel="stylesheet" href="_static/nature.css" type="text/css" />
<link rel="stylesheet" href="_static/pygments.css" type="text/css" />
<script type="text/javascript">
var DOCUMENTATION_OPTIONS = {
URL_ROOT: './',
VERSION: 'master',
COLLAPSE_INDEX: false,
FILE_SUFFIX: '.html',
HAS_SOURCE: true
};
</script>
<script type="text/javascript" src="_static/jquery.js"></script>
<script type="text/javascript" src="_static/underscore.js"></script>
<script type="text/javascript" src="_static/doctools.js"></script>
<link rel="top" title="PySpark master documentation" href="index.html" />
<link rel="next" title="pyspark.mllib package" href="pyspark.mllib.html" />
<link rel="prev" title="pyspark.streaming module" href="pyspark.streaming.html" />
</head>
<body role="document">
<div class="related" role="navigation" aria-label="related navigation">
<h3>Navigation</h3>
<ul>
<li class="right" style="margin-right: 10px">
<a href="pyspark.mllib.html" title="pyspark.mllib package"
accesskey="N">next</a></li>
<li class="right" >
<a href="pyspark.streaming.html" title="pyspark.streaming module"
accesskey="P">previous</a> |</li>
<li class="nav-item nav-item-0"><a href="index.html">PySpark master documentation</a> »</li>
</ul>
</div>
<div class="document">
<div class="documentwrapper">
<div class="bodywrapper">
<div class="body" role="main">
<div class="section" id="pyspark-ml-package">
<h1>pyspark.ml package<a class="headerlink" href="#pyspark-ml-package" title="Permalink to this headline">¶</a></h1>
<div class="section" id="module-pyspark.ml">
<span id="ml-pipeline-apis"></span><h2>ML Pipeline APIs<a class="headerlink" href="#module-pyspark.ml" title="Permalink to this headline">¶</a></h2>
<dl class="class">
<dt id="pyspark.ml.Transformer">
<em class="property">class </em><code class="descclassname">pyspark.ml.</code><code class="descname">Transformer</code><a class="reference internal" href="_modules/pyspark/ml/pipeline.html#Transformer"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.Transformer" title="Permalink to this definition">¶</a></dt>
<dd><p>Abstract class for transformers that transform one dataset into
another.</p>
<dl class="method">
<dt id="pyspark.ml.Transformer.copy">
<code class="descname">copy</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.Transformer.copy" title="Permalink to this definition">¶</a></dt>
<dd><p>Creates a copy of this instance with the same uid and some
extra params. The default implementation creates a
shallow copy using <code class="xref py py-func docutils literal"><span class="pre">copy.copy()</span></code>, and then copies the
embedded and extra parameters over and returns the copy.
Subclasses should override this method if the default approach
is not sufficient.
:param extra: Extra parameters to copy to the new instance
:return: Copy of this instance</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.Transformer.explainParam">
<code class="descname">explainParam</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.Transformer.explainParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Explains a single param and returns its name, doc, and optional
default value and user-supplied value in a string.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.Transformer.explainParams">
<code class="descname">explainParams</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.Transformer.explainParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns the documentation of all params with their optionally
default values and user-supplied values.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.Transformer.extractParamMap">
<code class="descname">extractParamMap</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.Transformer.extractParamMap" title="Permalink to this definition">¶</a></dt>
<dd><p>Extracts the embedded default param values and user-supplied
values, and then merges them with extra values from input into
a flat param map, where the latter value is used if there exist
conflicts, i.e., with ordering: default param values <
user-supplied values < extra.
:param extra: extra param values
:return: merged param map</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.Transformer.getOrDefault">
<code class="descname">getOrDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.Transformer.getOrDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of a param in the user-supplied param map or its
default value. Raises an error if neither is set.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.Transformer.getParam">
<code class="descname">getParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.Transformer.getParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets a param by its name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.Transformer.hasDefault">
<code class="descname">hasDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.Transformer.hasDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param has a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.Transformer.hasParam">
<code class="descname">hasParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.Transformer.hasParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Tests whether this instance contains a param with a given
(string) name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.Transformer.isDefined">
<code class="descname">isDefined</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.Transformer.isDefined" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user or has
a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.Transformer.isSet">
<code class="descname">isSet</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.Transformer.isSet" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.Transformer.params">
<code class="descname">params</code><a class="headerlink" href="#pyspark.ml.Transformer.params" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns all params ordered by name. The default implementation
uses <code class="xref py py-func docutils literal"><span class="pre">dir()</span></code> to get all attributes of type
<code class="xref py py-class docutils literal"><span class="pre">Param</span></code>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.Transformer.transform">
<code class="descname">transform</code><span class="sig-paren">(</span><em>dataset</em>, <em>params=None</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/pipeline.html#Transformer.transform"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.Transformer.transform" title="Permalink to this definition">¶</a></dt>
<dd><p>Transforms the input dataset with optional parameters.</p>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple">
<li><strong>dataset</strong> – input dataset, which is an instance of
<a class="reference internal" href="pyspark.sql.html#pyspark.sql.DataFrame" title="pyspark.sql.DataFrame"><code class="xref py py-class docutils literal"><span class="pre">pyspark.sql.DataFrame</span></code></a></li>
<li><strong>params</strong> – an optional param map that overrides embedded
params.</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body"><p class="first last">transformed dataset</p>
</td>
</tr>
</tbody>
</table>
</dd></dl>
</dd></dl>
<dl class="class">
<dt id="pyspark.ml.Estimator">
<em class="property">class </em><code class="descclassname">pyspark.ml.</code><code class="descname">Estimator</code><a class="reference internal" href="_modules/pyspark/ml/pipeline.html#Estimator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.Estimator" title="Permalink to this definition">¶</a></dt>
<dd><p>Abstract class for estimators that fit models to data.</p>
<dl class="method">
<dt id="pyspark.ml.Estimator.copy">
<code class="descname">copy</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.Estimator.copy" title="Permalink to this definition">¶</a></dt>
<dd><p>Creates a copy of this instance with the same uid and some
extra params. The default implementation creates a
shallow copy using <code class="xref py py-func docutils literal"><span class="pre">copy.copy()</span></code>, and then copies the
embedded and extra parameters over and returns the copy.
Subclasses should override this method if the default approach
is not sufficient.
:param extra: Extra parameters to copy to the new instance
:return: Copy of this instance</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.Estimator.explainParam">
<code class="descname">explainParam</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.Estimator.explainParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Explains a single param and returns its name, doc, and optional
default value and user-supplied value in a string.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.Estimator.explainParams">
<code class="descname">explainParams</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.Estimator.explainParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns the documentation of all params with their optionally
default values and user-supplied values.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.Estimator.extractParamMap">
<code class="descname">extractParamMap</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.Estimator.extractParamMap" title="Permalink to this definition">¶</a></dt>
<dd><p>Extracts the embedded default param values and user-supplied
values, and then merges them with extra values from input into
a flat param map, where the latter value is used if there exist
conflicts, i.e., with ordering: default param values <
user-supplied values < extra.
:param extra: extra param values
:return: merged param map</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.Estimator.fit">
<code class="descname">fit</code><span class="sig-paren">(</span><em>dataset</em>, <em>params=None</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/pipeline.html#Estimator.fit"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.Estimator.fit" title="Permalink to this definition">¶</a></dt>
<dd><p>Fits a model to the input dataset with optional parameters.</p>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple">
<li><strong>dataset</strong> – input dataset, which is an instance of
<a class="reference internal" href="pyspark.sql.html#pyspark.sql.DataFrame" title="pyspark.sql.DataFrame"><code class="xref py py-class docutils literal"><span class="pre">pyspark.sql.DataFrame</span></code></a></li>
<li><strong>params</strong> – an optional param map that overrides embedded
params. If a list/tuple of param maps is given,
this calls fit on each param map and returns a
list of models.</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body"><p class="first last">fitted model(s)</p>
</td>
</tr>
</tbody>
</table>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.Estimator.getOrDefault">
<code class="descname">getOrDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.Estimator.getOrDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of a param in the user-supplied param map or its
default value. Raises an error if neither is set.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.Estimator.getParam">
<code class="descname">getParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.Estimator.getParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets a param by its name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.Estimator.hasDefault">
<code class="descname">hasDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.Estimator.hasDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param has a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.Estimator.hasParam">
<code class="descname">hasParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.Estimator.hasParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Tests whether this instance contains a param with a given
(string) name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.Estimator.isDefined">
<code class="descname">isDefined</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.Estimator.isDefined" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user or has
a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.Estimator.isSet">
<code class="descname">isSet</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.Estimator.isSet" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.Estimator.params">
<code class="descname">params</code><a class="headerlink" href="#pyspark.ml.Estimator.params" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns all params ordered by name. The default implementation
uses <code class="xref py py-func docutils literal"><span class="pre">dir()</span></code> to get all attributes of type
<code class="xref py py-class docutils literal"><span class="pre">Param</span></code>.</p>
</dd></dl>
</dd></dl>
<dl class="class">
<dt id="pyspark.ml.Model">
<em class="property">class </em><code class="descclassname">pyspark.ml.</code><code class="descname">Model</code><a class="reference internal" href="_modules/pyspark/ml/pipeline.html#Model"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.Model" title="Permalink to this definition">¶</a></dt>
<dd><p>Abstract class for models that are fitted by estimators.</p>
<dl class="method">
<dt id="pyspark.ml.Model.copy">
<code class="descname">copy</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.Model.copy" title="Permalink to this definition">¶</a></dt>
<dd><p>Creates a copy of this instance with the same uid and some
extra params. The default implementation creates a
shallow copy using <code class="xref py py-func docutils literal"><span class="pre">copy.copy()</span></code>, and then copies the
embedded and extra parameters over and returns the copy.
Subclasses should override this method if the default approach
is not sufficient.
:param extra: Extra parameters to copy to the new instance
:return: Copy of this instance</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.Model.explainParam">
<code class="descname">explainParam</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.Model.explainParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Explains a single param and returns its name, doc, and optional
default value and user-supplied value in a string.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.Model.explainParams">
<code class="descname">explainParams</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.Model.explainParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns the documentation of all params with their optionally
default values and user-supplied values.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.Model.extractParamMap">
<code class="descname">extractParamMap</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.Model.extractParamMap" title="Permalink to this definition">¶</a></dt>
<dd><p>Extracts the embedded default param values and user-supplied
values, and then merges them with extra values from input into
a flat param map, where the latter value is used if there exist
conflicts, i.e., with ordering: default param values <
user-supplied values < extra.
:param extra: extra param values
:return: merged param map</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.Model.getOrDefault">
<code class="descname">getOrDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.Model.getOrDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of a param in the user-supplied param map or its
default value. Raises an error if neither is set.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.Model.getParam">
<code class="descname">getParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.Model.getParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets a param by its name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.Model.hasDefault">
<code class="descname">hasDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.Model.hasDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param has a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.Model.hasParam">
<code class="descname">hasParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.Model.hasParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Tests whether this instance contains a param with a given
(string) name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.Model.isDefined">
<code class="descname">isDefined</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.Model.isDefined" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user or has
a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.Model.isSet">
<code class="descname">isSet</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.Model.isSet" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.Model.params">
<code class="descname">params</code><a class="headerlink" href="#pyspark.ml.Model.params" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns all params ordered by name. The default implementation
uses <code class="xref py py-func docutils literal"><span class="pre">dir()</span></code> to get all attributes of type
<code class="xref py py-class docutils literal"><span class="pre">Param</span></code>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.Model.transform">
<code class="descname">transform</code><span class="sig-paren">(</span><em>dataset</em>, <em>params=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.Model.transform" title="Permalink to this definition">¶</a></dt>
<dd><p>Transforms the input dataset with optional parameters.</p>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple">
<li><strong>dataset</strong> – input dataset, which is an instance of
<a class="reference internal" href="pyspark.sql.html#pyspark.sql.DataFrame" title="pyspark.sql.DataFrame"><code class="xref py py-class docutils literal"><span class="pre">pyspark.sql.DataFrame</span></code></a></li>
<li><strong>params</strong> – an optional param map that overrides embedded
params.</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body"><p class="first last">transformed dataset</p>
</td>
</tr>
</tbody>
</table>
</dd></dl>
</dd></dl>
<dl class="class">
<dt id="pyspark.ml.Pipeline">
<em class="property">class </em><code class="descclassname">pyspark.ml.</code><code class="descname">Pipeline</code><span class="sig-paren">(</span><em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/pipeline.html#Pipeline"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.Pipeline" title="Permalink to this definition">¶</a></dt>
<dd><p>A simple pipeline, which acts as an estimator. A Pipeline consists
of a sequence of stages, each of which is either an
<a class="reference internal" href="#pyspark.ml.Estimator" title="pyspark.ml.Estimator"><code class="xref py py-class docutils literal"><span class="pre">Estimator</span></code></a> or a <a class="reference internal" href="#pyspark.ml.Transformer" title="pyspark.ml.Transformer"><code class="xref py py-class docutils literal"><span class="pre">Transformer</span></code></a>. When
<a class="reference internal" href="#pyspark.ml.Pipeline.fit" title="pyspark.ml.Pipeline.fit"><code class="xref py py-meth docutils literal"><span class="pre">Pipeline.fit()</span></code></a> is called, the stages are executed in
order. If a stage is an <a class="reference internal" href="#pyspark.ml.Estimator" title="pyspark.ml.Estimator"><code class="xref py py-class docutils literal"><span class="pre">Estimator</span></code></a>, its
<a class="reference internal" href="#pyspark.ml.Estimator.fit" title="pyspark.ml.Estimator.fit"><code class="xref py py-meth docutils literal"><span class="pre">Estimator.fit()</span></code></a> method will be called on the input
dataset to fit a model. Then the model, which is a transformer,
will be used to transform the dataset as the input to the next
stage. If a stage is a <a class="reference internal" href="#pyspark.ml.Transformer" title="pyspark.ml.Transformer"><code class="xref py py-class docutils literal"><span class="pre">Transformer</span></code></a>, its
<a class="reference internal" href="#pyspark.ml.Transformer.transform" title="pyspark.ml.Transformer.transform"><code class="xref py py-meth docutils literal"><span class="pre">Transformer.transform()</span></code></a> method will be called to produce
the dataset for the next stage. The fitted model from a
<a class="reference internal" href="#pyspark.ml.Pipeline" title="pyspark.ml.Pipeline"><code class="xref py py-class docutils literal"><span class="pre">Pipeline</span></code></a> is an <a class="reference internal" href="#pyspark.ml.PipelineModel" title="pyspark.ml.PipelineModel"><code class="xref py py-class docutils literal"><span class="pre">PipelineModel</span></code></a>, which
consists of fitted models and transformers, corresponding to the
pipeline stages. If there are no stages, the pipeline acts as an
identity transformer.</p>
<dl class="method">
<dt id="pyspark.ml.Pipeline.copy">
<code class="descname">copy</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/pipeline.html#Pipeline.copy"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.Pipeline.copy" title="Permalink to this definition">¶</a></dt>
<dd><p>Creates a copy of this instance with the same uid and some
extra params. The default implementation creates a
shallow copy using <code class="xref py py-func docutils literal"><span class="pre">copy.copy()</span></code>, and then copies the
embedded and extra parameters over and returns the copy.
Subclasses should override this method if the default approach
is not sufficient.
:param extra: Extra parameters to copy to the new instance
:return: Copy of this instance</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.Pipeline.explainParam">
<code class="descname">explainParam</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.Pipeline.explainParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Explains a single param and returns its name, doc, and optional
default value and user-supplied value in a string.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.Pipeline.explainParams">
<code class="descname">explainParams</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.Pipeline.explainParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns the documentation of all params with their optionally
default values and user-supplied values.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.Pipeline.extractParamMap">
<code class="descname">extractParamMap</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.Pipeline.extractParamMap" title="Permalink to this definition">¶</a></dt>
<dd><p>Extracts the embedded default param values and user-supplied
values, and then merges them with extra values from input into
a flat param map, where the latter value is used if there exist
conflicts, i.e., with ordering: default param values <
user-supplied values < extra.
:param extra: extra param values
:return: merged param map</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.Pipeline.fit">
<code class="descname">fit</code><span class="sig-paren">(</span><em>dataset</em>, <em>params=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.Pipeline.fit" title="Permalink to this definition">¶</a></dt>
<dd><p>Fits a model to the input dataset with optional parameters.</p>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple">
<li><strong>dataset</strong> – input dataset, which is an instance of
<a class="reference internal" href="pyspark.sql.html#pyspark.sql.DataFrame" title="pyspark.sql.DataFrame"><code class="xref py py-class docutils literal"><span class="pre">pyspark.sql.DataFrame</span></code></a></li>
<li><strong>params</strong> – an optional param map that overrides embedded
params. If a list/tuple of param maps is given,
this calls fit on each param map and returns a
list of models.</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body"><p class="first last">fitted model(s)</p>
</td>
</tr>
</tbody>
</table>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.Pipeline.getOrDefault">
<code class="descname">getOrDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.Pipeline.getOrDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of a param in the user-supplied param map or its
default value. Raises an error if neither is set.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.Pipeline.getParam">
<code class="descname">getParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.Pipeline.getParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets a param by its name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.Pipeline.getStages">
<code class="descname">getStages</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/pipeline.html#Pipeline.getStages"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.Pipeline.getStages" title="Permalink to this definition">¶</a></dt>
<dd><p>Get pipeline stages.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.Pipeline.hasDefault">
<code class="descname">hasDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.Pipeline.hasDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param has a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.Pipeline.hasParam">
<code class="descname">hasParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.Pipeline.hasParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Tests whether this instance contains a param with a given
(string) name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.Pipeline.isDefined">
<code class="descname">isDefined</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.Pipeline.isDefined" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user or has
a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.Pipeline.isSet">
<code class="descname">isSet</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.Pipeline.isSet" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.Pipeline.params">
<code class="descname">params</code><a class="headerlink" href="#pyspark.ml.Pipeline.params" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns all params ordered by name. The default implementation
uses <code class="xref py py-func docutils literal"><span class="pre">dir()</span></code> to get all attributes of type
<code class="xref py py-class docutils literal"><span class="pre">Param</span></code>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.Pipeline.setParams">
<code class="descname">setParams</code><span class="sig-paren">(</span><em>self</em>, <em>stages=None</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/pipeline.html#Pipeline.setParams"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.Pipeline.setParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets params for Pipeline.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.Pipeline.setStages">
<code class="descname">setStages</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/pipeline.html#Pipeline.setStages"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.Pipeline.setStages" title="Permalink to this definition">¶</a></dt>
<dd><p>Set pipeline stages.
:param value: a list of transformers or estimators
:return: the pipeline instance</p>
</dd></dl>
</dd></dl>
<dl class="class">
<dt id="pyspark.ml.PipelineModel">
<em class="property">class </em><code class="descclassname">pyspark.ml.</code><code class="descname">PipelineModel</code><span class="sig-paren">(</span><em>stages</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/pipeline.html#PipelineModel"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.PipelineModel" title="Permalink to this definition">¶</a></dt>
<dd><p>Represents a compiled pipeline with transformers and fitted models.</p>
<dl class="method">
<dt id="pyspark.ml.PipelineModel.copy">
<code class="descname">copy</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/pipeline.html#PipelineModel.copy"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.PipelineModel.copy" title="Permalink to this definition">¶</a></dt>
<dd><p>Creates a copy of this instance with the same uid and some
extra params. The default implementation creates a
shallow copy using <code class="xref py py-func docutils literal"><span class="pre">copy.copy()</span></code>, and then copies the
embedded and extra parameters over and returns the copy.
Subclasses should override this method if the default approach
is not sufficient.
:param extra: Extra parameters to copy to the new instance
:return: Copy of this instance</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.PipelineModel.explainParam">
<code class="descname">explainParam</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.PipelineModel.explainParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Explains a single param and returns its name, doc, and optional
default value and user-supplied value in a string.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.PipelineModel.explainParams">
<code class="descname">explainParams</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.PipelineModel.explainParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns the documentation of all params with their optionally
default values and user-supplied values.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.PipelineModel.extractParamMap">
<code class="descname">extractParamMap</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.PipelineModel.extractParamMap" title="Permalink to this definition">¶</a></dt>
<dd><p>Extracts the embedded default param values and user-supplied
values, and then merges them with extra values from input into
a flat param map, where the latter value is used if there exist
conflicts, i.e., with ordering: default param values <
user-supplied values < extra.
:param extra: extra param values
:return: merged param map</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.PipelineModel.getOrDefault">
<code class="descname">getOrDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.PipelineModel.getOrDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of a param in the user-supplied param map or its
default value. Raises an error if neither is set.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.PipelineModel.getParam">
<code class="descname">getParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.PipelineModel.getParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets a param by its name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.PipelineModel.hasDefault">
<code class="descname">hasDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.PipelineModel.hasDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param has a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.PipelineModel.hasParam">
<code class="descname">hasParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.PipelineModel.hasParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Tests whether this instance contains a param with a given
(string) name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.PipelineModel.isDefined">
<code class="descname">isDefined</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.PipelineModel.isDefined" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user or has
a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.PipelineModel.isSet">
<code class="descname">isSet</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.PipelineModel.isSet" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.PipelineModel.params">
<code class="descname">params</code><a class="headerlink" href="#pyspark.ml.PipelineModel.params" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns all params ordered by name. The default implementation
uses <code class="xref py py-func docutils literal"><span class="pre">dir()</span></code> to get all attributes of type
<code class="xref py py-class docutils literal"><span class="pre">Param</span></code>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.PipelineModel.transform">
<code class="descname">transform</code><span class="sig-paren">(</span><em>dataset</em>, <em>params=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.PipelineModel.transform" title="Permalink to this definition">¶</a></dt>
<dd><p>Transforms the input dataset with optional parameters.</p>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple">
<li><strong>dataset</strong> – input dataset, which is an instance of
<a class="reference internal" href="pyspark.sql.html#pyspark.sql.DataFrame" title="pyspark.sql.DataFrame"><code class="xref py py-class docutils literal"><span class="pre">pyspark.sql.DataFrame</span></code></a></li>
<li><strong>params</strong> – an optional param map that overrides embedded
params.</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body"><p class="first last">transformed dataset</p>
</td>
</tr>
</tbody>
</table>
</dd></dl>
</dd></dl>
</div>
<div class="section" id="module-pyspark.ml.param">
<span id="pyspark-ml-param-module"></span><h2>pyspark.ml.param module<a class="headerlink" href="#module-pyspark.ml.param" title="Permalink to this headline">¶</a></h2>
<dl class="class">
<dt id="pyspark.ml.param.Param">
<em class="property">class </em><code class="descclassname">pyspark.ml.param.</code><code class="descname">Param</code><span class="sig-paren">(</span><em>parent</em>, <em>name</em>, <em>doc</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/param.html#Param"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.param.Param" title="Permalink to this definition">¶</a></dt>
<dd><p>A param with self-contained documentation.</p>
</dd></dl>
<dl class="class">
<dt id="pyspark.ml.param.Params">
<em class="property">class </em><code class="descclassname">pyspark.ml.param.</code><code class="descname">Params</code><a class="reference internal" href="_modules/pyspark/ml/param.html#Params"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.param.Params" title="Permalink to this definition">¶</a></dt>
<dd><p>Components that take parameters. This also provides an internal
param map to store parameter values attached to the instance.</p>
<dl class="method">
<dt id="pyspark.ml.param.Params.copy">
<code class="descname">copy</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/param.html#Params.copy"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.param.Params.copy" title="Permalink to this definition">¶</a></dt>
<dd><p>Creates a copy of this instance with the same uid and some
extra params. The default implementation creates a
shallow copy using <code class="xref py py-func docutils literal"><span class="pre">copy.copy()</span></code>, and then copies the
embedded and extra parameters over and returns the copy.
Subclasses should override this method if the default approach
is not sufficient.
:param extra: Extra parameters to copy to the new instance
:return: Copy of this instance</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.param.Params.explainParam">
<code class="descname">explainParam</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/param.html#Params.explainParam"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.param.Params.explainParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Explains a single param and returns its name, doc, and optional
default value and user-supplied value in a string.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.param.Params.explainParams">
<code class="descname">explainParams</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/param.html#Params.explainParams"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.param.Params.explainParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns the documentation of all params with their optionally
default values and user-supplied values.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.param.Params.extractParamMap">
<code class="descname">extractParamMap</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/param.html#Params.extractParamMap"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.param.Params.extractParamMap" title="Permalink to this definition">¶</a></dt>
<dd><p>Extracts the embedded default param values and user-supplied
values, and then merges them with extra values from input into
a flat param map, where the latter value is used if there exist
conflicts, i.e., with ordering: default param values <
user-supplied values < extra.
:param extra: extra param values
:return: merged param map</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.param.Params.getOrDefault">
<code class="descname">getOrDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/param.html#Params.getOrDefault"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.param.Params.getOrDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of a param in the user-supplied param map or its
default value. Raises an error if neither is set.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.param.Params.getParam">
<code class="descname">getParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/param.html#Params.getParam"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.param.Params.getParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets a param by its name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.param.Params.hasDefault">
<code class="descname">hasDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/param.html#Params.hasDefault"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.param.Params.hasDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param has a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.param.Params.hasParam">
<code class="descname">hasParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/param.html#Params.hasParam"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.param.Params.hasParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Tests whether this instance contains a param with a given
(string) name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.param.Params.isDefined">
<code class="descname">isDefined</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/param.html#Params.isDefined"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.param.Params.isDefined" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user or has
a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.param.Params.isSet">
<code class="descname">isSet</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/param.html#Params.isSet"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.param.Params.isSet" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.param.Params.params">
<code class="descname">params</code><a class="headerlink" href="#pyspark.ml.param.Params.params" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns all params ordered by name. The default implementation
uses <code class="xref py py-func docutils literal"><span class="pre">dir()</span></code> to get all attributes of type
<a class="reference internal" href="#pyspark.ml.param.Param" title="pyspark.ml.param.Param"><code class="xref py py-class docutils literal"><span class="pre">Param</span></code></a>.</p>
</dd></dl>
</dd></dl>
</div>
<div class="section" id="module-pyspark.ml.feature">
<span id="pyspark-ml-feature-module"></span><h2>pyspark.ml.feature module<a class="headerlink" href="#module-pyspark.ml.feature" title="Permalink to this headline">¶</a></h2>
<dl class="class">
<dt id="pyspark.ml.feature.Binarizer">
<em class="property">class </em><code class="descclassname">pyspark.ml.feature.</code><code class="descname">Binarizer</code><span class="sig-paren">(</span><em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/feature.html#Binarizer"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.feature.Binarizer" title="Permalink to this definition">¶</a></dt>
<dd><p>Binarize a column of continuous features given a threshold.</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="n">df</span> <span class="o">=</span> <span class="n">sqlContext</span><span class="o">.</span><span class="n">createDataFrame</span><span class="p">([(</span><span class="mf">0.5</span><span class="p">,)],</span> <span class="p">[</span><span class="s">"values"</span><span class="p">])</span>
<span class="gp">>>> </span><span class="n">binarizer</span> <span class="o">=</span> <span class="n">Binarizer</span><span class="p">(</span><span class="n">threshold</span><span class="o">=</span><span class="mf">1.0</span><span class="p">,</span> <span class="n">inputCol</span><span class="o">=</span><span class="s">"values"</span><span class="p">,</span> <span class="n">outputCol</span><span class="o">=</span><span class="s">"features"</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">binarizer</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">df</span><span class="p">)</span><span class="o">.</span><span class="n">head</span><span class="p">()</span><span class="o">.</span><span class="n">features</span>
<span class="go">0.0</span>
<span class="gp">>>> </span><span class="n">binarizer</span><span class="o">.</span><span class="n">setParams</span><span class="p">(</span><span class="n">outputCol</span><span class="o">=</span><span class="s">"freqs"</span><span class="p">)</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">df</span><span class="p">)</span><span class="o">.</span><span class="n">head</span><span class="p">()</span><span class="o">.</span><span class="n">freqs</span>
<span class="go">0.0</span>
<span class="gp">>>> </span><span class="n">params</span> <span class="o">=</span> <span class="p">{</span><span class="n">binarizer</span><span class="o">.</span><span class="n">threshold</span><span class="p">:</span> <span class="o">-</span><span class="mf">0.5</span><span class="p">,</span> <span class="n">binarizer</span><span class="o">.</span><span class="n">outputCol</span><span class="p">:</span> <span class="s">"vector"</span><span class="p">}</span>
<span class="gp">>>> </span><span class="n">binarizer</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">df</span><span class="p">,</span> <span class="n">params</span><span class="p">)</span><span class="o">.</span><span class="n">head</span><span class="p">()</span><span class="o">.</span><span class="n">vector</span>
<span class="go">1.0</span>
</pre></div>
</div>
<dl class="method">
<dt id="pyspark.ml.feature.Binarizer.copy">
<code class="descname">copy</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Binarizer.copy" title="Permalink to this definition">¶</a></dt>
<dd><p>Creates a copy of this instance with the same uid and some
extra params. The default implementation creates a
shallow copy using <code class="xref py py-func docutils literal"><span class="pre">copy.copy()</span></code>, and then copies the
embedded and extra parameters over and returns the copy.
Subclasses should override this method if the default approach
is not sufficient.
:param extra: Extra parameters to copy to the new instance
:return: Copy of this instance</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Binarizer.explainParam">
<code class="descname">explainParam</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Binarizer.explainParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Explains a single param and returns its name, doc, and optional
default value and user-supplied value in a string.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Binarizer.explainParams">
<code class="descname">explainParams</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Binarizer.explainParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns the documentation of all params with their optionally
default values and user-supplied values.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Binarizer.extractParamMap">
<code class="descname">extractParamMap</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Binarizer.extractParamMap" title="Permalink to this definition">¶</a></dt>
<dd><p>Extracts the embedded default param values and user-supplied
values, and then merges them with extra values from input into
a flat param map, where the latter value is used if there exist
conflicts, i.e., with ordering: default param values <
user-supplied values < extra.
:param extra: extra param values
:return: merged param map</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Binarizer.getInputCol">
<code class="descname">getInputCol</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Binarizer.getInputCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of inputCol or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Binarizer.getOrDefault">
<code class="descname">getOrDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Binarizer.getOrDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of a param in the user-supplied param map or its
default value. Raises an error if neither is set.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Binarizer.getOutputCol">
<code class="descname">getOutputCol</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Binarizer.getOutputCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of outputCol or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Binarizer.getParam">
<code class="descname">getParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Binarizer.getParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets a param by its name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Binarizer.getThreshold">
<code class="descname">getThreshold</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/feature.html#Binarizer.getThreshold"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.feature.Binarizer.getThreshold" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of threshold or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Binarizer.hasDefault">
<code class="descname">hasDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Binarizer.hasDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param has a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Binarizer.hasParam">
<code class="descname">hasParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Binarizer.hasParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Tests whether this instance contains a param with a given
(string) name.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.feature.Binarizer.inputCol">
<code class="descname">inputCol</code><em class="property"> = Param(parent='undefined', name='inputCol', doc='input column name')</em><a class="headerlink" href="#pyspark.ml.feature.Binarizer.inputCol" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Binarizer.isDefined">
<code class="descname">isDefined</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Binarizer.isDefined" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user or has
a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Binarizer.isSet">
<code class="descname">isSet</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Binarizer.isSet" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.feature.Binarizer.outputCol">
<code class="descname">outputCol</code><em class="property"> = Param(parent='undefined', name='outputCol', doc='output column name')</em><a class="headerlink" href="#pyspark.ml.feature.Binarizer.outputCol" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.feature.Binarizer.params">
<code class="descname">params</code><a class="headerlink" href="#pyspark.ml.feature.Binarizer.params" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns all params ordered by name. The default implementation
uses <code class="xref py py-func docutils literal"><span class="pre">dir()</span></code> to get all attributes of type
<code class="xref py py-class docutils literal"><span class="pre">Param</span></code>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Binarizer.setInputCol">
<code class="descname">setInputCol</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Binarizer.setInputCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.feature.Binarizer.inputCol" title="pyspark.ml.feature.Binarizer.inputCol"><code class="xref py py-attr docutils literal"><span class="pre">inputCol</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Binarizer.setOutputCol">
<code class="descname">setOutputCol</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Binarizer.setOutputCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.feature.Binarizer.outputCol" title="pyspark.ml.feature.Binarizer.outputCol"><code class="xref py py-attr docutils literal"><span class="pre">outputCol</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Binarizer.setParams">
<code class="descname">setParams</code><span class="sig-paren">(</span><em>self</em>, <em>threshold=0.0</em>, <em>inputCol=None</em>, <em>outputCol=None</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/feature.html#Binarizer.setParams"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.feature.Binarizer.setParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets params for this Binarizer.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Binarizer.setThreshold">
<code class="descname">setThreshold</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/feature.html#Binarizer.setThreshold"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.feature.Binarizer.setThreshold" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.feature.Binarizer.threshold" title="pyspark.ml.feature.Binarizer.threshold"><code class="xref py py-attr docutils literal"><span class="pre">threshold</span></code></a>.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.feature.Binarizer.threshold">
<code class="descname">threshold</code><em class="property"> = Param(parent='undefined', name='threshold', doc='threshold in binary classification prediction, in range [0, 1]')</em><a class="headerlink" href="#pyspark.ml.feature.Binarizer.threshold" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Binarizer.transform">
<code class="descname">transform</code><span class="sig-paren">(</span><em>dataset</em>, <em>params=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Binarizer.transform" title="Permalink to this definition">¶</a></dt>
<dd><p>Transforms the input dataset with optional parameters.</p>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple">
<li><strong>dataset</strong> – input dataset, which is an instance of
<a class="reference internal" href="pyspark.sql.html#pyspark.sql.DataFrame" title="pyspark.sql.DataFrame"><code class="xref py py-class docutils literal"><span class="pre">pyspark.sql.DataFrame</span></code></a></li>
<li><strong>params</strong> – an optional param map that overrides embedded
params.</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body"><p class="first last">transformed dataset</p>
</td>
</tr>
</tbody>
</table>
</dd></dl>
</dd></dl>
<dl class="class">
<dt id="pyspark.ml.feature.Bucketizer">
<em class="property">class </em><code class="descclassname">pyspark.ml.feature.</code><code class="descname">Bucketizer</code><span class="sig-paren">(</span><em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/feature.html#Bucketizer"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.feature.Bucketizer" title="Permalink to this definition">¶</a></dt>
<dd><p>Maps a column of continuous features to a column of feature buckets.</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="n">df</span> <span class="o">=</span> <span class="n">sqlContext</span><span class="o">.</span><span class="n">createDataFrame</span><span class="p">([(</span><span class="mf">0.1</span><span class="p">,),</span> <span class="p">(</span><span class="mf">0.4</span><span class="p">,),</span> <span class="p">(</span><span class="mf">1.2</span><span class="p">,),</span> <span class="p">(</span><span class="mf">1.5</span><span class="p">,)],</span> <span class="p">[</span><span class="s">"values"</span><span class="p">])</span>
<span class="gp">>>> </span><span class="n">bucketizer</span> <span class="o">=</span> <span class="n">Bucketizer</span><span class="p">(</span><span class="n">splits</span><span class="o">=</span><span class="p">[</span><span class="o">-</span><span class="nb">float</span><span class="p">(</span><span class="s">"inf"</span><span class="p">),</span> <span class="mf">0.5</span><span class="p">,</span> <span class="mf">1.4</span><span class="p">,</span> <span class="nb">float</span><span class="p">(</span><span class="s">"inf"</span><span class="p">)],</span>
<span class="gp">... </span> <span class="n">inputCol</span><span class="o">=</span><span class="s">"values"</span><span class="p">,</span> <span class="n">outputCol</span><span class="o">=</span><span class="s">"buckets"</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">bucketed</span> <span class="o">=</span> <span class="n">bucketizer</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">df</span><span class="p">)</span><span class="o">.</span><span class="n">collect</span><span class="p">()</span>
<span class="gp">>>> </span><span class="n">bucketed</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">buckets</span>
<span class="go">0.0</span>
<span class="gp">>>> </span><span class="n">bucketed</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span><span class="o">.</span><span class="n">buckets</span>
<span class="go">0.0</span>
<span class="gp">>>> </span><span class="n">bucketed</span><span class="p">[</span><span class="mi">2</span><span class="p">]</span><span class="o">.</span><span class="n">buckets</span>
<span class="go">1.0</span>
<span class="gp">>>> </span><span class="n">bucketed</span><span class="p">[</span><span class="mi">3</span><span class="p">]</span><span class="o">.</span><span class="n">buckets</span>
<span class="go">2.0</span>
<span class="gp">>>> </span><span class="n">bucketizer</span><span class="o">.</span><span class="n">setParams</span><span class="p">(</span><span class="n">outputCol</span><span class="o">=</span><span class="s">"b"</span><span class="p">)</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">df</span><span class="p">)</span><span class="o">.</span><span class="n">head</span><span class="p">()</span><span class="o">.</span><span class="n">b</span>
<span class="go">0.0</span>
</pre></div>
</div>
<dl class="method">
<dt id="pyspark.ml.feature.Bucketizer.copy">
<code class="descname">copy</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Bucketizer.copy" title="Permalink to this definition">¶</a></dt>
<dd><p>Creates a copy of this instance with the same uid and some
extra params. The default implementation creates a
shallow copy using <code class="xref py py-func docutils literal"><span class="pre">copy.copy()</span></code>, and then copies the
embedded and extra parameters over and returns the copy.
Subclasses should override this method if the default approach
is not sufficient.
:param extra: Extra parameters to copy to the new instance
:return: Copy of this instance</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Bucketizer.explainParam">
<code class="descname">explainParam</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Bucketizer.explainParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Explains a single param and returns its name, doc, and optional
default value and user-supplied value in a string.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Bucketizer.explainParams">
<code class="descname">explainParams</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Bucketizer.explainParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns the documentation of all params with their optionally
default values and user-supplied values.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Bucketizer.extractParamMap">
<code class="descname">extractParamMap</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Bucketizer.extractParamMap" title="Permalink to this definition">¶</a></dt>
<dd><p>Extracts the embedded default param values and user-supplied
values, and then merges them with extra values from input into
a flat param map, where the latter value is used if there exist
conflicts, i.e., with ordering: default param values <
user-supplied values < extra.
:param extra: extra param values
:return: merged param map</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Bucketizer.getInputCol">
<code class="descname">getInputCol</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Bucketizer.getInputCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of inputCol or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Bucketizer.getOrDefault">
<code class="descname">getOrDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Bucketizer.getOrDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of a param in the user-supplied param map or its
default value. Raises an error if neither is set.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Bucketizer.getOutputCol">
<code class="descname">getOutputCol</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Bucketizer.getOutputCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of outputCol or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Bucketizer.getParam">
<code class="descname">getParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Bucketizer.getParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets a param by its name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Bucketizer.getSplits">
<code class="descname">getSplits</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/feature.html#Bucketizer.getSplits"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.feature.Bucketizer.getSplits" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of threshold or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Bucketizer.hasDefault">
<code class="descname">hasDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Bucketizer.hasDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param has a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Bucketizer.hasParam">
<code class="descname">hasParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Bucketizer.hasParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Tests whether this instance contains a param with a given
(string) name.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.feature.Bucketizer.inputCol">
<code class="descname">inputCol</code><em class="property"> = Param(parent='undefined', name='inputCol', doc='input column name')</em><a class="headerlink" href="#pyspark.ml.feature.Bucketizer.inputCol" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Bucketizer.isDefined">
<code class="descname">isDefined</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Bucketizer.isDefined" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user or has
a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Bucketizer.isSet">
<code class="descname">isSet</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Bucketizer.isSet" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.feature.Bucketizer.outputCol">
<code class="descname">outputCol</code><em class="property"> = Param(parent='undefined', name='outputCol', doc='output column name')</em><a class="headerlink" href="#pyspark.ml.feature.Bucketizer.outputCol" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.feature.Bucketizer.params">
<code class="descname">params</code><a class="headerlink" href="#pyspark.ml.feature.Bucketizer.params" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns all params ordered by name. The default implementation
uses <code class="xref py py-func docutils literal"><span class="pre">dir()</span></code> to get all attributes of type
<code class="xref py py-class docutils literal"><span class="pre">Param</span></code>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Bucketizer.setInputCol">
<code class="descname">setInputCol</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Bucketizer.setInputCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.feature.Bucketizer.inputCol" title="pyspark.ml.feature.Bucketizer.inputCol"><code class="xref py py-attr docutils literal"><span class="pre">inputCol</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Bucketizer.setOutputCol">
<code class="descname">setOutputCol</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Bucketizer.setOutputCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.feature.Bucketizer.outputCol" title="pyspark.ml.feature.Bucketizer.outputCol"><code class="xref py py-attr docutils literal"><span class="pre">outputCol</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Bucketizer.setParams">
<code class="descname">setParams</code><span class="sig-paren">(</span><em>self</em>, <em>splits=None</em>, <em>inputCol=None</em>, <em>outputCol=None</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/feature.html#Bucketizer.setParams"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.feature.Bucketizer.setParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets params for this Bucketizer.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Bucketizer.setSplits">
<code class="descname">setSplits</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/feature.html#Bucketizer.setSplits"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.feature.Bucketizer.setSplits" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.feature.Bucketizer.splits" title="pyspark.ml.feature.Bucketizer.splits"><code class="xref py py-attr docutils literal"><span class="pre">splits</span></code></a>.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.feature.Bucketizer.splits">
<code class="descname">splits</code><em class="property"> = Param(parent='undefined', name='splits', doc='Split points for mapping continuous features into buckets. With n+1 splits, there are n buckets. A bucket defined by splits x,y holds values in the range [x,y) except the last bucket, which also includes y. The splits should be strictly increasing. Values at -inf, inf must be explicitly provided to cover all Double values; otherwise, values outside the splits specified will be treated as errors.')</em><a class="headerlink" href="#pyspark.ml.feature.Bucketizer.splits" title="Permalink to this definition">¶</a></dt>
<dd><p>param for Splitting points for mapping continuous features into buckets. With n+1 splits,</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Bucketizer.transform">
<code class="descname">transform</code><span class="sig-paren">(</span><em>dataset</em>, <em>params=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Bucketizer.transform" title="Permalink to this definition">¶</a></dt>
<dd><p>Transforms the input dataset with optional parameters.</p>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple">
<li><strong>dataset</strong> – input dataset, which is an instance of
<a class="reference internal" href="pyspark.sql.html#pyspark.sql.DataFrame" title="pyspark.sql.DataFrame"><code class="xref py py-class docutils literal"><span class="pre">pyspark.sql.DataFrame</span></code></a></li>
<li><strong>params</strong> – an optional param map that overrides embedded
params.</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body"><p class="first last">transformed dataset</p>
</td>
</tr>
</tbody>
</table>
</dd></dl>
</dd></dl>
<dl class="class">
<dt id="pyspark.ml.feature.ElementwiseProduct">
<em class="property">class </em><code class="descclassname">pyspark.ml.feature.</code><code class="descname">ElementwiseProduct</code><span class="sig-paren">(</span><em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/feature.html#ElementwiseProduct"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.feature.ElementwiseProduct" title="Permalink to this definition">¶</a></dt>
<dd><p>Outputs the Hadamard product (i.e., the element-wise product) of each input vector
with a provided “weight” vector. In other words, it scales each column of the dataset
by a scalar multiplier.</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="kn">from</span> <span class="nn">pyspark.mllib.linalg</span> <span class="kn">import</span> <span class="n">Vectors</span>
<span class="gp">>>> </span><span class="n">df</span> <span class="o">=</span> <span class="n">sqlContext</span><span class="o">.</span><span class="n">createDataFrame</span><span class="p">([(</span><span class="n">Vectors</span><span class="o">.</span><span class="n">dense</span><span class="p">([</span><span class="mf">2.0</span><span class="p">,</span> <span class="mf">1.0</span><span class="p">,</span> <span class="mf">3.0</span><span class="p">]),)],</span> <span class="p">[</span><span class="s">"values"</span><span class="p">])</span>
<span class="gp">>>> </span><span class="n">ep</span> <span class="o">=</span> <span class="n">ElementwiseProduct</span><span class="p">(</span><span class="n">scalingVec</span><span class="o">=</span><span class="n">Vectors</span><span class="o">.</span><span class="n">dense</span><span class="p">([</span><span class="mf">1.0</span><span class="p">,</span> <span class="mf">2.0</span><span class="p">,</span> <span class="mf">3.0</span><span class="p">]),</span>
<span class="gp">... </span> <span class="n">inputCol</span><span class="o">=</span><span class="s">"values"</span><span class="p">,</span> <span class="n">outputCol</span><span class="o">=</span><span class="s">"eprod"</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">ep</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">df</span><span class="p">)</span><span class="o">.</span><span class="n">head</span><span class="p">()</span><span class="o">.</span><span class="n">eprod</span>
<span class="go">DenseVector([2.0, 2.0, 9.0])</span>
<span class="gp">>>> </span><span class="n">ep</span><span class="o">.</span><span class="n">setParams</span><span class="p">(</span><span class="n">scalingVec</span><span class="o">=</span><span class="n">Vectors</span><span class="o">.</span><span class="n">dense</span><span class="p">([</span><span class="mf">2.0</span><span class="p">,</span> <span class="mf">3.0</span><span class="p">,</span> <span class="mf">5.0</span><span class="p">]))</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">df</span><span class="p">)</span><span class="o">.</span><span class="n">head</span><span class="p">()</span><span class="o">.</span><span class="n">eprod</span>
<span class="go">DenseVector([4.0, 3.0, 15.0])</span>
</pre></div>
</div>
<dl class="method">
<dt id="pyspark.ml.feature.ElementwiseProduct.copy">
<code class="descname">copy</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.ElementwiseProduct.copy" title="Permalink to this definition">¶</a></dt>
<dd><p>Creates a copy of this instance with the same uid and some
extra params. The default implementation creates a
shallow copy using <code class="xref py py-func docutils literal"><span class="pre">copy.copy()</span></code>, and then copies the
embedded and extra parameters over and returns the copy.
Subclasses should override this method if the default approach
is not sufficient.
:param extra: Extra parameters to copy to the new instance
:return: Copy of this instance</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.ElementwiseProduct.explainParam">
<code class="descname">explainParam</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.ElementwiseProduct.explainParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Explains a single param and returns its name, doc, and optional
default value and user-supplied value in a string.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.ElementwiseProduct.explainParams">
<code class="descname">explainParams</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.ElementwiseProduct.explainParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns the documentation of all params with their optionally
default values and user-supplied values.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.ElementwiseProduct.extractParamMap">
<code class="descname">extractParamMap</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.ElementwiseProduct.extractParamMap" title="Permalink to this definition">¶</a></dt>
<dd><p>Extracts the embedded default param values and user-supplied
values, and then merges them with extra values from input into
a flat param map, where the latter value is used if there exist
conflicts, i.e., with ordering: default param values <
user-supplied values < extra.
:param extra: extra param values
:return: merged param map</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.ElementwiseProduct.getInputCol">
<code class="descname">getInputCol</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.ElementwiseProduct.getInputCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of inputCol or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.ElementwiseProduct.getOrDefault">
<code class="descname">getOrDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.ElementwiseProduct.getOrDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of a param in the user-supplied param map or its
default value. Raises an error if neither is set.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.ElementwiseProduct.getOutputCol">
<code class="descname">getOutputCol</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.ElementwiseProduct.getOutputCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of outputCol or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.ElementwiseProduct.getParam">
<code class="descname">getParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.ElementwiseProduct.getParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets a param by its name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.ElementwiseProduct.getScalingVec">
<code class="descname">getScalingVec</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/feature.html#ElementwiseProduct.getScalingVec"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.feature.ElementwiseProduct.getScalingVec" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of scalingVec or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.ElementwiseProduct.hasDefault">
<code class="descname">hasDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.ElementwiseProduct.hasDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param has a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.ElementwiseProduct.hasParam">
<code class="descname">hasParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.ElementwiseProduct.hasParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Tests whether this instance contains a param with a given
(string) name.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.feature.ElementwiseProduct.inputCol">
<code class="descname">inputCol</code><em class="property"> = Param(parent='undefined', name='inputCol', doc='input column name')</em><a class="headerlink" href="#pyspark.ml.feature.ElementwiseProduct.inputCol" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.ElementwiseProduct.isDefined">
<code class="descname">isDefined</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.ElementwiseProduct.isDefined" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user or has
a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.ElementwiseProduct.isSet">
<code class="descname">isSet</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.ElementwiseProduct.isSet" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.feature.ElementwiseProduct.outputCol">
<code class="descname">outputCol</code><em class="property"> = Param(parent='undefined', name='outputCol', doc='output column name')</em><a class="headerlink" href="#pyspark.ml.feature.ElementwiseProduct.outputCol" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.feature.ElementwiseProduct.params">
<code class="descname">params</code><a class="headerlink" href="#pyspark.ml.feature.ElementwiseProduct.params" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns all params ordered by name. The default implementation
uses <code class="xref py py-func docutils literal"><span class="pre">dir()</span></code> to get all attributes of type
<code class="xref py py-class docutils literal"><span class="pre">Param</span></code>.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.feature.ElementwiseProduct.scalingVec">
<code class="descname">scalingVec</code><em class="property"> = Param(parent='undefined', name='scalingVec', doc='vector for hadamard product, it must be MLlib Vector type.')</em><a class="headerlink" href="#pyspark.ml.feature.ElementwiseProduct.scalingVec" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.ElementwiseProduct.setInputCol">
<code class="descname">setInputCol</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.ElementwiseProduct.setInputCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.feature.ElementwiseProduct.inputCol" title="pyspark.ml.feature.ElementwiseProduct.inputCol"><code class="xref py py-attr docutils literal"><span class="pre">inputCol</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.ElementwiseProduct.setOutputCol">
<code class="descname">setOutputCol</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.ElementwiseProduct.setOutputCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.feature.ElementwiseProduct.outputCol" title="pyspark.ml.feature.ElementwiseProduct.outputCol"><code class="xref py py-attr docutils literal"><span class="pre">outputCol</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.ElementwiseProduct.setParams">
<code class="descname">setParams</code><span class="sig-paren">(</span><em>self</em>, <em>scalingVec=None</em>, <em>inputCol=None</em>, <em>outputCol=None</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/feature.html#ElementwiseProduct.setParams"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.feature.ElementwiseProduct.setParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets params for this ElementwiseProduct.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.ElementwiseProduct.setScalingVec">
<code class="descname">setScalingVec</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/feature.html#ElementwiseProduct.setScalingVec"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.feature.ElementwiseProduct.setScalingVec" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.feature.ElementwiseProduct.scalingVec" title="pyspark.ml.feature.ElementwiseProduct.scalingVec"><code class="xref py py-attr docutils literal"><span class="pre">scalingVec</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.ElementwiseProduct.transform">
<code class="descname">transform</code><span class="sig-paren">(</span><em>dataset</em>, <em>params=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.ElementwiseProduct.transform" title="Permalink to this definition">¶</a></dt>
<dd><p>Transforms the input dataset with optional parameters.</p>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple">
<li><strong>dataset</strong> – input dataset, which is an instance of
<a class="reference internal" href="pyspark.sql.html#pyspark.sql.DataFrame" title="pyspark.sql.DataFrame"><code class="xref py py-class docutils literal"><span class="pre">pyspark.sql.DataFrame</span></code></a></li>
<li><strong>params</strong> – an optional param map that overrides embedded
params.</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body"><p class="first last">transformed dataset</p>
</td>
</tr>
</tbody>
</table>
</dd></dl>
</dd></dl>
<dl class="class">
<dt id="pyspark.ml.feature.HashingTF">
<em class="property">class </em><code class="descclassname">pyspark.ml.feature.</code><code class="descname">HashingTF</code><span class="sig-paren">(</span><em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/feature.html#HashingTF"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.feature.HashingTF" title="Permalink to this definition">¶</a></dt>
<dd><p>Maps a sequence of terms to their term frequencies using the
hashing trick.</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="n">df</span> <span class="o">=</span> <span class="n">sqlContext</span><span class="o">.</span><span class="n">createDataFrame</span><span class="p">([([</span><span class="s">"a"</span><span class="p">,</span> <span class="s">"b"</span><span class="p">,</span> <span class="s">"c"</span><span class="p">],)],</span> <span class="p">[</span><span class="s">"words"</span><span class="p">])</span>
<span class="gp">>>> </span><span class="n">hashingTF</span> <span class="o">=</span> <span class="n">HashingTF</span><span class="p">(</span><span class="n">numFeatures</span><span class="o">=</span><span class="mi">10</span><span class="p">,</span> <span class="n">inputCol</span><span class="o">=</span><span class="s">"words"</span><span class="p">,</span> <span class="n">outputCol</span><span class="o">=</span><span class="s">"features"</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">hashingTF</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">df</span><span class="p">)</span><span class="o">.</span><span class="n">head</span><span class="p">()</span><span class="o">.</span><span class="n">features</span>
<span class="go">SparseVector(10, {7: 1.0, 8: 1.0, 9: 1.0})</span>
<span class="gp">>>> </span><span class="n">hashingTF</span><span class="o">.</span><span class="n">setParams</span><span class="p">(</span><span class="n">outputCol</span><span class="o">=</span><span class="s">"freqs"</span><span class="p">)</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">df</span><span class="p">)</span><span class="o">.</span><span class="n">head</span><span class="p">()</span><span class="o">.</span><span class="n">freqs</span>
<span class="go">SparseVector(10, {7: 1.0, 8: 1.0, 9: 1.0})</span>
<span class="gp">>>> </span><span class="n">params</span> <span class="o">=</span> <span class="p">{</span><span class="n">hashingTF</span><span class="o">.</span><span class="n">numFeatures</span><span class="p">:</span> <span class="mi">5</span><span class="p">,</span> <span class="n">hashingTF</span><span class="o">.</span><span class="n">outputCol</span><span class="p">:</span> <span class="s">"vector"</span><span class="p">}</span>
<span class="gp">>>> </span><span class="n">hashingTF</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">df</span><span class="p">,</span> <span class="n">params</span><span class="p">)</span><span class="o">.</span><span class="n">head</span><span class="p">()</span><span class="o">.</span><span class="n">vector</span>
<span class="go">SparseVector(5, {2: 1.0, 3: 1.0, 4: 1.0})</span>
</pre></div>
</div>
<dl class="method">
<dt id="pyspark.ml.feature.HashingTF.copy">
<code class="descname">copy</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.HashingTF.copy" title="Permalink to this definition">¶</a></dt>
<dd><p>Creates a copy of this instance with the same uid and some
extra params. The default implementation creates a
shallow copy using <code class="xref py py-func docutils literal"><span class="pre">copy.copy()</span></code>, and then copies the
embedded and extra parameters over and returns the copy.
Subclasses should override this method if the default approach
is not sufficient.
:param extra: Extra parameters to copy to the new instance
:return: Copy of this instance</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.HashingTF.explainParam">
<code class="descname">explainParam</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.HashingTF.explainParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Explains a single param and returns its name, doc, and optional
default value and user-supplied value in a string.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.HashingTF.explainParams">
<code class="descname">explainParams</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.HashingTF.explainParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns the documentation of all params with their optionally
default values and user-supplied values.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.HashingTF.extractParamMap">
<code class="descname">extractParamMap</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.HashingTF.extractParamMap" title="Permalink to this definition">¶</a></dt>
<dd><p>Extracts the embedded default param values and user-supplied
values, and then merges them with extra values from input into
a flat param map, where the latter value is used if there exist
conflicts, i.e., with ordering: default param values <
user-supplied values < extra.
:param extra: extra param values
:return: merged param map</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.HashingTF.getInputCol">
<code class="descname">getInputCol</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.HashingTF.getInputCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of inputCol or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.HashingTF.getNumFeatures">
<code class="descname">getNumFeatures</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.HashingTF.getNumFeatures" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of numFeatures or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.HashingTF.getOrDefault">
<code class="descname">getOrDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.HashingTF.getOrDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of a param in the user-supplied param map or its
default value. Raises an error if neither is set.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.HashingTF.getOutputCol">
<code class="descname">getOutputCol</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.HashingTF.getOutputCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of outputCol or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.HashingTF.getParam">
<code class="descname">getParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.HashingTF.getParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets a param by its name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.HashingTF.hasDefault">
<code class="descname">hasDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.HashingTF.hasDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param has a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.HashingTF.hasParam">
<code class="descname">hasParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.HashingTF.hasParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Tests whether this instance contains a param with a given
(string) name.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.feature.HashingTF.inputCol">
<code class="descname">inputCol</code><em class="property"> = Param(parent='undefined', name='inputCol', doc='input column name')</em><a class="headerlink" href="#pyspark.ml.feature.HashingTF.inputCol" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.HashingTF.isDefined">
<code class="descname">isDefined</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.HashingTF.isDefined" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user or has
a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.HashingTF.isSet">
<code class="descname">isSet</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.HashingTF.isSet" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.feature.HashingTF.numFeatures">
<code class="descname">numFeatures</code><em class="property"> = Param(parent='undefined', name='numFeatures', doc='number of features')</em><a class="headerlink" href="#pyspark.ml.feature.HashingTF.numFeatures" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.feature.HashingTF.outputCol">
<code class="descname">outputCol</code><em class="property"> = Param(parent='undefined', name='outputCol', doc='output column name')</em><a class="headerlink" href="#pyspark.ml.feature.HashingTF.outputCol" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.feature.HashingTF.params">
<code class="descname">params</code><a class="headerlink" href="#pyspark.ml.feature.HashingTF.params" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns all params ordered by name. The default implementation
uses <code class="xref py py-func docutils literal"><span class="pre">dir()</span></code> to get all attributes of type
<code class="xref py py-class docutils literal"><span class="pre">Param</span></code>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.HashingTF.setInputCol">
<code class="descname">setInputCol</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.HashingTF.setInputCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.feature.HashingTF.inputCol" title="pyspark.ml.feature.HashingTF.inputCol"><code class="xref py py-attr docutils literal"><span class="pre">inputCol</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.HashingTF.setNumFeatures">
<code class="descname">setNumFeatures</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.HashingTF.setNumFeatures" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.feature.HashingTF.numFeatures" title="pyspark.ml.feature.HashingTF.numFeatures"><code class="xref py py-attr docutils literal"><span class="pre">numFeatures</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.HashingTF.setOutputCol">
<code class="descname">setOutputCol</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.HashingTF.setOutputCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.feature.HashingTF.outputCol" title="pyspark.ml.feature.HashingTF.outputCol"><code class="xref py py-attr docutils literal"><span class="pre">outputCol</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.HashingTF.setParams">
<code class="descname">setParams</code><span class="sig-paren">(</span><em>self</em>, <em>numFeatures=1 << 18</em>, <em>inputCol=None</em>, <em>outputCol=None</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/feature.html#HashingTF.setParams"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.feature.HashingTF.setParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets params for this HashingTF.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.HashingTF.transform">
<code class="descname">transform</code><span class="sig-paren">(</span><em>dataset</em>, <em>params=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.HashingTF.transform" title="Permalink to this definition">¶</a></dt>
<dd><p>Transforms the input dataset with optional parameters.</p>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple">
<li><strong>dataset</strong> – input dataset, which is an instance of
<a class="reference internal" href="pyspark.sql.html#pyspark.sql.DataFrame" title="pyspark.sql.DataFrame"><code class="xref py py-class docutils literal"><span class="pre">pyspark.sql.DataFrame</span></code></a></li>
<li><strong>params</strong> – an optional param map that overrides embedded
params.</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body"><p class="first last">transformed dataset</p>
</td>
</tr>
</tbody>
</table>
</dd></dl>
</dd></dl>
<dl class="class">
<dt id="pyspark.ml.feature.IDF">
<em class="property">class </em><code class="descclassname">pyspark.ml.feature.</code><code class="descname">IDF</code><span class="sig-paren">(</span><em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/feature.html#IDF"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.feature.IDF" title="Permalink to this definition">¶</a></dt>
<dd><p>Compute the Inverse Document Frequency (IDF) given a collection of documents.</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="kn">from</span> <span class="nn">pyspark.mllib.linalg</span> <span class="kn">import</span> <span class="n">DenseVector</span>
<span class="gp">>>> </span><span class="n">df</span> <span class="o">=</span> <span class="n">sqlContext</span><span class="o">.</span><span class="n">createDataFrame</span><span class="p">([(</span><span class="n">DenseVector</span><span class="p">([</span><span class="mf">1.0</span><span class="p">,</span> <span class="mf">2.0</span><span class="p">]),),</span>
<span class="gp">... </span> <span class="p">(</span><span class="n">DenseVector</span><span class="p">([</span><span class="mf">0.0</span><span class="p">,</span> <span class="mf">1.0</span><span class="p">]),),</span> <span class="p">(</span><span class="n">DenseVector</span><span class="p">([</span><span class="mf">3.0</span><span class="p">,</span> <span class="mf">0.2</span><span class="p">]),)],</span> <span class="p">[</span><span class="s">"tf"</span><span class="p">])</span>
<span class="gp">>>> </span><span class="n">idf</span> <span class="o">=</span> <span class="n">IDF</span><span class="p">(</span><span class="n">minDocFreq</span><span class="o">=</span><span class="mi">3</span><span class="p">,</span> <span class="n">inputCol</span><span class="o">=</span><span class="s">"tf"</span><span class="p">,</span> <span class="n">outputCol</span><span class="o">=</span><span class="s">"idf"</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">idf</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">df</span><span class="p">)</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">df</span><span class="p">)</span><span class="o">.</span><span class="n">head</span><span class="p">()</span><span class="o">.</span><span class="n">idf</span>
<span class="go">DenseVector([0.0, 0.0])</span>
<span class="gp">>>> </span><span class="n">idf</span><span class="o">.</span><span class="n">setParams</span><span class="p">(</span><span class="n">outputCol</span><span class="o">=</span><span class="s">"freqs"</span><span class="p">)</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">df</span><span class="p">)</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">df</span><span class="p">)</span><span class="o">.</span><span class="n">collect</span><span class="p">()[</span><span class="mi">1</span><span class="p">]</span><span class="o">.</span><span class="n">freqs</span>
<span class="go">DenseVector([0.0, 0.0])</span>
<span class="gp">>>> </span><span class="n">params</span> <span class="o">=</span> <span class="p">{</span><span class="n">idf</span><span class="o">.</span><span class="n">minDocFreq</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span> <span class="n">idf</span><span class="o">.</span><span class="n">outputCol</span><span class="p">:</span> <span class="s">"vector"</span><span class="p">}</span>
<span class="gp">>>> </span><span class="n">idf</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">df</span><span class="p">,</span> <span class="n">params</span><span class="p">)</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">df</span><span class="p">)</span><span class="o">.</span><span class="n">head</span><span class="p">()</span><span class="o">.</span><span class="n">vector</span>
<span class="go">DenseVector([0.2877, 0.0])</span>
</pre></div>
</div>
<dl class="method">
<dt id="pyspark.ml.feature.IDF.copy">
<code class="descname">copy</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.IDF.copy" title="Permalink to this definition">¶</a></dt>
<dd><p>Creates a copy of this instance with the same uid and some
extra params. The default implementation creates a
shallow copy using <code class="xref py py-func docutils literal"><span class="pre">copy.copy()</span></code>, and then copies the
embedded and extra parameters over and returns the copy.
Subclasses should override this method if the default approach
is not sufficient.
:param extra: Extra parameters to copy to the new instance
:return: Copy of this instance</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.IDF.explainParam">
<code class="descname">explainParam</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.IDF.explainParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Explains a single param and returns its name, doc, and optional
default value and user-supplied value in a string.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.IDF.explainParams">
<code class="descname">explainParams</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.IDF.explainParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns the documentation of all params with their optionally
default values and user-supplied values.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.IDF.extractParamMap">
<code class="descname">extractParamMap</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.IDF.extractParamMap" title="Permalink to this definition">¶</a></dt>
<dd><p>Extracts the embedded default param values and user-supplied
values, and then merges them with extra values from input into
a flat param map, where the latter value is used if there exist
conflicts, i.e., with ordering: default param values <
user-supplied values < extra.
:param extra: extra param values
:return: merged param map</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.IDF.fit">
<code class="descname">fit</code><span class="sig-paren">(</span><em>dataset</em>, <em>params=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.IDF.fit" title="Permalink to this definition">¶</a></dt>
<dd><p>Fits a model to the input dataset with optional parameters.</p>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple">
<li><strong>dataset</strong> – input dataset, which is an instance of
<a class="reference internal" href="pyspark.sql.html#pyspark.sql.DataFrame" title="pyspark.sql.DataFrame"><code class="xref py py-class docutils literal"><span class="pre">pyspark.sql.DataFrame</span></code></a></li>
<li><strong>params</strong> – an optional param map that overrides embedded
params. If a list/tuple of param maps is given,
this calls fit on each param map and returns a
list of models.</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body"><p class="first last">fitted model(s)</p>
</td>
</tr>
</tbody>
</table>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.IDF.getInputCol">
<code class="descname">getInputCol</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.IDF.getInputCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of inputCol or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.IDF.getMinDocFreq">
<code class="descname">getMinDocFreq</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/feature.html#IDF.getMinDocFreq"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.feature.IDF.getMinDocFreq" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of minDocFreq or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.IDF.getOrDefault">
<code class="descname">getOrDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.IDF.getOrDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of a param in the user-supplied param map or its
default value. Raises an error if neither is set.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.IDF.getOutputCol">
<code class="descname">getOutputCol</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.IDF.getOutputCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of outputCol or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.IDF.getParam">
<code class="descname">getParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.IDF.getParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets a param by its name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.IDF.hasDefault">
<code class="descname">hasDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.IDF.hasDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param has a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.IDF.hasParam">
<code class="descname">hasParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.IDF.hasParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Tests whether this instance contains a param with a given
(string) name.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.feature.IDF.inputCol">
<code class="descname">inputCol</code><em class="property"> = Param(parent='undefined', name='inputCol', doc='input column name')</em><a class="headerlink" href="#pyspark.ml.feature.IDF.inputCol" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.IDF.isDefined">
<code class="descname">isDefined</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.IDF.isDefined" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user or has
a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.IDF.isSet">
<code class="descname">isSet</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.IDF.isSet" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.feature.IDF.minDocFreq">
<code class="descname">minDocFreq</code><em class="property"> = Param(parent='undefined', name='minDocFreq', doc='minimum of documents in which a term should appear for filtering')</em><a class="headerlink" href="#pyspark.ml.feature.IDF.minDocFreq" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.feature.IDF.outputCol">
<code class="descname">outputCol</code><em class="property"> = Param(parent='undefined', name='outputCol', doc='output column name')</em><a class="headerlink" href="#pyspark.ml.feature.IDF.outputCol" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.feature.IDF.params">
<code class="descname">params</code><a class="headerlink" href="#pyspark.ml.feature.IDF.params" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns all params ordered by name. The default implementation
uses <code class="xref py py-func docutils literal"><span class="pre">dir()</span></code> to get all attributes of type
<code class="xref py py-class docutils literal"><span class="pre">Param</span></code>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.IDF.setInputCol">
<code class="descname">setInputCol</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.IDF.setInputCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.feature.IDF.inputCol" title="pyspark.ml.feature.IDF.inputCol"><code class="xref py py-attr docutils literal"><span class="pre">inputCol</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.IDF.setMinDocFreq">
<code class="descname">setMinDocFreq</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/feature.html#IDF.setMinDocFreq"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.feature.IDF.setMinDocFreq" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.feature.IDF.minDocFreq" title="pyspark.ml.feature.IDF.minDocFreq"><code class="xref py py-attr docutils literal"><span class="pre">minDocFreq</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.IDF.setOutputCol">
<code class="descname">setOutputCol</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.IDF.setOutputCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.feature.IDF.outputCol" title="pyspark.ml.feature.IDF.outputCol"><code class="xref py py-attr docutils literal"><span class="pre">outputCol</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.IDF.setParams">
<code class="descname">setParams</code><span class="sig-paren">(</span><em>self</em>, <em>minDocFreq=0</em>, <em>inputCol=None</em>, <em>outputCol=None</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/feature.html#IDF.setParams"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.feature.IDF.setParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets params for this IDF.</p>
</dd></dl>
</dd></dl>
<dl class="class">
<dt id="pyspark.ml.feature.IDFModel">
<em class="property">class </em><code class="descclassname">pyspark.ml.feature.</code><code class="descname">IDFModel</code><span class="sig-paren">(</span><em>java_model</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/feature.html#IDFModel"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.feature.IDFModel" title="Permalink to this definition">¶</a></dt>
<dd><p>Model fitted by IDF.</p>
<dl class="method">
<dt id="pyspark.ml.feature.IDFModel.copy">
<code class="descname">copy</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.IDFModel.copy" title="Permalink to this definition">¶</a></dt>
<dd><p>Creates a copy of this instance with the same uid and some
extra params. This implementation first calls Params.copy and
then make a copy of the companion Java model with extra params.
So both the Python wrapper and the Java model get copied.
:param extra: Extra parameters to copy to the new instance
:return: Copy of this instance</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.IDFModel.explainParam">
<code class="descname">explainParam</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.IDFModel.explainParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Explains a single param and returns its name, doc, and optional
default value and user-supplied value in a string.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.IDFModel.explainParams">
<code class="descname">explainParams</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.IDFModel.explainParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns the documentation of all params with their optionally
default values and user-supplied values.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.IDFModel.extractParamMap">
<code class="descname">extractParamMap</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.IDFModel.extractParamMap" title="Permalink to this definition">¶</a></dt>
<dd><p>Extracts the embedded default param values and user-supplied
values, and then merges them with extra values from input into
a flat param map, where the latter value is used if there exist
conflicts, i.e., with ordering: default param values <
user-supplied values < extra.
:param extra: extra param values
:return: merged param map</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.IDFModel.getOrDefault">
<code class="descname">getOrDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.IDFModel.getOrDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of a param in the user-supplied param map or its
default value. Raises an error if neither is set.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.IDFModel.getParam">
<code class="descname">getParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.IDFModel.getParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets a param by its name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.IDFModel.hasDefault">
<code class="descname">hasDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.IDFModel.hasDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param has a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.IDFModel.hasParam">
<code class="descname">hasParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.IDFModel.hasParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Tests whether this instance contains a param with a given
(string) name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.IDFModel.isDefined">
<code class="descname">isDefined</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.IDFModel.isDefined" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user or has
a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.IDFModel.isSet">
<code class="descname">isSet</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.IDFModel.isSet" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.feature.IDFModel.params">
<code class="descname">params</code><a class="headerlink" href="#pyspark.ml.feature.IDFModel.params" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns all params ordered by name. The default implementation
uses <code class="xref py py-func docutils literal"><span class="pre">dir()</span></code> to get all attributes of type
<code class="xref py py-class docutils literal"><span class="pre">Param</span></code>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.IDFModel.transform">
<code class="descname">transform</code><span class="sig-paren">(</span><em>dataset</em>, <em>params=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.IDFModel.transform" title="Permalink to this definition">¶</a></dt>
<dd><p>Transforms the input dataset with optional parameters.</p>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple">
<li><strong>dataset</strong> – input dataset, which is an instance of
<a class="reference internal" href="pyspark.sql.html#pyspark.sql.DataFrame" title="pyspark.sql.DataFrame"><code class="xref py py-class docutils literal"><span class="pre">pyspark.sql.DataFrame</span></code></a></li>
<li><strong>params</strong> – an optional param map that overrides embedded
params.</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body"><p class="first last">transformed dataset</p>
</td>
</tr>
</tbody>
</table>
</dd></dl>
</dd></dl>
<dl class="class">
<dt id="pyspark.ml.feature.NGram">
<em class="property">class </em><code class="descclassname">pyspark.ml.feature.</code><code class="descname">NGram</code><span class="sig-paren">(</span><em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/feature.html#NGram"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.feature.NGram" title="Permalink to this definition">¶</a></dt>
<dd><p>A feature transformer that converts the input array of strings into an array of n-grams. Null
values in the input array are ignored.
It returns an array of n-grams where each n-gram is represented by a space-separated string of
words.
When the input is empty, an empty array is returned.
When the input array length is less than n (number of elements per n-gram), no n-grams are
returned.</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="n">df</span> <span class="o">=</span> <span class="n">sqlContext</span><span class="o">.</span><span class="n">createDataFrame</span><span class="p">([</span><span class="n">Row</span><span class="p">(</span><span class="n">inputTokens</span><span class="o">=</span><span class="p">[</span><span class="s">"a"</span><span class="p">,</span> <span class="s">"b"</span><span class="p">,</span> <span class="s">"c"</span><span class="p">,</span> <span class="s">"d"</span><span class="p">,</span> <span class="s">"e"</span><span class="p">])])</span>
<span class="gp">>>> </span><span class="n">ngram</span> <span class="o">=</span> <span class="n">NGram</span><span class="p">(</span><span class="n">n</span><span class="o">=</span><span class="mi">2</span><span class="p">,</span> <span class="n">inputCol</span><span class="o">=</span><span class="s">"inputTokens"</span><span class="p">,</span> <span class="n">outputCol</span><span class="o">=</span><span class="s">"nGrams"</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">ngram</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">df</span><span class="p">)</span><span class="o">.</span><span class="n">head</span><span class="p">()</span>
<span class="go">Row(inputTokens=[u'a', u'b', u'c', u'd', u'e'], nGrams=[u'a b', u'b c', u'c d', u'd e'])</span>
<span class="gp">>>> </span><span class="c"># Change n-gram length</span>
<span class="gp">>>> </span><span class="n">ngram</span><span class="o">.</span><span class="n">setParams</span><span class="p">(</span><span class="n">n</span><span class="o">=</span><span class="mi">4</span><span class="p">)</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">df</span><span class="p">)</span><span class="o">.</span><span class="n">head</span><span class="p">()</span>
<span class="go">Row(inputTokens=[u'a', u'b', u'c', u'd', u'e'], nGrams=[u'a b c d', u'b c d e'])</span>
<span class="gp">>>> </span><span class="c"># Temporarily modify output column.</span>
<span class="gp">>>> </span><span class="n">ngram</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">df</span><span class="p">,</span> <span class="p">{</span><span class="n">ngram</span><span class="o">.</span><span class="n">outputCol</span><span class="p">:</span> <span class="s">"output"</span><span class="p">})</span><span class="o">.</span><span class="n">head</span><span class="p">()</span>
<span class="go">Row(inputTokens=[u'a', u'b', u'c', u'd', u'e'], output=[u'a b c d', u'b c d e'])</span>
<span class="gp">>>> </span><span class="n">ngram</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">df</span><span class="p">)</span><span class="o">.</span><span class="n">head</span><span class="p">()</span>
<span class="go">Row(inputTokens=[u'a', u'b', u'c', u'd', u'e'], nGrams=[u'a b c d', u'b c d e'])</span>
<span class="gp">>>> </span><span class="c"># Must use keyword arguments to specify params.</span>
<span class="gp">>>> </span><span class="n">ngram</span><span class="o">.</span><span class="n">setParams</span><span class="p">(</span><span class="s">"text"</span><span class="p">)</span>
<span class="gt">Traceback (most recent call last):</span>
<span class="o">...</span>
<span class="gr">TypeError</span>: <span class="n">Method setParams forces keyword arguments.</span>
</pre></div>
</div>
<dl class="method">
<dt id="pyspark.ml.feature.NGram.copy">
<code class="descname">copy</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.NGram.copy" title="Permalink to this definition">¶</a></dt>
<dd><p>Creates a copy of this instance with the same uid and some
extra params. The default implementation creates a
shallow copy using <code class="xref py py-func docutils literal"><span class="pre">copy.copy()</span></code>, and then copies the
embedded and extra parameters over and returns the copy.
Subclasses should override this method if the default approach
is not sufficient.
:param extra: Extra parameters to copy to the new instance
:return: Copy of this instance</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.NGram.explainParam">
<code class="descname">explainParam</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.NGram.explainParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Explains a single param and returns its name, doc, and optional
default value and user-supplied value in a string.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.NGram.explainParams">
<code class="descname">explainParams</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.NGram.explainParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns the documentation of all params with their optionally
default values and user-supplied values.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.NGram.extractParamMap">
<code class="descname">extractParamMap</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.NGram.extractParamMap" title="Permalink to this definition">¶</a></dt>
<dd><p>Extracts the embedded default param values and user-supplied
values, and then merges them with extra values from input into
a flat param map, where the latter value is used if there exist
conflicts, i.e., with ordering: default param values <
user-supplied values < extra.
:param extra: extra param values
:return: merged param map</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.NGram.getInputCol">
<code class="descname">getInputCol</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.NGram.getInputCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of inputCol or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.NGram.getN">
<code class="descname">getN</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/feature.html#NGram.getN"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.feature.NGram.getN" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of n or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.NGram.getOrDefault">
<code class="descname">getOrDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.NGram.getOrDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of a param in the user-supplied param map or its
default value. Raises an error if neither is set.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.NGram.getOutputCol">
<code class="descname">getOutputCol</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.NGram.getOutputCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of outputCol or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.NGram.getParam">
<code class="descname">getParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.NGram.getParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets a param by its name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.NGram.hasDefault">
<code class="descname">hasDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.NGram.hasDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param has a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.NGram.hasParam">
<code class="descname">hasParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.NGram.hasParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Tests whether this instance contains a param with a given
(string) name.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.feature.NGram.inputCol">
<code class="descname">inputCol</code><em class="property"> = Param(parent='undefined', name='inputCol', doc='input column name')</em><a class="headerlink" href="#pyspark.ml.feature.NGram.inputCol" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.NGram.isDefined">
<code class="descname">isDefined</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.NGram.isDefined" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user or has
a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.NGram.isSet">
<code class="descname">isSet</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.NGram.isSet" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.feature.NGram.n">
<code class="descname">n</code><em class="property"> = Param(parent='undefined', name='n', doc='number of elements per n-gram (>=1)')</em><a class="headerlink" href="#pyspark.ml.feature.NGram.n" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.feature.NGram.outputCol">
<code class="descname">outputCol</code><em class="property"> = Param(parent='undefined', name='outputCol', doc='output column name')</em><a class="headerlink" href="#pyspark.ml.feature.NGram.outputCol" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.feature.NGram.params">
<code class="descname">params</code><a class="headerlink" href="#pyspark.ml.feature.NGram.params" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns all params ordered by name. The default implementation
uses <code class="xref py py-func docutils literal"><span class="pre">dir()</span></code> to get all attributes of type
<code class="xref py py-class docutils literal"><span class="pre">Param</span></code>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.NGram.setInputCol">
<code class="descname">setInputCol</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.NGram.setInputCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.feature.NGram.inputCol" title="pyspark.ml.feature.NGram.inputCol"><code class="xref py py-attr docutils literal"><span class="pre">inputCol</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.NGram.setN">
<code class="descname">setN</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/feature.html#NGram.setN"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.feature.NGram.setN" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.feature.NGram.n" title="pyspark.ml.feature.NGram.n"><code class="xref py py-attr docutils literal"><span class="pre">n</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.NGram.setOutputCol">
<code class="descname">setOutputCol</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.NGram.setOutputCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.feature.NGram.outputCol" title="pyspark.ml.feature.NGram.outputCol"><code class="xref py py-attr docutils literal"><span class="pre">outputCol</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.NGram.setParams">
<code class="descname">setParams</code><span class="sig-paren">(</span><em>self</em>, <em>n=2</em>, <em>inputCol=None</em>, <em>outputCol=None</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/feature.html#NGram.setParams"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.feature.NGram.setParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets params for this NGram.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.NGram.transform">
<code class="descname">transform</code><span class="sig-paren">(</span><em>dataset</em>, <em>params=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.NGram.transform" title="Permalink to this definition">¶</a></dt>
<dd><p>Transforms the input dataset with optional parameters.</p>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple">
<li><strong>dataset</strong> – input dataset, which is an instance of
<a class="reference internal" href="pyspark.sql.html#pyspark.sql.DataFrame" title="pyspark.sql.DataFrame"><code class="xref py py-class docutils literal"><span class="pre">pyspark.sql.DataFrame</span></code></a></li>
<li><strong>params</strong> – an optional param map that overrides embedded
params.</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body"><p class="first last">transformed dataset</p>
</td>
</tr>
</tbody>
</table>
</dd></dl>
</dd></dl>
<dl class="class">
<dt id="pyspark.ml.feature.Normalizer">
<em class="property">class </em><code class="descclassname">pyspark.ml.feature.</code><code class="descname">Normalizer</code><span class="sig-paren">(</span><em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/feature.html#Normalizer"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.feature.Normalizer" title="Permalink to this definition">¶</a></dt>
<dd><blockquote>
<div>Normalize a vector to have unit norm using the given p-norm.</div></blockquote>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="kn">from</span> <span class="nn">pyspark.mllib.linalg</span> <span class="kn">import</span> <span class="n">Vectors</span>
<span class="gp">>>> </span><span class="n">svec</span> <span class="o">=</span> <span class="n">Vectors</span><span class="o">.</span><span class="n">sparse</span><span class="p">(</span><span class="mi">4</span><span class="p">,</span> <span class="p">{</span><span class="mi">1</span><span class="p">:</span> <span class="mf">4.0</span><span class="p">,</span> <span class="mi">3</span><span class="p">:</span> <span class="mf">3.0</span><span class="p">})</span>
<span class="gp">>>> </span><span class="n">df</span> <span class="o">=</span> <span class="n">sqlContext</span><span class="o">.</span><span class="n">createDataFrame</span><span class="p">([(</span><span class="n">Vectors</span><span class="o">.</span><span class="n">dense</span><span class="p">([</span><span class="mf">3.0</span><span class="p">,</span> <span class="o">-</span><span class="mf">4.0</span><span class="p">]),</span> <span class="n">svec</span><span class="p">)],</span> <span class="p">[</span><span class="s">"dense"</span><span class="p">,</span> <span class="s">"sparse"</span><span class="p">])</span>
<span class="gp">>>> </span><span class="n">normalizer</span> <span class="o">=</span> <span class="n">Normalizer</span><span class="p">(</span><span class="n">p</span><span class="o">=</span><span class="mf">2.0</span><span class="p">,</span> <span class="n">inputCol</span><span class="o">=</span><span class="s">"dense"</span><span class="p">,</span> <span class="n">outputCol</span><span class="o">=</span><span class="s">"features"</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">normalizer</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">df</span><span class="p">)</span><span class="o">.</span><span class="n">head</span><span class="p">()</span><span class="o">.</span><span class="n">features</span>
<span class="go">DenseVector([0.6, -0.8])</span>
<span class="gp">>>> </span><span class="n">normalizer</span><span class="o">.</span><span class="n">setParams</span><span class="p">(</span><span class="n">inputCol</span><span class="o">=</span><span class="s">"sparse"</span><span class="p">,</span> <span class="n">outputCol</span><span class="o">=</span><span class="s">"freqs"</span><span class="p">)</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">df</span><span class="p">)</span><span class="o">.</span><span class="n">head</span><span class="p">()</span><span class="o">.</span><span class="n">freqs</span>
<span class="go">SparseVector(4, {1: 0.8, 3: 0.6})</span>
<span class="gp">>>> </span><span class="n">params</span> <span class="o">=</span> <span class="p">{</span><span class="n">normalizer</span><span class="o">.</span><span class="n">p</span><span class="p">:</span> <span class="mf">1.0</span><span class="p">,</span> <span class="n">normalizer</span><span class="o">.</span><span class="n">inputCol</span><span class="p">:</span> <span class="s">"dense"</span><span class="p">,</span> <span class="n">normalizer</span><span class="o">.</span><span class="n">outputCol</span><span class="p">:</span> <span class="s">"vector"</span><span class="p">}</span>
<span class="gp">>>> </span><span class="n">normalizer</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">df</span><span class="p">,</span> <span class="n">params</span><span class="p">)</span><span class="o">.</span><span class="n">head</span><span class="p">()</span><span class="o">.</span><span class="n">vector</span>
<span class="go">DenseVector([0.4286, -0.5714])</span>
</pre></div>
</div>
<dl class="method">
<dt id="pyspark.ml.feature.Normalizer.copy">
<code class="descname">copy</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Normalizer.copy" title="Permalink to this definition">¶</a></dt>
<dd><p>Creates a copy of this instance with the same uid and some
extra params. The default implementation creates a
shallow copy using <code class="xref py py-func docutils literal"><span class="pre">copy.copy()</span></code>, and then copies the
embedded and extra parameters over and returns the copy.
Subclasses should override this method if the default approach
is not sufficient.
:param extra: Extra parameters to copy to the new instance
:return: Copy of this instance</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Normalizer.explainParam">
<code class="descname">explainParam</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Normalizer.explainParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Explains a single param and returns its name, doc, and optional
default value and user-supplied value in a string.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Normalizer.explainParams">
<code class="descname">explainParams</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Normalizer.explainParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns the documentation of all params with their optionally
default values and user-supplied values.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Normalizer.extractParamMap">
<code class="descname">extractParamMap</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Normalizer.extractParamMap" title="Permalink to this definition">¶</a></dt>
<dd><p>Extracts the embedded default param values and user-supplied
values, and then merges them with extra values from input into
a flat param map, where the latter value is used if there exist
conflicts, i.e., with ordering: default param values <
user-supplied values < extra.
:param extra: extra param values
:return: merged param map</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Normalizer.getInputCol">
<code class="descname">getInputCol</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Normalizer.getInputCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of inputCol or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Normalizer.getOrDefault">
<code class="descname">getOrDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Normalizer.getOrDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of a param in the user-supplied param map or its
default value. Raises an error if neither is set.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Normalizer.getOutputCol">
<code class="descname">getOutputCol</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Normalizer.getOutputCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of outputCol or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Normalizer.getP">
<code class="descname">getP</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/feature.html#Normalizer.getP"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.feature.Normalizer.getP" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of p or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Normalizer.getParam">
<code class="descname">getParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Normalizer.getParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets a param by its name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Normalizer.hasDefault">
<code class="descname">hasDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Normalizer.hasDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param has a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Normalizer.hasParam">
<code class="descname">hasParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Normalizer.hasParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Tests whether this instance contains a param with a given
(string) name.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.feature.Normalizer.inputCol">
<code class="descname">inputCol</code><em class="property"> = Param(parent='undefined', name='inputCol', doc='input column name')</em><a class="headerlink" href="#pyspark.ml.feature.Normalizer.inputCol" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Normalizer.isDefined">
<code class="descname">isDefined</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Normalizer.isDefined" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user or has
a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Normalizer.isSet">
<code class="descname">isSet</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Normalizer.isSet" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.feature.Normalizer.outputCol">
<code class="descname">outputCol</code><em class="property"> = Param(parent='undefined', name='outputCol', doc='output column name')</em><a class="headerlink" href="#pyspark.ml.feature.Normalizer.outputCol" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.feature.Normalizer.p">
<code class="descname">p</code><em class="property"> = Param(parent='undefined', name='p', doc='the p norm value.')</em><a class="headerlink" href="#pyspark.ml.feature.Normalizer.p" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.feature.Normalizer.params">
<code class="descname">params</code><a class="headerlink" href="#pyspark.ml.feature.Normalizer.params" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns all params ordered by name. The default implementation
uses <code class="xref py py-func docutils literal"><span class="pre">dir()</span></code> to get all attributes of type
<code class="xref py py-class docutils literal"><span class="pre">Param</span></code>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Normalizer.setInputCol">
<code class="descname">setInputCol</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Normalizer.setInputCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.feature.Normalizer.inputCol" title="pyspark.ml.feature.Normalizer.inputCol"><code class="xref py py-attr docutils literal"><span class="pre">inputCol</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Normalizer.setOutputCol">
<code class="descname">setOutputCol</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Normalizer.setOutputCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.feature.Normalizer.outputCol" title="pyspark.ml.feature.Normalizer.outputCol"><code class="xref py py-attr docutils literal"><span class="pre">outputCol</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Normalizer.setP">
<code class="descname">setP</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/feature.html#Normalizer.setP"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.feature.Normalizer.setP" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.feature.Normalizer.p" title="pyspark.ml.feature.Normalizer.p"><code class="xref py py-attr docutils literal"><span class="pre">p</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Normalizer.setParams">
<code class="descname">setParams</code><span class="sig-paren">(</span><em>self</em>, <em>p=2.0</em>, <em>inputCol=None</em>, <em>outputCol=None</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/feature.html#Normalizer.setParams"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.feature.Normalizer.setParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets params for this Normalizer.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Normalizer.transform">
<code class="descname">transform</code><span class="sig-paren">(</span><em>dataset</em>, <em>params=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Normalizer.transform" title="Permalink to this definition">¶</a></dt>
<dd><p>Transforms the input dataset with optional parameters.</p>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple">
<li><strong>dataset</strong> – input dataset, which is an instance of
<a class="reference internal" href="pyspark.sql.html#pyspark.sql.DataFrame" title="pyspark.sql.DataFrame"><code class="xref py py-class docutils literal"><span class="pre">pyspark.sql.DataFrame</span></code></a></li>
<li><strong>params</strong> – an optional param map that overrides embedded
params.</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body"><p class="first last">transformed dataset</p>
</td>
</tr>
</tbody>
</table>
</dd></dl>
</dd></dl>
<dl class="class">
<dt id="pyspark.ml.feature.OneHotEncoder">
<em class="property">class </em><code class="descclassname">pyspark.ml.feature.</code><code class="descname">OneHotEncoder</code><span class="sig-paren">(</span><em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/feature.html#OneHotEncoder"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.feature.OneHotEncoder" title="Permalink to this definition">¶</a></dt>
<dd><p>A one-hot encoder that maps a column of category indices to a
column of binary vectors, with at most a single one-value per row
that indicates the input category index.
For example with 5 categories, an input value of 2.0 would map to
an output vector of <cite>[0.0, 0.0, 1.0, 0.0]</cite>.
The last category is not included by default (configurable via
<a class="reference internal" href="#pyspark.ml.feature.OneHotEncoder.dropLast" title="pyspark.ml.feature.OneHotEncoder.dropLast"><code class="xref py py-attr docutils literal"><span class="pre">dropLast</span></code></a>) because it makes the vector entries sum up to
one, and hence linearly dependent.
So an input value of 4.0 maps to <cite>[0.0, 0.0, 0.0, 0.0]</cite>.
Note that this is different from scikit-learn’s OneHotEncoder,
which keeps all categories.
The output vectors are sparse.</p>
<div class="admonition seealso">
<p class="first admonition-title">See also</p>
<p class="last"><a class="reference internal" href="#pyspark.ml.feature.StringIndexer" title="pyspark.ml.feature.StringIndexer"><code class="xref py py-class docutils literal"><span class="pre">StringIndexer</span></code></a> for converting categorical values into
category indices</p>
</div>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="n">stringIndexer</span> <span class="o">=</span> <span class="n">StringIndexer</span><span class="p">(</span><span class="n">inputCol</span><span class="o">=</span><span class="s">"label"</span><span class="p">,</span> <span class="n">outputCol</span><span class="o">=</span><span class="s">"indexed"</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">model</span> <span class="o">=</span> <span class="n">stringIndexer</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">stringIndDf</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">td</span> <span class="o">=</span> <span class="n">model</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">stringIndDf</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">encoder</span> <span class="o">=</span> <span class="n">OneHotEncoder</span><span class="p">(</span><span class="n">inputCol</span><span class="o">=</span><span class="s">"indexed"</span><span class="p">,</span> <span class="n">outputCol</span><span class="o">=</span><span class="s">"features"</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">encoder</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">td</span><span class="p">)</span><span class="o">.</span><span class="n">head</span><span class="p">()</span><span class="o">.</span><span class="n">features</span>
<span class="go">SparseVector(2, {0: 1.0})</span>
<span class="gp">>>> </span><span class="n">encoder</span><span class="o">.</span><span class="n">setParams</span><span class="p">(</span><span class="n">outputCol</span><span class="o">=</span><span class="s">"freqs"</span><span class="p">)</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">td</span><span class="p">)</span><span class="o">.</span><span class="n">head</span><span class="p">()</span><span class="o">.</span><span class="n">freqs</span>
<span class="go">SparseVector(2, {0: 1.0})</span>
<span class="gp">>>> </span><span class="n">params</span> <span class="o">=</span> <span class="p">{</span><span class="n">encoder</span><span class="o">.</span><span class="n">dropLast</span><span class="p">:</span> <span class="bp">False</span><span class="p">,</span> <span class="n">encoder</span><span class="o">.</span><span class="n">outputCol</span><span class="p">:</span> <span class="s">"test"</span><span class="p">}</span>
<span class="gp">>>> </span><span class="n">encoder</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">td</span><span class="p">,</span> <span class="n">params</span><span class="p">)</span><span class="o">.</span><span class="n">head</span><span class="p">()</span><span class="o">.</span><span class="n">test</span>
<span class="go">SparseVector(3, {0: 1.0})</span>
</pre></div>
</div>
<dl class="method">
<dt id="pyspark.ml.feature.OneHotEncoder.copy">
<code class="descname">copy</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.OneHotEncoder.copy" title="Permalink to this definition">¶</a></dt>
<dd><p>Creates a copy of this instance with the same uid and some
extra params. The default implementation creates a
shallow copy using <code class="xref py py-func docutils literal"><span class="pre">copy.copy()</span></code>, and then copies the
embedded and extra parameters over and returns the copy.
Subclasses should override this method if the default approach
is not sufficient.
:param extra: Extra parameters to copy to the new instance
:return: Copy of this instance</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.feature.OneHotEncoder.dropLast">
<code class="descname">dropLast</code><em class="property"> = Param(parent='undefined', name='dropLast', doc='whether to drop the last category')</em><a class="headerlink" href="#pyspark.ml.feature.OneHotEncoder.dropLast" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.OneHotEncoder.explainParam">
<code class="descname">explainParam</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.OneHotEncoder.explainParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Explains a single param and returns its name, doc, and optional
default value and user-supplied value in a string.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.OneHotEncoder.explainParams">
<code class="descname">explainParams</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.OneHotEncoder.explainParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns the documentation of all params with their optionally
default values and user-supplied values.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.OneHotEncoder.extractParamMap">
<code class="descname">extractParamMap</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.OneHotEncoder.extractParamMap" title="Permalink to this definition">¶</a></dt>
<dd><p>Extracts the embedded default param values and user-supplied
values, and then merges them with extra values from input into
a flat param map, where the latter value is used if there exist
conflicts, i.e., with ordering: default param values <
user-supplied values < extra.
:param extra: extra param values
:return: merged param map</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.OneHotEncoder.getDropLast">
<code class="descname">getDropLast</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/feature.html#OneHotEncoder.getDropLast"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.feature.OneHotEncoder.getDropLast" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of dropLast or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.OneHotEncoder.getInputCol">
<code class="descname">getInputCol</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.OneHotEncoder.getInputCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of inputCol or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.OneHotEncoder.getOrDefault">
<code class="descname">getOrDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.OneHotEncoder.getOrDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of a param in the user-supplied param map or its
default value. Raises an error if neither is set.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.OneHotEncoder.getOutputCol">
<code class="descname">getOutputCol</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.OneHotEncoder.getOutputCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of outputCol or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.OneHotEncoder.getParam">
<code class="descname">getParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.OneHotEncoder.getParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets a param by its name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.OneHotEncoder.hasDefault">
<code class="descname">hasDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.OneHotEncoder.hasDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param has a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.OneHotEncoder.hasParam">
<code class="descname">hasParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.OneHotEncoder.hasParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Tests whether this instance contains a param with a given
(string) name.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.feature.OneHotEncoder.inputCol">
<code class="descname">inputCol</code><em class="property"> = Param(parent='undefined', name='inputCol', doc='input column name')</em><a class="headerlink" href="#pyspark.ml.feature.OneHotEncoder.inputCol" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.OneHotEncoder.isDefined">
<code class="descname">isDefined</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.OneHotEncoder.isDefined" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user or has
a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.OneHotEncoder.isSet">
<code class="descname">isSet</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.OneHotEncoder.isSet" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.feature.OneHotEncoder.outputCol">
<code class="descname">outputCol</code><em class="property"> = Param(parent='undefined', name='outputCol', doc='output column name')</em><a class="headerlink" href="#pyspark.ml.feature.OneHotEncoder.outputCol" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.feature.OneHotEncoder.params">
<code class="descname">params</code><a class="headerlink" href="#pyspark.ml.feature.OneHotEncoder.params" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns all params ordered by name. The default implementation
uses <code class="xref py py-func docutils literal"><span class="pre">dir()</span></code> to get all attributes of type
<code class="xref py py-class docutils literal"><span class="pre">Param</span></code>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.OneHotEncoder.setDropLast">
<code class="descname">setDropLast</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/feature.html#OneHotEncoder.setDropLast"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.feature.OneHotEncoder.setDropLast" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.feature.OneHotEncoder.dropLast" title="pyspark.ml.feature.OneHotEncoder.dropLast"><code class="xref py py-attr docutils literal"><span class="pre">dropLast</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.OneHotEncoder.setInputCol">
<code class="descname">setInputCol</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.OneHotEncoder.setInputCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.feature.OneHotEncoder.inputCol" title="pyspark.ml.feature.OneHotEncoder.inputCol"><code class="xref py py-attr docutils literal"><span class="pre">inputCol</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.OneHotEncoder.setOutputCol">
<code class="descname">setOutputCol</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.OneHotEncoder.setOutputCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.feature.OneHotEncoder.outputCol" title="pyspark.ml.feature.OneHotEncoder.outputCol"><code class="xref py py-attr docutils literal"><span class="pre">outputCol</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.OneHotEncoder.setParams">
<code class="descname">setParams</code><span class="sig-paren">(</span><em>self</em>, <em>dropLast=True</em>, <em>inputCol=None</em>, <em>outputCol=None</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/feature.html#OneHotEncoder.setParams"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.feature.OneHotEncoder.setParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets params for this OneHotEncoder.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.OneHotEncoder.transform">
<code class="descname">transform</code><span class="sig-paren">(</span><em>dataset</em>, <em>params=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.OneHotEncoder.transform" title="Permalink to this definition">¶</a></dt>
<dd><p>Transforms the input dataset with optional parameters.</p>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple">
<li><strong>dataset</strong> – input dataset, which is an instance of
<a class="reference internal" href="pyspark.sql.html#pyspark.sql.DataFrame" title="pyspark.sql.DataFrame"><code class="xref py py-class docutils literal"><span class="pre">pyspark.sql.DataFrame</span></code></a></li>
<li><strong>params</strong> – an optional param map that overrides embedded
params.</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body"><p class="first last">transformed dataset</p>
</td>
</tr>
</tbody>
</table>
</dd></dl>
</dd></dl>
<dl class="class">
<dt id="pyspark.ml.feature.PolynomialExpansion">
<em class="property">class </em><code class="descclassname">pyspark.ml.feature.</code><code class="descname">PolynomialExpansion</code><span class="sig-paren">(</span><em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/feature.html#PolynomialExpansion"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.feature.PolynomialExpansion" title="Permalink to this definition">¶</a></dt>
<dd><p>Perform feature expansion in a polynomial space. As said in wikipedia of Polynomial Expansion,
which is available at <cite>http://en.wikipedia.org/wiki/Polynomial_expansion</cite>, “In mathematics, an
expansion of a product of sums expresses it as a sum of products by using the fact that
multiplication distributes over addition”. Take a 2-variable feature vector as an example:
<cite>(x, y)</cite>, if we want to expand it with degree 2, then we get <cite>(x, x * x, y, x * y, y * y)</cite>.</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="kn">from</span> <span class="nn">pyspark.mllib.linalg</span> <span class="kn">import</span> <span class="n">Vectors</span>
<span class="gp">>>> </span><span class="n">df</span> <span class="o">=</span> <span class="n">sqlContext</span><span class="o">.</span><span class="n">createDataFrame</span><span class="p">([(</span><span class="n">Vectors</span><span class="o">.</span><span class="n">dense</span><span class="p">([</span><span class="mf">0.5</span><span class="p">,</span> <span class="mf">2.0</span><span class="p">]),)],</span> <span class="p">[</span><span class="s">"dense"</span><span class="p">])</span>
<span class="gp">>>> </span><span class="n">px</span> <span class="o">=</span> <span class="n">PolynomialExpansion</span><span class="p">(</span><span class="n">degree</span><span class="o">=</span><span class="mi">2</span><span class="p">,</span> <span class="n">inputCol</span><span class="o">=</span><span class="s">"dense"</span><span class="p">,</span> <span class="n">outputCol</span><span class="o">=</span><span class="s">"expanded"</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">px</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">df</span><span class="p">)</span><span class="o">.</span><span class="n">head</span><span class="p">()</span><span class="o">.</span><span class="n">expanded</span>
<span class="go">DenseVector([0.5, 0.25, 2.0, 1.0, 4.0])</span>
<span class="gp">>>> </span><span class="n">px</span><span class="o">.</span><span class="n">setParams</span><span class="p">(</span><span class="n">outputCol</span><span class="o">=</span><span class="s">"test"</span><span class="p">)</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">df</span><span class="p">)</span><span class="o">.</span><span class="n">head</span><span class="p">()</span><span class="o">.</span><span class="n">test</span>
<span class="go">DenseVector([0.5, 0.25, 2.0, 1.0, 4.0])</span>
</pre></div>
</div>
<dl class="method">
<dt id="pyspark.ml.feature.PolynomialExpansion.copy">
<code class="descname">copy</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.PolynomialExpansion.copy" title="Permalink to this definition">¶</a></dt>
<dd><p>Creates a copy of this instance with the same uid and some
extra params. The default implementation creates a
shallow copy using <code class="xref py py-func docutils literal"><span class="pre">copy.copy()</span></code>, and then copies the
embedded and extra parameters over and returns the copy.
Subclasses should override this method if the default approach
is not sufficient.
:param extra: Extra parameters to copy to the new instance
:return: Copy of this instance</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.feature.PolynomialExpansion.degree">
<code class="descname">degree</code><em class="property"> = Param(parent='undefined', name='degree', doc='the polynomial degree to expand (>= 1)')</em><a class="headerlink" href="#pyspark.ml.feature.PolynomialExpansion.degree" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.PolynomialExpansion.explainParam">
<code class="descname">explainParam</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.PolynomialExpansion.explainParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Explains a single param and returns its name, doc, and optional
default value and user-supplied value in a string.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.PolynomialExpansion.explainParams">
<code class="descname">explainParams</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.PolynomialExpansion.explainParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns the documentation of all params with their optionally
default values and user-supplied values.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.PolynomialExpansion.extractParamMap">
<code class="descname">extractParamMap</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.PolynomialExpansion.extractParamMap" title="Permalink to this definition">¶</a></dt>
<dd><p>Extracts the embedded default param values and user-supplied
values, and then merges them with extra values from input into
a flat param map, where the latter value is used if there exist
conflicts, i.e., with ordering: default param values <
user-supplied values < extra.
:param extra: extra param values
:return: merged param map</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.PolynomialExpansion.getDegree">
<code class="descname">getDegree</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/feature.html#PolynomialExpansion.getDegree"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.feature.PolynomialExpansion.getDegree" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of degree or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.PolynomialExpansion.getInputCol">
<code class="descname">getInputCol</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.PolynomialExpansion.getInputCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of inputCol or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.PolynomialExpansion.getOrDefault">
<code class="descname">getOrDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.PolynomialExpansion.getOrDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of a param in the user-supplied param map or its
default value. Raises an error if neither is set.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.PolynomialExpansion.getOutputCol">
<code class="descname">getOutputCol</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.PolynomialExpansion.getOutputCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of outputCol or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.PolynomialExpansion.getParam">
<code class="descname">getParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.PolynomialExpansion.getParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets a param by its name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.PolynomialExpansion.hasDefault">
<code class="descname">hasDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.PolynomialExpansion.hasDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param has a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.PolynomialExpansion.hasParam">
<code class="descname">hasParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.PolynomialExpansion.hasParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Tests whether this instance contains a param with a given
(string) name.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.feature.PolynomialExpansion.inputCol">
<code class="descname">inputCol</code><em class="property"> = Param(parent='undefined', name='inputCol', doc='input column name')</em><a class="headerlink" href="#pyspark.ml.feature.PolynomialExpansion.inputCol" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.PolynomialExpansion.isDefined">
<code class="descname">isDefined</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.PolynomialExpansion.isDefined" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user or has
a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.PolynomialExpansion.isSet">
<code class="descname">isSet</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.PolynomialExpansion.isSet" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.feature.PolynomialExpansion.outputCol">
<code class="descname">outputCol</code><em class="property"> = Param(parent='undefined', name='outputCol', doc='output column name')</em><a class="headerlink" href="#pyspark.ml.feature.PolynomialExpansion.outputCol" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.feature.PolynomialExpansion.params">
<code class="descname">params</code><a class="headerlink" href="#pyspark.ml.feature.PolynomialExpansion.params" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns all params ordered by name. The default implementation
uses <code class="xref py py-func docutils literal"><span class="pre">dir()</span></code> to get all attributes of type
<code class="xref py py-class docutils literal"><span class="pre">Param</span></code>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.PolynomialExpansion.setDegree">
<code class="descname">setDegree</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/feature.html#PolynomialExpansion.setDegree"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.feature.PolynomialExpansion.setDegree" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.feature.PolynomialExpansion.degree" title="pyspark.ml.feature.PolynomialExpansion.degree"><code class="xref py py-attr docutils literal"><span class="pre">degree</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.PolynomialExpansion.setInputCol">
<code class="descname">setInputCol</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.PolynomialExpansion.setInputCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.feature.PolynomialExpansion.inputCol" title="pyspark.ml.feature.PolynomialExpansion.inputCol"><code class="xref py py-attr docutils literal"><span class="pre">inputCol</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.PolynomialExpansion.setOutputCol">
<code class="descname">setOutputCol</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.PolynomialExpansion.setOutputCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.feature.PolynomialExpansion.outputCol" title="pyspark.ml.feature.PolynomialExpansion.outputCol"><code class="xref py py-attr docutils literal"><span class="pre">outputCol</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.PolynomialExpansion.setParams">
<code class="descname">setParams</code><span class="sig-paren">(</span><em>self</em>, <em>degree=2</em>, <em>inputCol=None</em>, <em>outputCol=None</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/feature.html#PolynomialExpansion.setParams"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.feature.PolynomialExpansion.setParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets params for this PolynomialExpansion.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.PolynomialExpansion.transform">
<code class="descname">transform</code><span class="sig-paren">(</span><em>dataset</em>, <em>params=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.PolynomialExpansion.transform" title="Permalink to this definition">¶</a></dt>
<dd><p>Transforms the input dataset with optional parameters.</p>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple">
<li><strong>dataset</strong> – input dataset, which is an instance of
<a class="reference internal" href="pyspark.sql.html#pyspark.sql.DataFrame" title="pyspark.sql.DataFrame"><code class="xref py py-class docutils literal"><span class="pre">pyspark.sql.DataFrame</span></code></a></li>
<li><strong>params</strong> – an optional param map that overrides embedded
params.</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body"><p class="first last">transformed dataset</p>
</td>
</tr>
</tbody>
</table>
</dd></dl>
</dd></dl>
<dl class="class">
<dt id="pyspark.ml.feature.RegexTokenizer">
<em class="property">class </em><code class="descclassname">pyspark.ml.feature.</code><code class="descname">RegexTokenizer</code><span class="sig-paren">(</span><em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/feature.html#RegexTokenizer"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.feature.RegexTokenizer" title="Permalink to this definition">¶</a></dt>
<dd><p>A regex based tokenizer that extracts tokens either by using the
provided regex pattern (in Java dialect) to split the text
(default) or repeatedly matching the regex (if gaps is false).
Optional parameters also allow filtering tokens using a minimal
length.
It returns an array of strings that can be empty.</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="n">df</span> <span class="o">=</span> <span class="n">sqlContext</span><span class="o">.</span><span class="n">createDataFrame</span><span class="p">([(</span><span class="s">"a b c"</span><span class="p">,)],</span> <span class="p">[</span><span class="s">"text"</span><span class="p">])</span>
<span class="gp">>>> </span><span class="n">reTokenizer</span> <span class="o">=</span> <span class="n">RegexTokenizer</span><span class="p">(</span><span class="n">inputCol</span><span class="o">=</span><span class="s">"text"</span><span class="p">,</span> <span class="n">outputCol</span><span class="o">=</span><span class="s">"words"</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">reTokenizer</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">df</span><span class="p">)</span><span class="o">.</span><span class="n">head</span><span class="p">()</span>
<span class="go">Row(text=u'a b c', words=[u'a', u'b', u'c'])</span>
<span class="gp">>>> </span><span class="c"># Change a parameter.</span>
<span class="gp">>>> </span><span class="n">reTokenizer</span><span class="o">.</span><span class="n">setParams</span><span class="p">(</span><span class="n">outputCol</span><span class="o">=</span><span class="s">"tokens"</span><span class="p">)</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">df</span><span class="p">)</span><span class="o">.</span><span class="n">head</span><span class="p">()</span>
<span class="go">Row(text=u'a b c', tokens=[u'a', u'b', u'c'])</span>
<span class="gp">>>> </span><span class="c"># Temporarily modify a parameter.</span>
<span class="gp">>>> </span><span class="n">reTokenizer</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">df</span><span class="p">,</span> <span class="p">{</span><span class="n">reTokenizer</span><span class="o">.</span><span class="n">outputCol</span><span class="p">:</span> <span class="s">"words"</span><span class="p">})</span><span class="o">.</span><span class="n">head</span><span class="p">()</span>
<span class="go">Row(text=u'a b c', words=[u'a', u'b', u'c'])</span>
<span class="gp">>>> </span><span class="n">reTokenizer</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">df</span><span class="p">)</span><span class="o">.</span><span class="n">head</span><span class="p">()</span>
<span class="go">Row(text=u'a b c', tokens=[u'a', u'b', u'c'])</span>
<span class="gp">>>> </span><span class="c"># Must use keyword arguments to specify params.</span>
<span class="gp">>>> </span><span class="n">reTokenizer</span><span class="o">.</span><span class="n">setParams</span><span class="p">(</span><span class="s">"text"</span><span class="p">)</span>
<span class="gt">Traceback (most recent call last):</span>
<span class="o">...</span>
<span class="gr">TypeError</span>: <span class="n">Method setParams forces keyword arguments.</span>
</pre></div>
</div>
<dl class="method">
<dt id="pyspark.ml.feature.RegexTokenizer.copy">
<code class="descname">copy</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.RegexTokenizer.copy" title="Permalink to this definition">¶</a></dt>
<dd><p>Creates a copy of this instance with the same uid and some
extra params. The default implementation creates a
shallow copy using <code class="xref py py-func docutils literal"><span class="pre">copy.copy()</span></code>, and then copies the
embedded and extra parameters over and returns the copy.
Subclasses should override this method if the default approach
is not sufficient.
:param extra: Extra parameters to copy to the new instance
:return: Copy of this instance</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.RegexTokenizer.explainParam">
<code class="descname">explainParam</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.RegexTokenizer.explainParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Explains a single param and returns its name, doc, and optional
default value and user-supplied value in a string.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.RegexTokenizer.explainParams">
<code class="descname">explainParams</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.RegexTokenizer.explainParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns the documentation of all params with their optionally
default values and user-supplied values.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.RegexTokenizer.extractParamMap">
<code class="descname">extractParamMap</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.RegexTokenizer.extractParamMap" title="Permalink to this definition">¶</a></dt>
<dd><p>Extracts the embedded default param values and user-supplied
values, and then merges them with extra values from input into
a flat param map, where the latter value is used if there exist
conflicts, i.e., with ordering: default param values <
user-supplied values < extra.
:param extra: extra param values
:return: merged param map</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.feature.RegexTokenizer.gaps">
<code class="descname">gaps</code><em class="property"> = Param(parent='undefined', name='gaps', doc='whether regex splits on gaps (True) or matches tokens')</em><a class="headerlink" href="#pyspark.ml.feature.RegexTokenizer.gaps" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.RegexTokenizer.getGaps">
<code class="descname">getGaps</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/feature.html#RegexTokenizer.getGaps"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.feature.RegexTokenizer.getGaps" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of gaps or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.RegexTokenizer.getInputCol">
<code class="descname">getInputCol</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.RegexTokenizer.getInputCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of inputCol or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.RegexTokenizer.getMinTokenLength">
<code class="descname">getMinTokenLength</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/feature.html#RegexTokenizer.getMinTokenLength"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.feature.RegexTokenizer.getMinTokenLength" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of minTokenLength or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.RegexTokenizer.getOrDefault">
<code class="descname">getOrDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.RegexTokenizer.getOrDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of a param in the user-supplied param map or its
default value. Raises an error if neither is set.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.RegexTokenizer.getOutputCol">
<code class="descname">getOutputCol</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.RegexTokenizer.getOutputCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of outputCol or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.RegexTokenizer.getParam">
<code class="descname">getParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.RegexTokenizer.getParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets a param by its name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.RegexTokenizer.getPattern">
<code class="descname">getPattern</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/feature.html#RegexTokenizer.getPattern"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.feature.RegexTokenizer.getPattern" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of pattern or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.RegexTokenizer.hasDefault">
<code class="descname">hasDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.RegexTokenizer.hasDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param has a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.RegexTokenizer.hasParam">
<code class="descname">hasParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.RegexTokenizer.hasParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Tests whether this instance contains a param with a given
(string) name.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.feature.RegexTokenizer.inputCol">
<code class="descname">inputCol</code><em class="property"> = Param(parent='undefined', name='inputCol', doc='input column name')</em><a class="headerlink" href="#pyspark.ml.feature.RegexTokenizer.inputCol" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.RegexTokenizer.isDefined">
<code class="descname">isDefined</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.RegexTokenizer.isDefined" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user or has
a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.RegexTokenizer.isSet">
<code class="descname">isSet</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.RegexTokenizer.isSet" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.feature.RegexTokenizer.minTokenLength">
<code class="descname">minTokenLength</code><em class="property"> = Param(parent='undefined', name='minTokenLength', doc='minimum token length (>= 0)')</em><a class="headerlink" href="#pyspark.ml.feature.RegexTokenizer.minTokenLength" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.feature.RegexTokenizer.outputCol">
<code class="descname">outputCol</code><em class="property"> = Param(parent='undefined', name='outputCol', doc='output column name')</em><a class="headerlink" href="#pyspark.ml.feature.RegexTokenizer.outputCol" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.feature.RegexTokenizer.params">
<code class="descname">params</code><a class="headerlink" href="#pyspark.ml.feature.RegexTokenizer.params" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns all params ordered by name. The default implementation
uses <code class="xref py py-func docutils literal"><span class="pre">dir()</span></code> to get all attributes of type
<code class="xref py py-class docutils literal"><span class="pre">Param</span></code>.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.feature.RegexTokenizer.pattern">
<code class="descname">pattern</code><em class="property"> = Param(parent='undefined', name='pattern', doc='regex pattern (Java dialect) used for tokenizing')</em><a class="headerlink" href="#pyspark.ml.feature.RegexTokenizer.pattern" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.RegexTokenizer.setGaps">
<code class="descname">setGaps</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/feature.html#RegexTokenizer.setGaps"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.feature.RegexTokenizer.setGaps" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.feature.RegexTokenizer.gaps" title="pyspark.ml.feature.RegexTokenizer.gaps"><code class="xref py py-attr docutils literal"><span class="pre">gaps</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.RegexTokenizer.setInputCol">
<code class="descname">setInputCol</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.RegexTokenizer.setInputCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.feature.RegexTokenizer.inputCol" title="pyspark.ml.feature.RegexTokenizer.inputCol"><code class="xref py py-attr docutils literal"><span class="pre">inputCol</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.RegexTokenizer.setMinTokenLength">
<code class="descname">setMinTokenLength</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/feature.html#RegexTokenizer.setMinTokenLength"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.feature.RegexTokenizer.setMinTokenLength" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.feature.RegexTokenizer.minTokenLength" title="pyspark.ml.feature.RegexTokenizer.minTokenLength"><code class="xref py py-attr docutils literal"><span class="pre">minTokenLength</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.RegexTokenizer.setOutputCol">
<code class="descname">setOutputCol</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.RegexTokenizer.setOutputCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.feature.RegexTokenizer.outputCol" title="pyspark.ml.feature.RegexTokenizer.outputCol"><code class="xref py py-attr docutils literal"><span class="pre">outputCol</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.RegexTokenizer.setParams">
<code class="descname">setParams</code><span class="sig-paren">(</span><em>self</em>, <em>minTokenLength=1</em>, <em>gaps=True</em>, <em>pattern="s+"</em>, <em>inputCol=None</em>, <em>outputCol=None</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/feature.html#RegexTokenizer.setParams"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.feature.RegexTokenizer.setParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets params for this RegexTokenizer.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.RegexTokenizer.setPattern">
<code class="descname">setPattern</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/feature.html#RegexTokenizer.setPattern"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.feature.RegexTokenizer.setPattern" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.feature.RegexTokenizer.pattern" title="pyspark.ml.feature.RegexTokenizer.pattern"><code class="xref py py-attr docutils literal"><span class="pre">pattern</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.RegexTokenizer.transform">
<code class="descname">transform</code><span class="sig-paren">(</span><em>dataset</em>, <em>params=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.RegexTokenizer.transform" title="Permalink to this definition">¶</a></dt>
<dd><p>Transforms the input dataset with optional parameters.</p>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple">
<li><strong>dataset</strong> – input dataset, which is an instance of
<a class="reference internal" href="pyspark.sql.html#pyspark.sql.DataFrame" title="pyspark.sql.DataFrame"><code class="xref py py-class docutils literal"><span class="pre">pyspark.sql.DataFrame</span></code></a></li>
<li><strong>params</strong> – an optional param map that overrides embedded
params.</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body"><p class="first last">transformed dataset</p>
</td>
</tr>
</tbody>
</table>
</dd></dl>
</dd></dl>
<dl class="class">
<dt id="pyspark.ml.feature.StandardScaler">
<em class="property">class </em><code class="descclassname">pyspark.ml.feature.</code><code class="descname">StandardScaler</code><span class="sig-paren">(</span><em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/feature.html#StandardScaler"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.feature.StandardScaler" title="Permalink to this definition">¶</a></dt>
<dd><p>Standardizes features by removing the mean and scaling to unit variance using column summary
statistics on the samples in the training set.</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="kn">from</span> <span class="nn">pyspark.mllib.linalg</span> <span class="kn">import</span> <span class="n">Vectors</span>
<span class="gp">>>> </span><span class="n">df</span> <span class="o">=</span> <span class="n">sqlContext</span><span class="o">.</span><span class="n">createDataFrame</span><span class="p">([(</span><span class="n">Vectors</span><span class="o">.</span><span class="n">dense</span><span class="p">([</span><span class="mf">0.0</span><span class="p">]),),</span> <span class="p">(</span><span class="n">Vectors</span><span class="o">.</span><span class="n">dense</span><span class="p">([</span><span class="mf">2.0</span><span class="p">]),)],</span> <span class="p">[</span><span class="s">"a"</span><span class="p">])</span>
<span class="gp">>>> </span><span class="n">standardScaler</span> <span class="o">=</span> <span class="n">StandardScaler</span><span class="p">(</span><span class="n">inputCol</span><span class="o">=</span><span class="s">"a"</span><span class="p">,</span> <span class="n">outputCol</span><span class="o">=</span><span class="s">"scaled"</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">model</span> <span class="o">=</span> <span class="n">standardScaler</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">df</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">model</span><span class="o">.</span><span class="n">mean</span>
<span class="go">DenseVector([1.0])</span>
<span class="gp">>>> </span><span class="n">model</span><span class="o">.</span><span class="n">std</span>
<span class="go">DenseVector([1.4142])</span>
<span class="gp">>>> </span><span class="n">model</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">df</span><span class="p">)</span><span class="o">.</span><span class="n">collect</span><span class="p">()[</span><span class="mi">1</span><span class="p">]</span><span class="o">.</span><span class="n">scaled</span>
<span class="go">DenseVector([1.4142])</span>
</pre></div>
</div>
<dl class="method">
<dt id="pyspark.ml.feature.StandardScaler.copy">
<code class="descname">copy</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.StandardScaler.copy" title="Permalink to this definition">¶</a></dt>
<dd><p>Creates a copy of this instance with the same uid and some
extra params. The default implementation creates a
shallow copy using <code class="xref py py-func docutils literal"><span class="pre">copy.copy()</span></code>, and then copies the
embedded and extra parameters over and returns the copy.
Subclasses should override this method if the default approach
is not sufficient.
:param extra: Extra parameters to copy to the new instance
:return: Copy of this instance</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.StandardScaler.explainParam">
<code class="descname">explainParam</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.StandardScaler.explainParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Explains a single param and returns its name, doc, and optional
default value and user-supplied value in a string.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.StandardScaler.explainParams">
<code class="descname">explainParams</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.StandardScaler.explainParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns the documentation of all params with their optionally
default values and user-supplied values.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.StandardScaler.extractParamMap">
<code class="descname">extractParamMap</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.StandardScaler.extractParamMap" title="Permalink to this definition">¶</a></dt>
<dd><p>Extracts the embedded default param values and user-supplied
values, and then merges them with extra values from input into
a flat param map, where the latter value is used if there exist
conflicts, i.e., with ordering: default param values <
user-supplied values < extra.
:param extra: extra param values
:return: merged param map</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.StandardScaler.fit">
<code class="descname">fit</code><span class="sig-paren">(</span><em>dataset</em>, <em>params=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.StandardScaler.fit" title="Permalink to this definition">¶</a></dt>
<dd><p>Fits a model to the input dataset with optional parameters.</p>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple">
<li><strong>dataset</strong> – input dataset, which is an instance of
<a class="reference internal" href="pyspark.sql.html#pyspark.sql.DataFrame" title="pyspark.sql.DataFrame"><code class="xref py py-class docutils literal"><span class="pre">pyspark.sql.DataFrame</span></code></a></li>
<li><strong>params</strong> – an optional param map that overrides embedded
params. If a list/tuple of param maps is given,
this calls fit on each param map and returns a
list of models.</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body"><p class="first last">fitted model(s)</p>
</td>
</tr>
</tbody>
</table>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.StandardScaler.getInputCol">
<code class="descname">getInputCol</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.StandardScaler.getInputCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of inputCol or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.StandardScaler.getOrDefault">
<code class="descname">getOrDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.StandardScaler.getOrDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of a param in the user-supplied param map or its
default value. Raises an error if neither is set.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.StandardScaler.getOutputCol">
<code class="descname">getOutputCol</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.StandardScaler.getOutputCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of outputCol or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.StandardScaler.getParam">
<code class="descname">getParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.StandardScaler.getParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets a param by its name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.StandardScaler.getWithMean">
<code class="descname">getWithMean</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/feature.html#StandardScaler.getWithMean"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.feature.StandardScaler.getWithMean" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of withMean or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.StandardScaler.getWithStd">
<code class="descname">getWithStd</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/feature.html#StandardScaler.getWithStd"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.feature.StandardScaler.getWithStd" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of withStd or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.StandardScaler.hasDefault">
<code class="descname">hasDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.StandardScaler.hasDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param has a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.StandardScaler.hasParam">
<code class="descname">hasParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.StandardScaler.hasParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Tests whether this instance contains a param with a given
(string) name.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.feature.StandardScaler.inputCol">
<code class="descname">inputCol</code><em class="property"> = Param(parent='undefined', name='inputCol', doc='input column name')</em><a class="headerlink" href="#pyspark.ml.feature.StandardScaler.inputCol" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.StandardScaler.isDefined">
<code class="descname">isDefined</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.StandardScaler.isDefined" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user or has
a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.StandardScaler.isSet">
<code class="descname">isSet</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.StandardScaler.isSet" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.feature.StandardScaler.outputCol">
<code class="descname">outputCol</code><em class="property"> = Param(parent='undefined', name='outputCol', doc='output column name')</em><a class="headerlink" href="#pyspark.ml.feature.StandardScaler.outputCol" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.feature.StandardScaler.params">
<code class="descname">params</code><a class="headerlink" href="#pyspark.ml.feature.StandardScaler.params" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns all params ordered by name. The default implementation
uses <code class="xref py py-func docutils literal"><span class="pre">dir()</span></code> to get all attributes of type
<code class="xref py py-class docutils literal"><span class="pre">Param</span></code>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.StandardScaler.setInputCol">
<code class="descname">setInputCol</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.StandardScaler.setInputCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.feature.StandardScaler.inputCol" title="pyspark.ml.feature.StandardScaler.inputCol"><code class="xref py py-attr docutils literal"><span class="pre">inputCol</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.StandardScaler.setOutputCol">
<code class="descname">setOutputCol</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.StandardScaler.setOutputCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.feature.StandardScaler.outputCol" title="pyspark.ml.feature.StandardScaler.outputCol"><code class="xref py py-attr docutils literal"><span class="pre">outputCol</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.StandardScaler.setParams">
<code class="descname">setParams</code><span class="sig-paren">(</span><em>self</em>, <em>withMean=False</em>, <em>withStd=True</em>, <em>inputCol=None</em>, <em>outputCol=None</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/feature.html#StandardScaler.setParams"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.feature.StandardScaler.setParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets params for this StandardScaler.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.StandardScaler.setWithMean">
<code class="descname">setWithMean</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/feature.html#StandardScaler.setWithMean"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.feature.StandardScaler.setWithMean" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.feature.StandardScaler.withMean" title="pyspark.ml.feature.StandardScaler.withMean"><code class="xref py py-attr docutils literal"><span class="pre">withMean</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.StandardScaler.setWithStd">
<code class="descname">setWithStd</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/feature.html#StandardScaler.setWithStd"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.feature.StandardScaler.setWithStd" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.feature.StandardScaler.withStd" title="pyspark.ml.feature.StandardScaler.withStd"><code class="xref py py-attr docutils literal"><span class="pre">withStd</span></code></a>.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.feature.StandardScaler.withMean">
<code class="descname">withMean</code><em class="property"> = Param(parent='undefined', name='withMean', doc='Center data with mean')</em><a class="headerlink" href="#pyspark.ml.feature.StandardScaler.withMean" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.feature.StandardScaler.withStd">
<code class="descname">withStd</code><em class="property"> = Param(parent='undefined', name='withStd', doc='Scale to unit standard deviation')</em><a class="headerlink" href="#pyspark.ml.feature.StandardScaler.withStd" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
</dd></dl>
<dl class="class">
<dt id="pyspark.ml.feature.StandardScalerModel">
<em class="property">class </em><code class="descclassname">pyspark.ml.feature.</code><code class="descname">StandardScalerModel</code><span class="sig-paren">(</span><em>java_model</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/feature.html#StandardScalerModel"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.feature.StandardScalerModel" title="Permalink to this definition">¶</a></dt>
<dd><p>Model fitted by StandardScaler.</p>
<dl class="method">
<dt id="pyspark.ml.feature.StandardScalerModel.copy">
<code class="descname">copy</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.StandardScalerModel.copy" title="Permalink to this definition">¶</a></dt>
<dd><p>Creates a copy of this instance with the same uid and some
extra params. This implementation first calls Params.copy and
then make a copy of the companion Java model with extra params.
So both the Python wrapper and the Java model get copied.
:param extra: Extra parameters to copy to the new instance
:return: Copy of this instance</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.StandardScalerModel.explainParam">
<code class="descname">explainParam</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.StandardScalerModel.explainParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Explains a single param and returns its name, doc, and optional
default value and user-supplied value in a string.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.StandardScalerModel.explainParams">
<code class="descname">explainParams</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.StandardScalerModel.explainParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns the documentation of all params with their optionally
default values and user-supplied values.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.StandardScalerModel.extractParamMap">
<code class="descname">extractParamMap</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.StandardScalerModel.extractParamMap" title="Permalink to this definition">¶</a></dt>
<dd><p>Extracts the embedded default param values and user-supplied
values, and then merges them with extra values from input into
a flat param map, where the latter value is used if there exist
conflicts, i.e., with ordering: default param values <
user-supplied values < extra.
:param extra: extra param values
:return: merged param map</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.StandardScalerModel.getOrDefault">
<code class="descname">getOrDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.StandardScalerModel.getOrDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of a param in the user-supplied param map or its
default value. Raises an error if neither is set.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.StandardScalerModel.getParam">
<code class="descname">getParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.StandardScalerModel.getParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets a param by its name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.StandardScalerModel.hasDefault">
<code class="descname">hasDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.StandardScalerModel.hasDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param has a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.StandardScalerModel.hasParam">
<code class="descname">hasParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.StandardScalerModel.hasParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Tests whether this instance contains a param with a given
(string) name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.StandardScalerModel.isDefined">
<code class="descname">isDefined</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.StandardScalerModel.isDefined" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user or has
a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.StandardScalerModel.isSet">
<code class="descname">isSet</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.StandardScalerModel.isSet" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.feature.StandardScalerModel.mean">
<code class="descname">mean</code><a class="headerlink" href="#pyspark.ml.feature.StandardScalerModel.mean" title="Permalink to this definition">¶</a></dt>
<dd><p>Mean of the StandardScalerModel.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.feature.StandardScalerModel.params">
<code class="descname">params</code><a class="headerlink" href="#pyspark.ml.feature.StandardScalerModel.params" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns all params ordered by name. The default implementation
uses <code class="xref py py-func docutils literal"><span class="pre">dir()</span></code> to get all attributes of type
<code class="xref py py-class docutils literal"><span class="pre">Param</span></code>.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.feature.StandardScalerModel.std">
<code class="descname">std</code><a class="headerlink" href="#pyspark.ml.feature.StandardScalerModel.std" title="Permalink to this definition">¶</a></dt>
<dd><p>Standard deviation of the StandardScalerModel.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.StandardScalerModel.transform">
<code class="descname">transform</code><span class="sig-paren">(</span><em>dataset</em>, <em>params=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.StandardScalerModel.transform" title="Permalink to this definition">¶</a></dt>
<dd><p>Transforms the input dataset with optional parameters.</p>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple">
<li><strong>dataset</strong> – input dataset, which is an instance of
<a class="reference internal" href="pyspark.sql.html#pyspark.sql.DataFrame" title="pyspark.sql.DataFrame"><code class="xref py py-class docutils literal"><span class="pre">pyspark.sql.DataFrame</span></code></a></li>
<li><strong>params</strong> – an optional param map that overrides embedded
params.</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body"><p class="first last">transformed dataset</p>
</td>
</tr>
</tbody>
</table>
</dd></dl>
</dd></dl>
<dl class="class">
<dt id="pyspark.ml.feature.StringIndexer">
<em class="property">class </em><code class="descclassname">pyspark.ml.feature.</code><code class="descname">StringIndexer</code><span class="sig-paren">(</span><em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/feature.html#StringIndexer"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.feature.StringIndexer" title="Permalink to this definition">¶</a></dt>
<dd><p>A label indexer that maps a string column of labels to an ML column of label indices.
If the input column is numeric, we cast it to string and index the string values.
The indices are in [0, numLabels), ordered by label frequencies.
So the most frequent label gets index 0.</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="n">stringIndexer</span> <span class="o">=</span> <span class="n">StringIndexer</span><span class="p">(</span><span class="n">inputCol</span><span class="o">=</span><span class="s">"label"</span><span class="p">,</span> <span class="n">outputCol</span><span class="o">=</span><span class="s">"indexed"</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">model</span> <span class="o">=</span> <span class="n">stringIndexer</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">stringIndDf</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">td</span> <span class="o">=</span> <span class="n">model</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">stringIndDf</span><span class="p">)</span>
<span class="gp">>>> </span><span class="nb">sorted</span><span class="p">(</span><span class="nb">set</span><span class="p">([(</span><span class="n">i</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="n">i</span><span class="p">[</span><span class="mi">1</span><span class="p">])</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="n">td</span><span class="o">.</span><span class="n">select</span><span class="p">(</span><span class="n">td</span><span class="o">.</span><span class="n">id</span><span class="p">,</span> <span class="n">td</span><span class="o">.</span><span class="n">indexed</span><span class="p">)</span><span class="o">.</span><span class="n">collect</span><span class="p">()]),</span>
<span class="gp">... </span> <span class="n">key</span><span class="o">=</span><span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="n">x</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span>
<span class="go">[(0, 0.0), (1, 2.0), (2, 1.0), (3, 0.0), (4, 0.0), (5, 1.0)]</span>
</pre></div>
</div>
<dl class="method">
<dt id="pyspark.ml.feature.StringIndexer.copy">
<code class="descname">copy</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.StringIndexer.copy" title="Permalink to this definition">¶</a></dt>
<dd><p>Creates a copy of this instance with the same uid and some
extra params. The default implementation creates a
shallow copy using <code class="xref py py-func docutils literal"><span class="pre">copy.copy()</span></code>, and then copies the
embedded and extra parameters over and returns the copy.
Subclasses should override this method if the default approach
is not sufficient.
:param extra: Extra parameters to copy to the new instance
:return: Copy of this instance</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.StringIndexer.explainParam">
<code class="descname">explainParam</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.StringIndexer.explainParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Explains a single param and returns its name, doc, and optional
default value and user-supplied value in a string.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.StringIndexer.explainParams">
<code class="descname">explainParams</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.StringIndexer.explainParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns the documentation of all params with their optionally
default values and user-supplied values.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.StringIndexer.extractParamMap">
<code class="descname">extractParamMap</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.StringIndexer.extractParamMap" title="Permalink to this definition">¶</a></dt>
<dd><p>Extracts the embedded default param values and user-supplied
values, and then merges them with extra values from input into
a flat param map, where the latter value is used if there exist
conflicts, i.e., with ordering: default param values <
user-supplied values < extra.
:param extra: extra param values
:return: merged param map</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.StringIndexer.fit">
<code class="descname">fit</code><span class="sig-paren">(</span><em>dataset</em>, <em>params=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.StringIndexer.fit" title="Permalink to this definition">¶</a></dt>
<dd><p>Fits a model to the input dataset with optional parameters.</p>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple">
<li><strong>dataset</strong> – input dataset, which is an instance of
<a class="reference internal" href="pyspark.sql.html#pyspark.sql.DataFrame" title="pyspark.sql.DataFrame"><code class="xref py py-class docutils literal"><span class="pre">pyspark.sql.DataFrame</span></code></a></li>
<li><strong>params</strong> – an optional param map that overrides embedded
params. If a list/tuple of param maps is given,
this calls fit on each param map and returns a
list of models.</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body"><p class="first last">fitted model(s)</p>
</td>
</tr>
</tbody>
</table>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.StringIndexer.getInputCol">
<code class="descname">getInputCol</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.StringIndexer.getInputCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of inputCol or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.StringIndexer.getOrDefault">
<code class="descname">getOrDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.StringIndexer.getOrDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of a param in the user-supplied param map or its
default value. Raises an error if neither is set.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.StringIndexer.getOutputCol">
<code class="descname">getOutputCol</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.StringIndexer.getOutputCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of outputCol or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.StringIndexer.getParam">
<code class="descname">getParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.StringIndexer.getParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets a param by its name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.StringIndexer.hasDefault">
<code class="descname">hasDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.StringIndexer.hasDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param has a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.StringIndexer.hasParam">
<code class="descname">hasParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.StringIndexer.hasParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Tests whether this instance contains a param with a given
(string) name.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.feature.StringIndexer.inputCol">
<code class="descname">inputCol</code><em class="property"> = Param(parent='undefined', name='inputCol', doc='input column name')</em><a class="headerlink" href="#pyspark.ml.feature.StringIndexer.inputCol" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.StringIndexer.isDefined">
<code class="descname">isDefined</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.StringIndexer.isDefined" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user or has
a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.StringIndexer.isSet">
<code class="descname">isSet</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.StringIndexer.isSet" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.feature.StringIndexer.outputCol">
<code class="descname">outputCol</code><em class="property"> = Param(parent='undefined', name='outputCol', doc='output column name')</em><a class="headerlink" href="#pyspark.ml.feature.StringIndexer.outputCol" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.feature.StringIndexer.params">
<code class="descname">params</code><a class="headerlink" href="#pyspark.ml.feature.StringIndexer.params" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns all params ordered by name. The default implementation
uses <code class="xref py py-func docutils literal"><span class="pre">dir()</span></code> to get all attributes of type
<code class="xref py py-class docutils literal"><span class="pre">Param</span></code>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.StringIndexer.setInputCol">
<code class="descname">setInputCol</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.StringIndexer.setInputCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.feature.StringIndexer.inputCol" title="pyspark.ml.feature.StringIndexer.inputCol"><code class="xref py py-attr docutils literal"><span class="pre">inputCol</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.StringIndexer.setOutputCol">
<code class="descname">setOutputCol</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.StringIndexer.setOutputCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.feature.StringIndexer.outputCol" title="pyspark.ml.feature.StringIndexer.outputCol"><code class="xref py py-attr docutils literal"><span class="pre">outputCol</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.StringIndexer.setParams">
<code class="descname">setParams</code><span class="sig-paren">(</span><em>self</em>, <em>inputCol=None</em>, <em>outputCol=None</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/feature.html#StringIndexer.setParams"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.feature.StringIndexer.setParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets params for this StringIndexer.</p>
</dd></dl>
</dd></dl>
<dl class="class">
<dt id="pyspark.ml.feature.StringIndexerModel">
<em class="property">class </em><code class="descclassname">pyspark.ml.feature.</code><code class="descname">StringIndexerModel</code><span class="sig-paren">(</span><em>java_model</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/feature.html#StringIndexerModel"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.feature.StringIndexerModel" title="Permalink to this definition">¶</a></dt>
<dd><p>Model fitted by StringIndexer.</p>
<dl class="method">
<dt id="pyspark.ml.feature.StringIndexerModel.copy">
<code class="descname">copy</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.StringIndexerModel.copy" title="Permalink to this definition">¶</a></dt>
<dd><p>Creates a copy of this instance with the same uid and some
extra params. This implementation first calls Params.copy and
then make a copy of the companion Java model with extra params.
So both the Python wrapper and the Java model get copied.
:param extra: Extra parameters to copy to the new instance
:return: Copy of this instance</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.StringIndexerModel.explainParam">
<code class="descname">explainParam</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.StringIndexerModel.explainParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Explains a single param and returns its name, doc, and optional
default value and user-supplied value in a string.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.StringIndexerModel.explainParams">
<code class="descname">explainParams</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.StringIndexerModel.explainParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns the documentation of all params with their optionally
default values and user-supplied values.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.StringIndexerModel.extractParamMap">
<code class="descname">extractParamMap</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.StringIndexerModel.extractParamMap" title="Permalink to this definition">¶</a></dt>
<dd><p>Extracts the embedded default param values and user-supplied
values, and then merges them with extra values from input into
a flat param map, where the latter value is used if there exist
conflicts, i.e., with ordering: default param values <
user-supplied values < extra.
:param extra: extra param values
:return: merged param map</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.StringIndexerModel.getOrDefault">
<code class="descname">getOrDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.StringIndexerModel.getOrDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of a param in the user-supplied param map or its
default value. Raises an error if neither is set.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.StringIndexerModel.getParam">
<code class="descname">getParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.StringIndexerModel.getParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets a param by its name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.StringIndexerModel.hasDefault">
<code class="descname">hasDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.StringIndexerModel.hasDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param has a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.StringIndexerModel.hasParam">
<code class="descname">hasParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.StringIndexerModel.hasParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Tests whether this instance contains a param with a given
(string) name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.StringIndexerModel.isDefined">
<code class="descname">isDefined</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.StringIndexerModel.isDefined" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user or has
a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.StringIndexerModel.isSet">
<code class="descname">isSet</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.StringIndexerModel.isSet" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.feature.StringIndexerModel.params">
<code class="descname">params</code><a class="headerlink" href="#pyspark.ml.feature.StringIndexerModel.params" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns all params ordered by name. The default implementation
uses <code class="xref py py-func docutils literal"><span class="pre">dir()</span></code> to get all attributes of type
<code class="xref py py-class docutils literal"><span class="pre">Param</span></code>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.StringIndexerModel.transform">
<code class="descname">transform</code><span class="sig-paren">(</span><em>dataset</em>, <em>params=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.StringIndexerModel.transform" title="Permalink to this definition">¶</a></dt>
<dd><p>Transforms the input dataset with optional parameters.</p>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple">
<li><strong>dataset</strong> – input dataset, which is an instance of
<a class="reference internal" href="pyspark.sql.html#pyspark.sql.DataFrame" title="pyspark.sql.DataFrame"><code class="xref py py-class docutils literal"><span class="pre">pyspark.sql.DataFrame</span></code></a></li>
<li><strong>params</strong> – an optional param map that overrides embedded
params.</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body"><p class="first last">transformed dataset</p>
</td>
</tr>
</tbody>
</table>
</dd></dl>
</dd></dl>
<dl class="class">
<dt id="pyspark.ml.feature.Tokenizer">
<em class="property">class </em><code class="descclassname">pyspark.ml.feature.</code><code class="descname">Tokenizer</code><span class="sig-paren">(</span><em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/feature.html#Tokenizer"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.feature.Tokenizer" title="Permalink to this definition">¶</a></dt>
<dd><p>A tokenizer that converts the input string to lowercase and then
splits it by white spaces.</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="n">df</span> <span class="o">=</span> <span class="n">sqlContext</span><span class="o">.</span><span class="n">createDataFrame</span><span class="p">([(</span><span class="s">"a b c"</span><span class="p">,)],</span> <span class="p">[</span><span class="s">"text"</span><span class="p">])</span>
<span class="gp">>>> </span><span class="n">tokenizer</span> <span class="o">=</span> <span class="n">Tokenizer</span><span class="p">(</span><span class="n">inputCol</span><span class="o">=</span><span class="s">"text"</span><span class="p">,</span> <span class="n">outputCol</span><span class="o">=</span><span class="s">"words"</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">tokenizer</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">df</span><span class="p">)</span><span class="o">.</span><span class="n">head</span><span class="p">()</span>
<span class="go">Row(text=u'a b c', words=[u'a', u'b', u'c'])</span>
<span class="gp">>>> </span><span class="c"># Change a parameter.</span>
<span class="gp">>>> </span><span class="n">tokenizer</span><span class="o">.</span><span class="n">setParams</span><span class="p">(</span><span class="n">outputCol</span><span class="o">=</span><span class="s">"tokens"</span><span class="p">)</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">df</span><span class="p">)</span><span class="o">.</span><span class="n">head</span><span class="p">()</span>
<span class="go">Row(text=u'a b c', tokens=[u'a', u'b', u'c'])</span>
<span class="gp">>>> </span><span class="c"># Temporarily modify a parameter.</span>
<span class="gp">>>> </span><span class="n">tokenizer</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">df</span><span class="p">,</span> <span class="p">{</span><span class="n">tokenizer</span><span class="o">.</span><span class="n">outputCol</span><span class="p">:</span> <span class="s">"words"</span><span class="p">})</span><span class="o">.</span><span class="n">head</span><span class="p">()</span>
<span class="go">Row(text=u'a b c', words=[u'a', u'b', u'c'])</span>
<span class="gp">>>> </span><span class="n">tokenizer</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">df</span><span class="p">)</span><span class="o">.</span><span class="n">head</span><span class="p">()</span>
<span class="go">Row(text=u'a b c', tokens=[u'a', u'b', u'c'])</span>
<span class="gp">>>> </span><span class="c"># Must use keyword arguments to specify params.</span>
<span class="gp">>>> </span><span class="n">tokenizer</span><span class="o">.</span><span class="n">setParams</span><span class="p">(</span><span class="s">"text"</span><span class="p">)</span>
<span class="gt">Traceback (most recent call last):</span>
<span class="o">...</span>
<span class="gr">TypeError</span>: <span class="n">Method setParams forces keyword arguments.</span>
</pre></div>
</div>
<dl class="method">
<dt id="pyspark.ml.feature.Tokenizer.copy">
<code class="descname">copy</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Tokenizer.copy" title="Permalink to this definition">¶</a></dt>
<dd><p>Creates a copy of this instance with the same uid and some
extra params. The default implementation creates a
shallow copy using <code class="xref py py-func docutils literal"><span class="pre">copy.copy()</span></code>, and then copies the
embedded and extra parameters over and returns the copy.
Subclasses should override this method if the default approach
is not sufficient.
:param extra: Extra parameters to copy to the new instance
:return: Copy of this instance</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Tokenizer.explainParam">
<code class="descname">explainParam</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Tokenizer.explainParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Explains a single param and returns its name, doc, and optional
default value and user-supplied value in a string.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Tokenizer.explainParams">
<code class="descname">explainParams</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Tokenizer.explainParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns the documentation of all params with their optionally
default values and user-supplied values.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Tokenizer.extractParamMap">
<code class="descname">extractParamMap</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Tokenizer.extractParamMap" title="Permalink to this definition">¶</a></dt>
<dd><p>Extracts the embedded default param values and user-supplied
values, and then merges them with extra values from input into
a flat param map, where the latter value is used if there exist
conflicts, i.e., with ordering: default param values <
user-supplied values < extra.
:param extra: extra param values
:return: merged param map</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Tokenizer.getInputCol">
<code class="descname">getInputCol</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Tokenizer.getInputCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of inputCol or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Tokenizer.getOrDefault">
<code class="descname">getOrDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Tokenizer.getOrDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of a param in the user-supplied param map or its
default value. Raises an error if neither is set.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Tokenizer.getOutputCol">
<code class="descname">getOutputCol</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Tokenizer.getOutputCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of outputCol or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Tokenizer.getParam">
<code class="descname">getParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Tokenizer.getParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets a param by its name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Tokenizer.hasDefault">
<code class="descname">hasDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Tokenizer.hasDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param has a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Tokenizer.hasParam">
<code class="descname">hasParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Tokenizer.hasParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Tests whether this instance contains a param with a given
(string) name.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.feature.Tokenizer.inputCol">
<code class="descname">inputCol</code><em class="property"> = Param(parent='undefined', name='inputCol', doc='input column name')</em><a class="headerlink" href="#pyspark.ml.feature.Tokenizer.inputCol" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Tokenizer.isDefined">
<code class="descname">isDefined</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Tokenizer.isDefined" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user or has
a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Tokenizer.isSet">
<code class="descname">isSet</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Tokenizer.isSet" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.feature.Tokenizer.outputCol">
<code class="descname">outputCol</code><em class="property"> = Param(parent='undefined', name='outputCol', doc='output column name')</em><a class="headerlink" href="#pyspark.ml.feature.Tokenizer.outputCol" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.feature.Tokenizer.params">
<code class="descname">params</code><a class="headerlink" href="#pyspark.ml.feature.Tokenizer.params" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns all params ordered by name. The default implementation
uses <code class="xref py py-func docutils literal"><span class="pre">dir()</span></code> to get all attributes of type
<code class="xref py py-class docutils literal"><span class="pre">Param</span></code>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Tokenizer.setInputCol">
<code class="descname">setInputCol</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Tokenizer.setInputCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.feature.Tokenizer.inputCol" title="pyspark.ml.feature.Tokenizer.inputCol"><code class="xref py py-attr docutils literal"><span class="pre">inputCol</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Tokenizer.setOutputCol">
<code class="descname">setOutputCol</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Tokenizer.setOutputCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.feature.Tokenizer.outputCol" title="pyspark.ml.feature.Tokenizer.outputCol"><code class="xref py py-attr docutils literal"><span class="pre">outputCol</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Tokenizer.setParams">
<code class="descname">setParams</code><span class="sig-paren">(</span><em>self</em>, <em>inputCol="input"</em>, <em>outputCol="output"</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/feature.html#Tokenizer.setParams"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.feature.Tokenizer.setParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets params for this Tokenizer.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Tokenizer.transform">
<code class="descname">transform</code><span class="sig-paren">(</span><em>dataset</em>, <em>params=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Tokenizer.transform" title="Permalink to this definition">¶</a></dt>
<dd><p>Transforms the input dataset with optional parameters.</p>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple">
<li><strong>dataset</strong> – input dataset, which is an instance of
<a class="reference internal" href="pyspark.sql.html#pyspark.sql.DataFrame" title="pyspark.sql.DataFrame"><code class="xref py py-class docutils literal"><span class="pre">pyspark.sql.DataFrame</span></code></a></li>
<li><strong>params</strong> – an optional param map that overrides embedded
params.</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body"><p class="first last">transformed dataset</p>
</td>
</tr>
</tbody>
</table>
</dd></dl>
</dd></dl>
<dl class="class">
<dt id="pyspark.ml.feature.VectorAssembler">
<em class="property">class </em><code class="descclassname">pyspark.ml.feature.</code><code class="descname">VectorAssembler</code><span class="sig-paren">(</span><em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/feature.html#VectorAssembler"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.feature.VectorAssembler" title="Permalink to this definition">¶</a></dt>
<dd><p>A feature transformer that merges multiple columns into a vector column.</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="n">df</span> <span class="o">=</span> <span class="n">sqlContext</span><span class="o">.</span><span class="n">createDataFrame</span><span class="p">([(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">3</span><span class="p">)],</span> <span class="p">[</span><span class="s">"a"</span><span class="p">,</span> <span class="s">"b"</span><span class="p">,</span> <span class="s">"c"</span><span class="p">])</span>
<span class="gp">>>> </span><span class="n">vecAssembler</span> <span class="o">=</span> <span class="n">VectorAssembler</span><span class="p">(</span><span class="n">inputCols</span><span class="o">=</span><span class="p">[</span><span class="s">"a"</span><span class="p">,</span> <span class="s">"b"</span><span class="p">,</span> <span class="s">"c"</span><span class="p">],</span> <span class="n">outputCol</span><span class="o">=</span><span class="s">"features"</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">vecAssembler</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">df</span><span class="p">)</span><span class="o">.</span><span class="n">head</span><span class="p">()</span><span class="o">.</span><span class="n">features</span>
<span class="go">DenseVector([1.0, 0.0, 3.0])</span>
<span class="gp">>>> </span><span class="n">vecAssembler</span><span class="o">.</span><span class="n">setParams</span><span class="p">(</span><span class="n">outputCol</span><span class="o">=</span><span class="s">"freqs"</span><span class="p">)</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">df</span><span class="p">)</span><span class="o">.</span><span class="n">head</span><span class="p">()</span><span class="o">.</span><span class="n">freqs</span>
<span class="go">DenseVector([1.0, 0.0, 3.0])</span>
<span class="gp">>>> </span><span class="n">params</span> <span class="o">=</span> <span class="p">{</span><span class="n">vecAssembler</span><span class="o">.</span><span class="n">inputCols</span><span class="p">:</span> <span class="p">[</span><span class="s">"b"</span><span class="p">,</span> <span class="s">"a"</span><span class="p">],</span> <span class="n">vecAssembler</span><span class="o">.</span><span class="n">outputCol</span><span class="p">:</span> <span class="s">"vector"</span><span class="p">}</span>
<span class="gp">>>> </span><span class="n">vecAssembler</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">df</span><span class="p">,</span> <span class="n">params</span><span class="p">)</span><span class="o">.</span><span class="n">head</span><span class="p">()</span><span class="o">.</span><span class="n">vector</span>
<span class="go">DenseVector([0.0, 1.0])</span>
</pre></div>
</div>
<dl class="method">
<dt id="pyspark.ml.feature.VectorAssembler.copy">
<code class="descname">copy</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.VectorAssembler.copy" title="Permalink to this definition">¶</a></dt>
<dd><p>Creates a copy of this instance with the same uid and some
extra params. The default implementation creates a
shallow copy using <code class="xref py py-func docutils literal"><span class="pre">copy.copy()</span></code>, and then copies the
embedded and extra parameters over and returns the copy.
Subclasses should override this method if the default approach
is not sufficient.
:param extra: Extra parameters to copy to the new instance
:return: Copy of this instance</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.VectorAssembler.explainParam">
<code class="descname">explainParam</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.VectorAssembler.explainParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Explains a single param and returns its name, doc, and optional
default value and user-supplied value in a string.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.VectorAssembler.explainParams">
<code class="descname">explainParams</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.VectorAssembler.explainParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns the documentation of all params with their optionally
default values and user-supplied values.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.VectorAssembler.extractParamMap">
<code class="descname">extractParamMap</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.VectorAssembler.extractParamMap" title="Permalink to this definition">¶</a></dt>
<dd><p>Extracts the embedded default param values and user-supplied
values, and then merges them with extra values from input into
a flat param map, where the latter value is used if there exist
conflicts, i.e., with ordering: default param values <
user-supplied values < extra.
:param extra: extra param values
:return: merged param map</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.VectorAssembler.getInputCols">
<code class="descname">getInputCols</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.VectorAssembler.getInputCols" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of inputCols or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.VectorAssembler.getOrDefault">
<code class="descname">getOrDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.VectorAssembler.getOrDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of a param in the user-supplied param map or its
default value. Raises an error if neither is set.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.VectorAssembler.getOutputCol">
<code class="descname">getOutputCol</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.VectorAssembler.getOutputCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of outputCol or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.VectorAssembler.getParam">
<code class="descname">getParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.VectorAssembler.getParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets a param by its name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.VectorAssembler.hasDefault">
<code class="descname">hasDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.VectorAssembler.hasDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param has a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.VectorAssembler.hasParam">
<code class="descname">hasParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.VectorAssembler.hasParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Tests whether this instance contains a param with a given
(string) name.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.feature.VectorAssembler.inputCols">
<code class="descname">inputCols</code><em class="property"> = Param(parent='undefined', name='inputCols', doc='input column names')</em><a class="headerlink" href="#pyspark.ml.feature.VectorAssembler.inputCols" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.VectorAssembler.isDefined">
<code class="descname">isDefined</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.VectorAssembler.isDefined" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user or has
a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.VectorAssembler.isSet">
<code class="descname">isSet</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.VectorAssembler.isSet" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.feature.VectorAssembler.outputCol">
<code class="descname">outputCol</code><em class="property"> = Param(parent='undefined', name='outputCol', doc='output column name')</em><a class="headerlink" href="#pyspark.ml.feature.VectorAssembler.outputCol" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.feature.VectorAssembler.params">
<code class="descname">params</code><a class="headerlink" href="#pyspark.ml.feature.VectorAssembler.params" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns all params ordered by name. The default implementation
uses <code class="xref py py-func docutils literal"><span class="pre">dir()</span></code> to get all attributes of type
<code class="xref py py-class docutils literal"><span class="pre">Param</span></code>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.VectorAssembler.setInputCols">
<code class="descname">setInputCols</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.VectorAssembler.setInputCols" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.feature.VectorAssembler.inputCols" title="pyspark.ml.feature.VectorAssembler.inputCols"><code class="xref py py-attr docutils literal"><span class="pre">inputCols</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.VectorAssembler.setOutputCol">
<code class="descname">setOutputCol</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.VectorAssembler.setOutputCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.feature.VectorAssembler.outputCol" title="pyspark.ml.feature.VectorAssembler.outputCol"><code class="xref py py-attr docutils literal"><span class="pre">outputCol</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.VectorAssembler.setParams">
<code class="descname">setParams</code><span class="sig-paren">(</span><em>self</em>, <em>inputCols=None</em>, <em>outputCol=None</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/feature.html#VectorAssembler.setParams"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.feature.VectorAssembler.setParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets params for this VectorAssembler.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.VectorAssembler.transform">
<code class="descname">transform</code><span class="sig-paren">(</span><em>dataset</em>, <em>params=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.VectorAssembler.transform" title="Permalink to this definition">¶</a></dt>
<dd><p>Transforms the input dataset with optional parameters.</p>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple">
<li><strong>dataset</strong> – input dataset, which is an instance of
<a class="reference internal" href="pyspark.sql.html#pyspark.sql.DataFrame" title="pyspark.sql.DataFrame"><code class="xref py py-class docutils literal"><span class="pre">pyspark.sql.DataFrame</span></code></a></li>
<li><strong>params</strong> – an optional param map that overrides embedded
params.</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body"><p class="first last">transformed dataset</p>
</td>
</tr>
</tbody>
</table>
</dd></dl>
</dd></dl>
<dl class="class">
<dt id="pyspark.ml.feature.VectorIndexer">
<em class="property">class </em><code class="descclassname">pyspark.ml.feature.</code><code class="descname">VectorIndexer</code><span class="sig-paren">(</span><em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/feature.html#VectorIndexer"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.feature.VectorIndexer" title="Permalink to this definition">¶</a></dt>
<dd><p>Class for indexing categorical feature columns in a dataset of [[Vector]].</p>
<dl class="docutils">
<dt>This has 2 usage modes:</dt>
<dd><blockquote class="first">
<div><ul>
<li><dl class="first docutils">
<dt>Automatically identify categorical features (default behavior)</dt>
<dd><ul class="first last simple">
<li>This helps process a dataset of unknown vectors into a dataset with some continuous
features and some categorical features. The choice between continuous and categorical
is based upon a maxCategories parameter.</li>
<li>Set maxCategories to the maximum number of categorical any categorical feature should
have.</li>
<li>E.g.: Feature 0 has unique values {-1.0, 0.0}, and feature 1 values {1.0, 3.0, 5.0}.
If maxCategories = 2, then feature 0 will be declared categorical and use indices {0, 1},
and feature 1 will be declared continuous.</li>
</ul>
</dd>
</dl>
</li>
<li><dl class="first docutils">
<dt>Index all features, if all features are categorical</dt>
<dd><ul class="first last simple">
<li>If maxCategories is set to be very large, then this will build an index of unique
values for all features.</li>
<li>Warning: This can cause problems if features are continuous since this will collect ALL
unique values to the driver.</li>
<li>E.g.: Feature 0 has unique values {-1.0, 0.0}, and feature 1 values {1.0, 3.0, 5.0}.
If maxCategories >= 3, then both features will be declared categorical.</li>
</ul>
</dd>
</dl>
</li>
</ul>
</div></blockquote>
<p class="last">This returns a model which can transform categorical features to use 0-based indices.</p>
</dd>
<dt>Index stability:</dt>
<dd><blockquote class="first">
<div><ul class="simple">
<li>This is not guaranteed to choose the same category index across multiple runs.</li>
<li>If a categorical feature includes value 0, then this is guaranteed to map value 0 to
index 0. This maintains vector sparsity.</li>
<li>More stability may be added in the future.</li>
</ul>
</div></blockquote>
<dl class="last docutils">
<dt>TODO: Future extensions: The following functionality is planned for the future:</dt>
<dd><ul class="first last simple">
<li>Preserve metadata in transform; if a feature’s metadata is already present,
do not recompute.</li>
<li>Specify certain features to not index, either via a parameter or via existing metadata.</li>
<li>Add warning if a categorical feature has only 1 category.</li>
<li>Add option for allowing unknown categories.</li>
</ul>
</dd>
</dl>
</dd>
</dl>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="kn">from</span> <span class="nn">pyspark.mllib.linalg</span> <span class="kn">import</span> <span class="n">Vectors</span>
<span class="gp">>>> </span><span class="n">df</span> <span class="o">=</span> <span class="n">sqlContext</span><span class="o">.</span><span class="n">createDataFrame</span><span class="p">([(</span><span class="n">Vectors</span><span class="o">.</span><span class="n">dense</span><span class="p">([</span><span class="o">-</span><span class="mf">1.0</span><span class="p">,</span> <span class="mf">0.0</span><span class="p">]),),</span>
<span class="gp">... </span> <span class="p">(</span><span class="n">Vectors</span><span class="o">.</span><span class="n">dense</span><span class="p">([</span><span class="mf">0.0</span><span class="p">,</span> <span class="mf">1.0</span><span class="p">]),),</span> <span class="p">(</span><span class="n">Vectors</span><span class="o">.</span><span class="n">dense</span><span class="p">([</span><span class="mf">0.0</span><span class="p">,</span> <span class="mf">2.0</span><span class="p">]),)],</span> <span class="p">[</span><span class="s">"a"</span><span class="p">])</span>
<span class="gp">>>> </span><span class="n">indexer</span> <span class="o">=</span> <span class="n">VectorIndexer</span><span class="p">(</span><span class="n">maxCategories</span><span class="o">=</span><span class="mi">2</span><span class="p">,</span> <span class="n">inputCol</span><span class="o">=</span><span class="s">"a"</span><span class="p">,</span> <span class="n">outputCol</span><span class="o">=</span><span class="s">"indexed"</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">model</span> <span class="o">=</span> <span class="n">indexer</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">df</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">model</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">df</span><span class="p">)</span><span class="o">.</span><span class="n">head</span><span class="p">()</span><span class="o">.</span><span class="n">indexed</span>
<span class="go">DenseVector([1.0, 0.0])</span>
<span class="gp">>>> </span><span class="n">indexer</span><span class="o">.</span><span class="n">setParams</span><span class="p">(</span><span class="n">outputCol</span><span class="o">=</span><span class="s">"test"</span><span class="p">)</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">df</span><span class="p">)</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">df</span><span class="p">)</span><span class="o">.</span><span class="n">collect</span><span class="p">()[</span><span class="mi">1</span><span class="p">]</span><span class="o">.</span><span class="n">test</span>
<span class="go">DenseVector([0.0, 1.0])</span>
<span class="gp">>>> </span><span class="n">params</span> <span class="o">=</span> <span class="p">{</span><span class="n">indexer</span><span class="o">.</span><span class="n">maxCategories</span><span class="p">:</span> <span class="mi">3</span><span class="p">,</span> <span class="n">indexer</span><span class="o">.</span><span class="n">outputCol</span><span class="p">:</span> <span class="s">"vector"</span><span class="p">}</span>
<span class="gp">>>> </span><span class="n">model2</span> <span class="o">=</span> <span class="n">indexer</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">df</span><span class="p">,</span> <span class="n">params</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">model2</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">df</span><span class="p">)</span><span class="o">.</span><span class="n">head</span><span class="p">()</span><span class="o">.</span><span class="n">vector</span>
<span class="go">DenseVector([1.0, 0.0])</span>
</pre></div>
</div>
<dl class="method">
<dt id="pyspark.ml.feature.VectorIndexer.copy">
<code class="descname">copy</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.VectorIndexer.copy" title="Permalink to this definition">¶</a></dt>
<dd><p>Creates a copy of this instance with the same uid and some
extra params. The default implementation creates a
shallow copy using <code class="xref py py-func docutils literal"><span class="pre">copy.copy()</span></code>, and then copies the
embedded and extra parameters over and returns the copy.
Subclasses should override this method if the default approach
is not sufficient.
:param extra: Extra parameters to copy to the new instance
:return: Copy of this instance</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.VectorIndexer.explainParam">
<code class="descname">explainParam</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.VectorIndexer.explainParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Explains a single param and returns its name, doc, and optional
default value and user-supplied value in a string.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.VectorIndexer.explainParams">
<code class="descname">explainParams</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.VectorIndexer.explainParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns the documentation of all params with their optionally
default values and user-supplied values.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.VectorIndexer.extractParamMap">
<code class="descname">extractParamMap</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.VectorIndexer.extractParamMap" title="Permalink to this definition">¶</a></dt>
<dd><p>Extracts the embedded default param values and user-supplied
values, and then merges them with extra values from input into
a flat param map, where the latter value is used if there exist
conflicts, i.e., with ordering: default param values <
user-supplied values < extra.
:param extra: extra param values
:return: merged param map</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.VectorIndexer.fit">
<code class="descname">fit</code><span class="sig-paren">(</span><em>dataset</em>, <em>params=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.VectorIndexer.fit" title="Permalink to this definition">¶</a></dt>
<dd><p>Fits a model to the input dataset with optional parameters.</p>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple">
<li><strong>dataset</strong> – input dataset, which is an instance of
<a class="reference internal" href="pyspark.sql.html#pyspark.sql.DataFrame" title="pyspark.sql.DataFrame"><code class="xref py py-class docutils literal"><span class="pre">pyspark.sql.DataFrame</span></code></a></li>
<li><strong>params</strong> – an optional param map that overrides embedded
params. If a list/tuple of param maps is given,
this calls fit on each param map and returns a
list of models.</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body"><p class="first last">fitted model(s)</p>
</td>
</tr>
</tbody>
</table>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.VectorIndexer.getInputCol">
<code class="descname">getInputCol</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.VectorIndexer.getInputCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of inputCol or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.VectorIndexer.getMaxCategories">
<code class="descname">getMaxCategories</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/feature.html#VectorIndexer.getMaxCategories"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.feature.VectorIndexer.getMaxCategories" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of maxCategories or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.VectorIndexer.getOrDefault">
<code class="descname">getOrDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.VectorIndexer.getOrDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of a param in the user-supplied param map or its
default value. Raises an error if neither is set.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.VectorIndexer.getOutputCol">
<code class="descname">getOutputCol</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.VectorIndexer.getOutputCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of outputCol or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.VectorIndexer.getParam">
<code class="descname">getParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.VectorIndexer.getParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets a param by its name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.VectorIndexer.hasDefault">
<code class="descname">hasDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.VectorIndexer.hasDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param has a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.VectorIndexer.hasParam">
<code class="descname">hasParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.VectorIndexer.hasParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Tests whether this instance contains a param with a given
(string) name.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.feature.VectorIndexer.inputCol">
<code class="descname">inputCol</code><em class="property"> = Param(parent='undefined', name='inputCol', doc='input column name')</em><a class="headerlink" href="#pyspark.ml.feature.VectorIndexer.inputCol" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.VectorIndexer.isDefined">
<code class="descname">isDefined</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.VectorIndexer.isDefined" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user or has
a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.VectorIndexer.isSet">
<code class="descname">isSet</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.VectorIndexer.isSet" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.feature.VectorIndexer.maxCategories">
<code class="descname">maxCategories</code><em class="property"> = Param(parent='undefined', name='maxCategories', doc='Threshold for the number of values a categorical feature can take (>= 2). If a feature is found to have > maxCategories values, then it is declared continuous.')</em><a class="headerlink" href="#pyspark.ml.feature.VectorIndexer.maxCategories" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.feature.VectorIndexer.outputCol">
<code class="descname">outputCol</code><em class="property"> = Param(parent='undefined', name='outputCol', doc='output column name')</em><a class="headerlink" href="#pyspark.ml.feature.VectorIndexer.outputCol" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.feature.VectorIndexer.params">
<code class="descname">params</code><a class="headerlink" href="#pyspark.ml.feature.VectorIndexer.params" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns all params ordered by name. The default implementation
uses <code class="xref py py-func docutils literal"><span class="pre">dir()</span></code> to get all attributes of type
<code class="xref py py-class docutils literal"><span class="pre">Param</span></code>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.VectorIndexer.setInputCol">
<code class="descname">setInputCol</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.VectorIndexer.setInputCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.feature.VectorIndexer.inputCol" title="pyspark.ml.feature.VectorIndexer.inputCol"><code class="xref py py-attr docutils literal"><span class="pre">inputCol</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.VectorIndexer.setMaxCategories">
<code class="descname">setMaxCategories</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/feature.html#VectorIndexer.setMaxCategories"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.feature.VectorIndexer.setMaxCategories" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.feature.VectorIndexer.maxCategories" title="pyspark.ml.feature.VectorIndexer.maxCategories"><code class="xref py py-attr docutils literal"><span class="pre">maxCategories</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.VectorIndexer.setOutputCol">
<code class="descname">setOutputCol</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.VectorIndexer.setOutputCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.feature.VectorIndexer.outputCol" title="pyspark.ml.feature.VectorIndexer.outputCol"><code class="xref py py-attr docutils literal"><span class="pre">outputCol</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.VectorIndexer.setParams">
<code class="descname">setParams</code><span class="sig-paren">(</span><em>self</em>, <em>maxCategories=20</em>, <em>inputCol=None</em>, <em>outputCol=None</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/feature.html#VectorIndexer.setParams"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.feature.VectorIndexer.setParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets params for this VectorIndexer.</p>
</dd></dl>
</dd></dl>
<dl class="class">
<dt id="pyspark.ml.feature.Word2Vec">
<em class="property">class </em><code class="descclassname">pyspark.ml.feature.</code><code class="descname">Word2Vec</code><span class="sig-paren">(</span><em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/feature.html#Word2Vec"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.feature.Word2Vec" title="Permalink to this definition">¶</a></dt>
<dd><p>Word2Vec trains a model of <cite>Map(String, Vector)</cite>, i.e. transforms a word into a code for further
natural language processing or machine learning process.</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="n">sent</span> <span class="o">=</span> <span class="p">(</span><span class="s">"a b "</span> <span class="o">*</span> <span class="mi">100</span> <span class="o">+</span> <span class="s">"a c "</span> <span class="o">*</span> <span class="mi">10</span><span class="p">)</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s">" "</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">doc</span> <span class="o">=</span> <span class="n">sqlContext</span><span class="o">.</span><span class="n">createDataFrame</span><span class="p">([(</span><span class="n">sent</span><span class="p">,),</span> <span class="p">(</span><span class="n">sent</span><span class="p">,)],</span> <span class="p">[</span><span class="s">"sentence"</span><span class="p">])</span>
<span class="gp">>>> </span><span class="n">model</span> <span class="o">=</span> <span class="n">Word2Vec</span><span class="p">(</span><span class="n">vectorSize</span><span class="o">=</span><span class="mi">5</span><span class="p">,</span> <span class="n">seed</span><span class="o">=</span><span class="mi">42</span><span class="p">,</span> <span class="n">inputCol</span><span class="o">=</span><span class="s">"sentence"</span><span class="p">,</span> <span class="n">outputCol</span><span class="o">=</span><span class="s">"model"</span><span class="p">)</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">doc</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">model</span><span class="o">.</span><span class="n">getVectors</span><span class="p">()</span><span class="o">.</span><span class="n">show</span><span class="p">()</span>
<span class="go">+----+--------------------+</span>
<span class="go">|word| vector|</span>
<span class="go">+----+--------------------+</span>
<span class="go">| a|[-0.3511952459812...|</span>
<span class="go">| b|[0.29077222943305...|</span>
<span class="go">| c|[0.02315592765808...|</span>
<span class="go">+----+--------------------+</span>
<span class="gp">...</span>
<span class="gp">>>> </span><span class="n">model</span><span class="o">.</span><span class="n">findSynonyms</span><span class="p">(</span><span class="s">"a"</span><span class="p">,</span> <span class="mi">2</span><span class="p">)</span><span class="o">.</span><span class="n">show</span><span class="p">()</span>
<span class="go">+----+-------------------+</span>
<span class="go">|word| similarity|</span>
<span class="go">+----+-------------------+</span>
<span class="go">| b|0.29255685145799626|</span>
<span class="go">| c|-0.5414068302988307|</span>
<span class="go">+----+-------------------+</span>
<span class="gp">...</span>
<span class="gp">>>> </span><span class="n">model</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">doc</span><span class="p">)</span><span class="o">.</span><span class="n">head</span><span class="p">()</span><span class="o">.</span><span class="n">model</span>
<span class="go">DenseVector([-0.0422, -0.5138, -0.2546, 0.6885, 0.276])</span>
</pre></div>
</div>
<dl class="method">
<dt id="pyspark.ml.feature.Word2Vec.copy">
<code class="descname">copy</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Word2Vec.copy" title="Permalink to this definition">¶</a></dt>
<dd><p>Creates a copy of this instance with the same uid and some
extra params. The default implementation creates a
shallow copy using <code class="xref py py-func docutils literal"><span class="pre">copy.copy()</span></code>, and then copies the
embedded and extra parameters over and returns the copy.
Subclasses should override this method if the default approach
is not sufficient.
:param extra: Extra parameters to copy to the new instance
:return: Copy of this instance</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Word2Vec.explainParam">
<code class="descname">explainParam</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Word2Vec.explainParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Explains a single param and returns its name, doc, and optional
default value and user-supplied value in a string.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Word2Vec.explainParams">
<code class="descname">explainParams</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Word2Vec.explainParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns the documentation of all params with their optionally
default values and user-supplied values.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Word2Vec.extractParamMap">
<code class="descname">extractParamMap</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Word2Vec.extractParamMap" title="Permalink to this definition">¶</a></dt>
<dd><p>Extracts the embedded default param values and user-supplied
values, and then merges them with extra values from input into
a flat param map, where the latter value is used if there exist
conflicts, i.e., with ordering: default param values <
user-supplied values < extra.
:param extra: extra param values
:return: merged param map</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Word2Vec.fit">
<code class="descname">fit</code><span class="sig-paren">(</span><em>dataset</em>, <em>params=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Word2Vec.fit" title="Permalink to this definition">¶</a></dt>
<dd><p>Fits a model to the input dataset with optional parameters.</p>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple">
<li><strong>dataset</strong> – input dataset, which is an instance of
<a class="reference internal" href="pyspark.sql.html#pyspark.sql.DataFrame" title="pyspark.sql.DataFrame"><code class="xref py py-class docutils literal"><span class="pre">pyspark.sql.DataFrame</span></code></a></li>
<li><strong>params</strong> – an optional param map that overrides embedded
params. If a list/tuple of param maps is given,
this calls fit on each param map and returns a
list of models.</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body"><p class="first last">fitted model(s)</p>
</td>
</tr>
</tbody>
</table>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Word2Vec.getInputCol">
<code class="descname">getInputCol</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Word2Vec.getInputCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of inputCol or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Word2Vec.getMaxIter">
<code class="descname">getMaxIter</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Word2Vec.getMaxIter" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of maxIter or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Word2Vec.getMinCount">
<code class="descname">getMinCount</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/feature.html#Word2Vec.getMinCount"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.feature.Word2Vec.getMinCount" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of minCount or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Word2Vec.getNumPartitions">
<code class="descname">getNumPartitions</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/feature.html#Word2Vec.getNumPartitions"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.feature.Word2Vec.getNumPartitions" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of numPartitions or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Word2Vec.getOrDefault">
<code class="descname">getOrDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Word2Vec.getOrDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of a param in the user-supplied param map or its
default value. Raises an error if neither is set.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Word2Vec.getOutputCol">
<code class="descname">getOutputCol</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Word2Vec.getOutputCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of outputCol or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Word2Vec.getParam">
<code class="descname">getParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Word2Vec.getParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets a param by its name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Word2Vec.getSeed">
<code class="descname">getSeed</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Word2Vec.getSeed" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of seed or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Word2Vec.getStepSize">
<code class="descname">getStepSize</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Word2Vec.getStepSize" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of stepSize or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Word2Vec.getVectorSize">
<code class="descname">getVectorSize</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/feature.html#Word2Vec.getVectorSize"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.feature.Word2Vec.getVectorSize" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of vectorSize or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Word2Vec.hasDefault">
<code class="descname">hasDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Word2Vec.hasDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param has a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Word2Vec.hasParam">
<code class="descname">hasParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Word2Vec.hasParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Tests whether this instance contains a param with a given
(string) name.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.feature.Word2Vec.inputCol">
<code class="descname">inputCol</code><em class="property"> = Param(parent='undefined', name='inputCol', doc='input column name')</em><a class="headerlink" href="#pyspark.ml.feature.Word2Vec.inputCol" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Word2Vec.isDefined">
<code class="descname">isDefined</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Word2Vec.isDefined" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user or has
a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Word2Vec.isSet">
<code class="descname">isSet</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Word2Vec.isSet" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.feature.Word2Vec.maxIter">
<code class="descname">maxIter</code><em class="property"> = Param(parent='undefined', name='maxIter', doc='max number of iterations (>= 0)')</em><a class="headerlink" href="#pyspark.ml.feature.Word2Vec.maxIter" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.feature.Word2Vec.minCount">
<code class="descname">minCount</code><em class="property"> = Param(parent='undefined', name='minCount', doc="the minimum number of times a token must appear to be included in the word2vec model's vocabulary")</em><a class="headerlink" href="#pyspark.ml.feature.Word2Vec.minCount" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.feature.Word2Vec.numPartitions">
<code class="descname">numPartitions</code><em class="property"> = Param(parent='undefined', name='numPartitions', doc='number of partitions for sentences of words')</em><a class="headerlink" href="#pyspark.ml.feature.Word2Vec.numPartitions" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.feature.Word2Vec.outputCol">
<code class="descname">outputCol</code><em class="property"> = Param(parent='undefined', name='outputCol', doc='output column name')</em><a class="headerlink" href="#pyspark.ml.feature.Word2Vec.outputCol" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.feature.Word2Vec.params">
<code class="descname">params</code><a class="headerlink" href="#pyspark.ml.feature.Word2Vec.params" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns all params ordered by name. The default implementation
uses <code class="xref py py-func docutils literal"><span class="pre">dir()</span></code> to get all attributes of type
<code class="xref py py-class docutils literal"><span class="pre">Param</span></code>.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.feature.Word2Vec.seed">
<code class="descname">seed</code><em class="property"> = Param(parent='undefined', name='seed', doc='random seed')</em><a class="headerlink" href="#pyspark.ml.feature.Word2Vec.seed" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Word2Vec.setInputCol">
<code class="descname">setInputCol</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Word2Vec.setInputCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.feature.Word2Vec.inputCol" title="pyspark.ml.feature.Word2Vec.inputCol"><code class="xref py py-attr docutils literal"><span class="pre">inputCol</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Word2Vec.setMaxIter">
<code class="descname">setMaxIter</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Word2Vec.setMaxIter" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.feature.Word2Vec.maxIter" title="pyspark.ml.feature.Word2Vec.maxIter"><code class="xref py py-attr docutils literal"><span class="pre">maxIter</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Word2Vec.setMinCount">
<code class="descname">setMinCount</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/feature.html#Word2Vec.setMinCount"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.feature.Word2Vec.setMinCount" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.feature.Word2Vec.minCount" title="pyspark.ml.feature.Word2Vec.minCount"><code class="xref py py-attr docutils literal"><span class="pre">minCount</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Word2Vec.setNumPartitions">
<code class="descname">setNumPartitions</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/feature.html#Word2Vec.setNumPartitions"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.feature.Word2Vec.setNumPartitions" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.feature.Word2Vec.numPartitions" title="pyspark.ml.feature.Word2Vec.numPartitions"><code class="xref py py-attr docutils literal"><span class="pre">numPartitions</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Word2Vec.setOutputCol">
<code class="descname">setOutputCol</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Word2Vec.setOutputCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.feature.Word2Vec.outputCol" title="pyspark.ml.feature.Word2Vec.outputCol"><code class="xref py py-attr docutils literal"><span class="pre">outputCol</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Word2Vec.setParams">
<code class="descname">setParams</code><span class="sig-paren">(</span><em>self</em>, <em>minCount=5</em>, <em>numPartitions=1</em>, <em>stepSize=0.025</em>, <em>maxIter=1</em>, <em>seed=None</em>, <em>inputCol=None</em>, <em>outputCol=None</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/feature.html#Word2Vec.setParams"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.feature.Word2Vec.setParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets params for this Word2Vec.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Word2Vec.setSeed">
<code class="descname">setSeed</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Word2Vec.setSeed" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.feature.Word2Vec.seed" title="pyspark.ml.feature.Word2Vec.seed"><code class="xref py py-attr docutils literal"><span class="pre">seed</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Word2Vec.setStepSize">
<code class="descname">setStepSize</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Word2Vec.setStepSize" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.feature.Word2Vec.stepSize" title="pyspark.ml.feature.Word2Vec.stepSize"><code class="xref py py-attr docutils literal"><span class="pre">stepSize</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Word2Vec.setVectorSize">
<code class="descname">setVectorSize</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/feature.html#Word2Vec.setVectorSize"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.feature.Word2Vec.setVectorSize" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.feature.Word2Vec.vectorSize" title="pyspark.ml.feature.Word2Vec.vectorSize"><code class="xref py py-attr docutils literal"><span class="pre">vectorSize</span></code></a>.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.feature.Word2Vec.stepSize">
<code class="descname">stepSize</code><em class="property"> = Param(parent='undefined', name='stepSize', doc='Step size to be used for each iteration of optimization.')</em><a class="headerlink" href="#pyspark.ml.feature.Word2Vec.stepSize" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.feature.Word2Vec.vectorSize">
<code class="descname">vectorSize</code><em class="property"> = Param(parent='undefined', name='vectorSize', doc='the dimension of codes after transforming from words')</em><a class="headerlink" href="#pyspark.ml.feature.Word2Vec.vectorSize" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
</dd></dl>
<dl class="class">
<dt id="pyspark.ml.feature.Word2VecModel">
<em class="property">class </em><code class="descclassname">pyspark.ml.feature.</code><code class="descname">Word2VecModel</code><span class="sig-paren">(</span><em>java_model</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/feature.html#Word2VecModel"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.feature.Word2VecModel" title="Permalink to this definition">¶</a></dt>
<dd><p>Model fitted by Word2Vec.</p>
<dl class="method">
<dt id="pyspark.ml.feature.Word2VecModel.copy">
<code class="descname">copy</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Word2VecModel.copy" title="Permalink to this definition">¶</a></dt>
<dd><p>Creates a copy of this instance with the same uid and some
extra params. This implementation first calls Params.copy and
then make a copy of the companion Java model with extra params.
So both the Python wrapper and the Java model get copied.
:param extra: Extra parameters to copy to the new instance
:return: Copy of this instance</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Word2VecModel.explainParam">
<code class="descname">explainParam</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Word2VecModel.explainParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Explains a single param and returns its name, doc, and optional
default value and user-supplied value in a string.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Word2VecModel.explainParams">
<code class="descname">explainParams</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Word2VecModel.explainParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns the documentation of all params with their optionally
default values and user-supplied values.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Word2VecModel.extractParamMap">
<code class="descname">extractParamMap</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Word2VecModel.extractParamMap" title="Permalink to this definition">¶</a></dt>
<dd><p>Extracts the embedded default param values and user-supplied
values, and then merges them with extra values from input into
a flat param map, where the latter value is used if there exist
conflicts, i.e., with ordering: default param values <
user-supplied values < extra.
:param extra: extra param values
:return: merged param map</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Word2VecModel.findSynonyms">
<code class="descname">findSynonyms</code><span class="sig-paren">(</span><em>word</em>, <em>num</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/feature.html#Word2VecModel.findSynonyms"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.feature.Word2VecModel.findSynonyms" title="Permalink to this definition">¶</a></dt>
<dd><p>Find “num” number of words closest in similarity to “word”.
word can be a string or vector representation.
Returns a dataframe with two fields word and similarity (which
gives the cosine similarity).</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Word2VecModel.getOrDefault">
<code class="descname">getOrDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Word2VecModel.getOrDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of a param in the user-supplied param map or its
default value. Raises an error if neither is set.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Word2VecModel.getParam">
<code class="descname">getParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Word2VecModel.getParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets a param by its name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Word2VecModel.getVectors">
<code class="descname">getVectors</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/feature.html#Word2VecModel.getVectors"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.feature.Word2VecModel.getVectors" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns the vector representation of the words as a dataframe
with two fields, word and vector.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Word2VecModel.hasDefault">
<code class="descname">hasDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Word2VecModel.hasDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param has a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Word2VecModel.hasParam">
<code class="descname">hasParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Word2VecModel.hasParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Tests whether this instance contains a param with a given
(string) name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Word2VecModel.isDefined">
<code class="descname">isDefined</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Word2VecModel.isDefined" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user or has
a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Word2VecModel.isSet">
<code class="descname">isSet</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Word2VecModel.isSet" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.feature.Word2VecModel.params">
<code class="descname">params</code><a class="headerlink" href="#pyspark.ml.feature.Word2VecModel.params" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns all params ordered by name. The default implementation
uses <code class="xref py py-func docutils literal"><span class="pre">dir()</span></code> to get all attributes of type
<code class="xref py py-class docutils literal"><span class="pre">Param</span></code>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.Word2VecModel.transform">
<code class="descname">transform</code><span class="sig-paren">(</span><em>dataset</em>, <em>params=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.Word2VecModel.transform" title="Permalink to this definition">¶</a></dt>
<dd><p>Transforms the input dataset with optional parameters.</p>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple">
<li><strong>dataset</strong> – input dataset, which is an instance of
<a class="reference internal" href="pyspark.sql.html#pyspark.sql.DataFrame" title="pyspark.sql.DataFrame"><code class="xref py py-class docutils literal"><span class="pre">pyspark.sql.DataFrame</span></code></a></li>
<li><strong>params</strong> – an optional param map that overrides embedded
params.</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body"><p class="first last">transformed dataset</p>
</td>
</tr>
</tbody>
</table>
</dd></dl>
</dd></dl>
<dl class="class">
<dt id="pyspark.ml.feature.PCA">
<em class="property">class </em><code class="descclassname">pyspark.ml.feature.</code><code class="descname">PCA</code><span class="sig-paren">(</span><em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/feature.html#PCA"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.feature.PCA" title="Permalink to this definition">¶</a></dt>
<dd><p>PCA trains a model to project vectors to a low-dimensional space using PCA.</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="kn">from</span> <span class="nn">pyspark.mllib.linalg</span> <span class="kn">import</span> <span class="n">Vectors</span>
<span class="gp">>>> </span><span class="n">data</span> <span class="o">=</span> <span class="p">[(</span><span class="n">Vectors</span><span class="o">.</span><span class="n">sparse</span><span class="p">(</span><span class="mi">5</span><span class="p">,</span> <span class="p">[(</span><span class="mi">1</span><span class="p">,</span> <span class="mf">1.0</span><span class="p">),</span> <span class="p">(</span><span class="mi">3</span><span class="p">,</span> <span class="mf">7.0</span><span class="p">)]),),</span>
<span class="gp">... </span> <span class="p">(</span><span class="n">Vectors</span><span class="o">.</span><span class="n">dense</span><span class="p">([</span><span class="mf">2.0</span><span class="p">,</span> <span class="mf">0.0</span><span class="p">,</span> <span class="mf">3.0</span><span class="p">,</span> <span class="mf">4.0</span><span class="p">,</span> <span class="mf">5.0</span><span class="p">]),),</span>
<span class="gp">... </span> <span class="p">(</span><span class="n">Vectors</span><span class="o">.</span><span class="n">dense</span><span class="p">([</span><span class="mf">4.0</span><span class="p">,</span> <span class="mf">0.0</span><span class="p">,</span> <span class="mf">0.0</span><span class="p">,</span> <span class="mf">6.0</span><span class="p">,</span> <span class="mf">7.0</span><span class="p">]),)]</span>
<span class="gp">>>> </span><span class="n">df</span> <span class="o">=</span> <span class="n">sqlContext</span><span class="o">.</span><span class="n">createDataFrame</span><span class="p">(</span><span class="n">data</span><span class="p">,[</span><span class="s">"features"</span><span class="p">])</span>
<span class="gp">>>> </span><span class="n">pca</span> <span class="o">=</span> <span class="n">PCA</span><span class="p">(</span><span class="n">k</span><span class="o">=</span><span class="mi">2</span><span class="p">,</span> <span class="n">inputCol</span><span class="o">=</span><span class="s">"features"</span><span class="p">,</span> <span class="n">outputCol</span><span class="o">=</span><span class="s">"pca_features"</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">model</span> <span class="o">=</span> <span class="n">pca</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">df</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">model</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">df</span><span class="p">)</span><span class="o">.</span><span class="n">collect</span><span class="p">()[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">pca_features</span>
<span class="go">DenseVector([1.648..., -4.013...])</span>
</pre></div>
</div>
<dl class="method">
<dt id="pyspark.ml.feature.PCA.copy">
<code class="descname">copy</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.PCA.copy" title="Permalink to this definition">¶</a></dt>
<dd><p>Creates a copy of this instance with the same uid and some
extra params. The default implementation creates a
shallow copy using <code class="xref py py-func docutils literal"><span class="pre">copy.copy()</span></code>, and then copies the
embedded and extra parameters over and returns the copy.
Subclasses should override this method if the default approach
is not sufficient.
:param extra: Extra parameters to copy to the new instance
:return: Copy of this instance</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.PCA.explainParam">
<code class="descname">explainParam</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.PCA.explainParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Explains a single param and returns its name, doc, and optional
default value and user-supplied value in a string.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.PCA.explainParams">
<code class="descname">explainParams</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.PCA.explainParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns the documentation of all params with their optionally
default values and user-supplied values.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.PCA.extractParamMap">
<code class="descname">extractParamMap</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.PCA.extractParamMap" title="Permalink to this definition">¶</a></dt>
<dd><p>Extracts the embedded default param values and user-supplied
values, and then merges them with extra values from input into
a flat param map, where the latter value is used if there exist
conflicts, i.e., with ordering: default param values <
user-supplied values < extra.
:param extra: extra param values
:return: merged param map</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.PCA.fit">
<code class="descname">fit</code><span class="sig-paren">(</span><em>dataset</em>, <em>params=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.PCA.fit" title="Permalink to this definition">¶</a></dt>
<dd><p>Fits a model to the input dataset with optional parameters.</p>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple">
<li><strong>dataset</strong> – input dataset, which is an instance of
<a class="reference internal" href="pyspark.sql.html#pyspark.sql.DataFrame" title="pyspark.sql.DataFrame"><code class="xref py py-class docutils literal"><span class="pre">pyspark.sql.DataFrame</span></code></a></li>
<li><strong>params</strong> – an optional param map that overrides embedded
params. If a list/tuple of param maps is given,
this calls fit on each param map and returns a
list of models.</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body"><p class="first last">fitted model(s)</p>
</td>
</tr>
</tbody>
</table>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.PCA.getInputCol">
<code class="descname">getInputCol</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.PCA.getInputCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of inputCol or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.PCA.getK">
<code class="descname">getK</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/feature.html#PCA.getK"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.feature.PCA.getK" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of k or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.PCA.getOrDefault">
<code class="descname">getOrDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.PCA.getOrDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of a param in the user-supplied param map or its
default value. Raises an error if neither is set.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.PCA.getOutputCol">
<code class="descname">getOutputCol</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.PCA.getOutputCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of outputCol or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.PCA.getParam">
<code class="descname">getParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.PCA.getParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets a param by its name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.PCA.hasDefault">
<code class="descname">hasDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.PCA.hasDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param has a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.PCA.hasParam">
<code class="descname">hasParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.PCA.hasParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Tests whether this instance contains a param with a given
(string) name.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.feature.PCA.inputCol">
<code class="descname">inputCol</code><em class="property"> = Param(parent='undefined', name='inputCol', doc='input column name')</em><a class="headerlink" href="#pyspark.ml.feature.PCA.inputCol" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.PCA.isDefined">
<code class="descname">isDefined</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.PCA.isDefined" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user or has
a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.PCA.isSet">
<code class="descname">isSet</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.PCA.isSet" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.feature.PCA.k">
<code class="descname">k</code><em class="property"> = Param(parent='undefined', name='k', doc='the number of principal components')</em><a class="headerlink" href="#pyspark.ml.feature.PCA.k" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.feature.PCA.outputCol">
<code class="descname">outputCol</code><em class="property"> = Param(parent='undefined', name='outputCol', doc='output column name')</em><a class="headerlink" href="#pyspark.ml.feature.PCA.outputCol" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.feature.PCA.params">
<code class="descname">params</code><a class="headerlink" href="#pyspark.ml.feature.PCA.params" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns all params ordered by name. The default implementation
uses <code class="xref py py-func docutils literal"><span class="pre">dir()</span></code> to get all attributes of type
<code class="xref py py-class docutils literal"><span class="pre">Param</span></code>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.PCA.setInputCol">
<code class="descname">setInputCol</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.PCA.setInputCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.feature.PCA.inputCol" title="pyspark.ml.feature.PCA.inputCol"><code class="xref py py-attr docutils literal"><span class="pre">inputCol</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.PCA.setK">
<code class="descname">setK</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/feature.html#PCA.setK"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.feature.PCA.setK" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.feature.PCA.k" title="pyspark.ml.feature.PCA.k"><code class="xref py py-attr docutils literal"><span class="pre">k</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.PCA.setOutputCol">
<code class="descname">setOutputCol</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.PCA.setOutputCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.feature.PCA.outputCol" title="pyspark.ml.feature.PCA.outputCol"><code class="xref py py-attr docutils literal"><span class="pre">outputCol</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.PCA.setParams">
<code class="descname">setParams</code><span class="sig-paren">(</span><em>self</em>, <em>k=None</em>, <em>inputCol=None</em>, <em>outputCol=None</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/feature.html#PCA.setParams"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.feature.PCA.setParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Set params for this PCA.</p>
</dd></dl>
</dd></dl>
<dl class="class">
<dt id="pyspark.ml.feature.PCAModel">
<em class="property">class </em><code class="descclassname">pyspark.ml.feature.</code><code class="descname">PCAModel</code><span class="sig-paren">(</span><em>java_model</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/feature.html#PCAModel"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.feature.PCAModel" title="Permalink to this definition">¶</a></dt>
<dd><p>Model fitted by PCA.</p>
<dl class="method">
<dt id="pyspark.ml.feature.PCAModel.copy">
<code class="descname">copy</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.PCAModel.copy" title="Permalink to this definition">¶</a></dt>
<dd><p>Creates a copy of this instance with the same uid and some
extra params. This implementation first calls Params.copy and
then make a copy of the companion Java model with extra params.
So both the Python wrapper and the Java model get copied.
:param extra: Extra parameters to copy to the new instance
:return: Copy of this instance</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.PCAModel.explainParam">
<code class="descname">explainParam</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.PCAModel.explainParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Explains a single param and returns its name, doc, and optional
default value and user-supplied value in a string.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.PCAModel.explainParams">
<code class="descname">explainParams</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.PCAModel.explainParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns the documentation of all params with their optionally
default values and user-supplied values.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.PCAModel.extractParamMap">
<code class="descname">extractParamMap</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.PCAModel.extractParamMap" title="Permalink to this definition">¶</a></dt>
<dd><p>Extracts the embedded default param values and user-supplied
values, and then merges them with extra values from input into
a flat param map, where the latter value is used if there exist
conflicts, i.e., with ordering: default param values <
user-supplied values < extra.
:param extra: extra param values
:return: merged param map</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.PCAModel.getOrDefault">
<code class="descname">getOrDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.PCAModel.getOrDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of a param in the user-supplied param map or its
default value. Raises an error if neither is set.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.PCAModel.getParam">
<code class="descname">getParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.PCAModel.getParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets a param by its name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.PCAModel.hasDefault">
<code class="descname">hasDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.PCAModel.hasDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param has a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.PCAModel.hasParam">
<code class="descname">hasParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.PCAModel.hasParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Tests whether this instance contains a param with a given
(string) name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.PCAModel.isDefined">
<code class="descname">isDefined</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.PCAModel.isDefined" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user or has
a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.PCAModel.isSet">
<code class="descname">isSet</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.PCAModel.isSet" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.feature.PCAModel.params">
<code class="descname">params</code><a class="headerlink" href="#pyspark.ml.feature.PCAModel.params" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns all params ordered by name. The default implementation
uses <code class="xref py py-func docutils literal"><span class="pre">dir()</span></code> to get all attributes of type
<code class="xref py py-class docutils literal"><span class="pre">Param</span></code>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.PCAModel.transform">
<code class="descname">transform</code><span class="sig-paren">(</span><em>dataset</em>, <em>params=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.PCAModel.transform" title="Permalink to this definition">¶</a></dt>
<dd><p>Transforms the input dataset with optional parameters.</p>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple">
<li><strong>dataset</strong> – input dataset, which is an instance of
<a class="reference internal" href="pyspark.sql.html#pyspark.sql.DataFrame" title="pyspark.sql.DataFrame"><code class="xref py py-class docutils literal"><span class="pre">pyspark.sql.DataFrame</span></code></a></li>
<li><strong>params</strong> – an optional param map that overrides embedded
params.</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body"><p class="first last">transformed dataset</p>
</td>
</tr>
</tbody>
</table>
</dd></dl>
</dd></dl>
<dl class="class">
<dt id="pyspark.ml.feature.RFormula">
<em class="property">class </em><code class="descclassname">pyspark.ml.feature.</code><code class="descname">RFormula</code><span class="sig-paren">(</span><em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/feature.html#RFormula"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.feature.RFormula" title="Permalink to this definition">¶</a></dt>
<dd><div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last">Experimental</p>
</div>
<p>Implements the transforms required for fitting a dataset against an
R model formula. Currently we support a limited subset of the R
operators, including ‘~’, ‘+’, ‘-‘, and ‘.’. Also see the R formula
docs:
<a class="reference external" href="http://stat.ethz.ch/R-manual/R-patched/library/stats/html/formula.html">http://stat.ethz.ch/R-manual/R-patched/library/stats/html/formula.html</a></p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="n">df</span> <span class="o">=</span> <span class="n">sqlContext</span><span class="o">.</span><span class="n">createDataFrame</span><span class="p">([</span>
<span class="gp">... </span> <span class="p">(</span><span class="mf">1.0</span><span class="p">,</span> <span class="mf">1.0</span><span class="p">,</span> <span class="s">"a"</span><span class="p">),</span>
<span class="gp">... </span> <span class="p">(</span><span class="mf">0.0</span><span class="p">,</span> <span class="mf">2.0</span><span class="p">,</span> <span class="s">"b"</span><span class="p">),</span>
<span class="gp">... </span> <span class="p">(</span><span class="mf">0.0</span><span class="p">,</span> <span class="mf">0.0</span><span class="p">,</span> <span class="s">"a"</span><span class="p">)</span>
<span class="gp">... </span><span class="p">],</span> <span class="p">[</span><span class="s">"y"</span><span class="p">,</span> <span class="s">"x"</span><span class="p">,</span> <span class="s">"s"</span><span class="p">])</span>
<span class="gp">>>> </span><span class="n">rf</span> <span class="o">=</span> <span class="n">RFormula</span><span class="p">(</span><span class="n">formula</span><span class="o">=</span><span class="s">"y ~ x + s"</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">rf</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">df</span><span class="p">)</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">df</span><span class="p">)</span><span class="o">.</span><span class="n">show</span><span class="p">()</span>
<span class="go">+---+---+---+---------+-----+</span>
<span class="go">| y| x| s| features|label|</span>
<span class="go">+---+---+---+---------+-----+</span>
<span class="go">|1.0|1.0| a|[1.0,1.0]| 1.0|</span>
<span class="go">|0.0|2.0| b|[2.0,0.0]| 0.0|</span>
<span class="go">|0.0|0.0| a|[0.0,1.0]| 0.0|</span>
<span class="go">+---+---+---+---------+-----+</span>
<span class="gp">...</span>
<span class="gp">>>> </span><span class="n">rf</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">df</span><span class="p">,</span> <span class="p">{</span><span class="n">rf</span><span class="o">.</span><span class="n">formula</span><span class="p">:</span> <span class="s">"y ~ . - s"</span><span class="p">})</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">df</span><span class="p">)</span><span class="o">.</span><span class="n">show</span><span class="p">()</span>
<span class="go">+---+---+---+--------+-----+</span>
<span class="go">| y| x| s|features|label|</span>
<span class="go">+---+---+---+--------+-----+</span>
<span class="go">|1.0|1.0| a| [1.0]| 1.0|</span>
<span class="go">|0.0|2.0| b| [2.0]| 0.0|</span>
<span class="go">|0.0|0.0| a| [0.0]| 0.0|</span>
<span class="go">+---+---+---+--------+-----+</span>
<span class="gp">...</span>
</pre></div>
</div>
<dl class="method">
<dt id="pyspark.ml.feature.RFormula.copy">
<code class="descname">copy</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.RFormula.copy" title="Permalink to this definition">¶</a></dt>
<dd><p>Creates a copy of this instance with the same uid and some
extra params. The default implementation creates a
shallow copy using <code class="xref py py-func docutils literal"><span class="pre">copy.copy()</span></code>, and then copies the
embedded and extra parameters over and returns the copy.
Subclasses should override this method if the default approach
is not sufficient.
:param extra: Extra parameters to copy to the new instance
:return: Copy of this instance</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.RFormula.explainParam">
<code class="descname">explainParam</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.RFormula.explainParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Explains a single param and returns its name, doc, and optional
default value and user-supplied value in a string.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.RFormula.explainParams">
<code class="descname">explainParams</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.RFormula.explainParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns the documentation of all params with their optionally
default values and user-supplied values.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.RFormula.extractParamMap">
<code class="descname">extractParamMap</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.RFormula.extractParamMap" title="Permalink to this definition">¶</a></dt>
<dd><p>Extracts the embedded default param values and user-supplied
values, and then merges them with extra values from input into
a flat param map, where the latter value is used if there exist
conflicts, i.e., with ordering: default param values <
user-supplied values < extra.
:param extra: extra param values
:return: merged param map</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.feature.RFormula.featuresCol">
<code class="descname">featuresCol</code><em class="property"> = Param(parent='undefined', name='featuresCol', doc='features column name')</em><a class="headerlink" href="#pyspark.ml.feature.RFormula.featuresCol" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.RFormula.fit">
<code class="descname">fit</code><span class="sig-paren">(</span><em>dataset</em>, <em>params=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.RFormula.fit" title="Permalink to this definition">¶</a></dt>
<dd><p>Fits a model to the input dataset with optional parameters.</p>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple">
<li><strong>dataset</strong> – input dataset, which is an instance of
<a class="reference internal" href="pyspark.sql.html#pyspark.sql.DataFrame" title="pyspark.sql.DataFrame"><code class="xref py py-class docutils literal"><span class="pre">pyspark.sql.DataFrame</span></code></a></li>
<li><strong>params</strong> – an optional param map that overrides embedded
params. If a list/tuple of param maps is given,
this calls fit on each param map and returns a
list of models.</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body"><p class="first last">fitted model(s)</p>
</td>
</tr>
</tbody>
</table>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.feature.RFormula.formula">
<code class="descname">formula</code><em class="property"> = Param(parent='undefined', name='formula', doc='R model formula')</em><a class="headerlink" href="#pyspark.ml.feature.RFormula.formula" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.RFormula.getFeaturesCol">
<code class="descname">getFeaturesCol</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.RFormula.getFeaturesCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of featuresCol or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.RFormula.getFormula">
<code class="descname">getFormula</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/feature.html#RFormula.getFormula"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.feature.RFormula.getFormula" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of <a class="reference internal" href="#pyspark.ml.feature.RFormula.formula" title="pyspark.ml.feature.RFormula.formula"><code class="xref py py-attr docutils literal"><span class="pre">formula</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.RFormula.getLabelCol">
<code class="descname">getLabelCol</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.RFormula.getLabelCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of labelCol or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.RFormula.getOrDefault">
<code class="descname">getOrDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.RFormula.getOrDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of a param in the user-supplied param map or its
default value. Raises an error if neither is set.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.RFormula.getParam">
<code class="descname">getParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.RFormula.getParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets a param by its name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.RFormula.hasDefault">
<code class="descname">hasDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.RFormula.hasDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param has a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.RFormula.hasParam">
<code class="descname">hasParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.RFormula.hasParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Tests whether this instance contains a param with a given
(string) name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.RFormula.isDefined">
<code class="descname">isDefined</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.RFormula.isDefined" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user or has
a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.RFormula.isSet">
<code class="descname">isSet</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.RFormula.isSet" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.feature.RFormula.labelCol">
<code class="descname">labelCol</code><em class="property"> = Param(parent='undefined', name='labelCol', doc='label column name')</em><a class="headerlink" href="#pyspark.ml.feature.RFormula.labelCol" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.feature.RFormula.params">
<code class="descname">params</code><a class="headerlink" href="#pyspark.ml.feature.RFormula.params" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns all params ordered by name. The default implementation
uses <code class="xref py py-func docutils literal"><span class="pre">dir()</span></code> to get all attributes of type
<code class="xref py py-class docutils literal"><span class="pre">Param</span></code>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.RFormula.setFeaturesCol">
<code class="descname">setFeaturesCol</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.RFormula.setFeaturesCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.feature.RFormula.featuresCol" title="pyspark.ml.feature.RFormula.featuresCol"><code class="xref py py-attr docutils literal"><span class="pre">featuresCol</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.RFormula.setFormula">
<code class="descname">setFormula</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/feature.html#RFormula.setFormula"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.feature.RFormula.setFormula" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.feature.RFormula.formula" title="pyspark.ml.feature.RFormula.formula"><code class="xref py py-attr docutils literal"><span class="pre">formula</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.RFormula.setLabelCol">
<code class="descname">setLabelCol</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.RFormula.setLabelCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.feature.RFormula.labelCol" title="pyspark.ml.feature.RFormula.labelCol"><code class="xref py py-attr docutils literal"><span class="pre">labelCol</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.RFormula.setParams">
<code class="descname">setParams</code><span class="sig-paren">(</span><em>self</em>, <em>formula=None</em>, <em>featuresCol="features"</em>, <em>labelCol="label"</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/feature.html#RFormula.setParams"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.feature.RFormula.setParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets params for RFormula.</p>
</dd></dl>
</dd></dl>
<dl class="class">
<dt id="pyspark.ml.feature.RFormulaModel">
<em class="property">class </em><code class="descclassname">pyspark.ml.feature.</code><code class="descname">RFormulaModel</code><span class="sig-paren">(</span><em>java_model</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/feature.html#RFormulaModel"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.feature.RFormulaModel" title="Permalink to this definition">¶</a></dt>
<dd><p>Model fitted by <a class="reference internal" href="#pyspark.ml.feature.RFormula" title="pyspark.ml.feature.RFormula"><code class="xref py py-class docutils literal"><span class="pre">RFormula</span></code></a>.</p>
<dl class="method">
<dt id="pyspark.ml.feature.RFormulaModel.copy">
<code class="descname">copy</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.RFormulaModel.copy" title="Permalink to this definition">¶</a></dt>
<dd><p>Creates a copy of this instance with the same uid and some
extra params. This implementation first calls Params.copy and
then make a copy of the companion Java model with extra params.
So both the Python wrapper and the Java model get copied.
:param extra: Extra parameters to copy to the new instance
:return: Copy of this instance</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.RFormulaModel.explainParam">
<code class="descname">explainParam</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.RFormulaModel.explainParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Explains a single param and returns its name, doc, and optional
default value and user-supplied value in a string.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.RFormulaModel.explainParams">
<code class="descname">explainParams</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.RFormulaModel.explainParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns the documentation of all params with their optionally
default values and user-supplied values.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.RFormulaModel.extractParamMap">
<code class="descname">extractParamMap</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.RFormulaModel.extractParamMap" title="Permalink to this definition">¶</a></dt>
<dd><p>Extracts the embedded default param values and user-supplied
values, and then merges them with extra values from input into
a flat param map, where the latter value is used if there exist
conflicts, i.e., with ordering: default param values <
user-supplied values < extra.
:param extra: extra param values
:return: merged param map</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.RFormulaModel.getOrDefault">
<code class="descname">getOrDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.RFormulaModel.getOrDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of a param in the user-supplied param map or its
default value. Raises an error if neither is set.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.RFormulaModel.getParam">
<code class="descname">getParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.RFormulaModel.getParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets a param by its name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.RFormulaModel.hasDefault">
<code class="descname">hasDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.RFormulaModel.hasDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param has a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.RFormulaModel.hasParam">
<code class="descname">hasParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.RFormulaModel.hasParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Tests whether this instance contains a param with a given
(string) name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.RFormulaModel.isDefined">
<code class="descname">isDefined</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.RFormulaModel.isDefined" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user or has
a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.RFormulaModel.isSet">
<code class="descname">isSet</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.RFormulaModel.isSet" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.feature.RFormulaModel.params">
<code class="descname">params</code><a class="headerlink" href="#pyspark.ml.feature.RFormulaModel.params" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns all params ordered by name. The default implementation
uses <code class="xref py py-func docutils literal"><span class="pre">dir()</span></code> to get all attributes of type
<code class="xref py py-class docutils literal"><span class="pre">Param</span></code>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.feature.RFormulaModel.transform">
<code class="descname">transform</code><span class="sig-paren">(</span><em>dataset</em>, <em>params=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.feature.RFormulaModel.transform" title="Permalink to this definition">¶</a></dt>
<dd><p>Transforms the input dataset with optional parameters.</p>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple">
<li><strong>dataset</strong> – input dataset, which is an instance of
<a class="reference internal" href="pyspark.sql.html#pyspark.sql.DataFrame" title="pyspark.sql.DataFrame"><code class="xref py py-class docutils literal"><span class="pre">pyspark.sql.DataFrame</span></code></a></li>
<li><strong>params</strong> – an optional param map that overrides embedded
params.</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body"><p class="first last">transformed dataset</p>
</td>
</tr>
</tbody>
</table>
</dd></dl>
</dd></dl>
</div>
<div class="section" id="module-pyspark.ml.classification">
<span id="pyspark-ml-classification-module"></span><h2>pyspark.ml.classification module<a class="headerlink" href="#module-pyspark.ml.classification" title="Permalink to this headline">¶</a></h2>
<dl class="class">
<dt id="pyspark.ml.classification.LogisticRegression">
<em class="property">class </em><code class="descclassname">pyspark.ml.classification.</code><code class="descname">LogisticRegression</code><span class="sig-paren">(</span><em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/classification.html#LogisticRegression"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.classification.LogisticRegression" title="Permalink to this definition">¶</a></dt>
<dd><p>Logistic regression.
Currently, this class only supports binary classification.</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="kn">from</span> <span class="nn">pyspark.sql</span> <span class="kn">import</span> <span class="n">Row</span>
<span class="gp">>>> </span><span class="kn">from</span> <span class="nn">pyspark.mllib.linalg</span> <span class="kn">import</span> <span class="n">Vectors</span>
<span class="gp">>>> </span><span class="n">df</span> <span class="o">=</span> <span class="n">sc</span><span class="o">.</span><span class="n">parallelize</span><span class="p">([</span>
<span class="gp">... </span> <span class="n">Row</span><span class="p">(</span><span class="n">label</span><span class="o">=</span><span class="mf">1.0</span><span class="p">,</span> <span class="n">features</span><span class="o">=</span><span class="n">Vectors</span><span class="o">.</span><span class="n">dense</span><span class="p">(</span><span class="mf">1.0</span><span class="p">)),</span>
<span class="gp">... </span> <span class="n">Row</span><span class="p">(</span><span class="n">label</span><span class="o">=</span><span class="mf">0.0</span><span class="p">,</span> <span class="n">features</span><span class="o">=</span><span class="n">Vectors</span><span class="o">.</span><span class="n">sparse</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="p">[],</span> <span class="p">[]))])</span><span class="o">.</span><span class="n">toDF</span><span class="p">()</span>
<span class="gp">>>> </span><span class="n">lr</span> <span class="o">=</span> <span class="n">LogisticRegression</span><span class="p">(</span><span class="n">maxIter</span><span class="o">=</span><span class="mi">5</span><span class="p">,</span> <span class="n">regParam</span><span class="o">=</span><span class="mf">0.01</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">model</span> <span class="o">=</span> <span class="n">lr</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">df</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">model</span><span class="o">.</span><span class="n">weights</span>
<span class="go">DenseVector([5.5...])</span>
<span class="gp">>>> </span><span class="n">model</span><span class="o">.</span><span class="n">intercept</span>
<span class="go">-2.68...</span>
<span class="gp">>>> </span><span class="n">test0</span> <span class="o">=</span> <span class="n">sc</span><span class="o">.</span><span class="n">parallelize</span><span class="p">([</span><span class="n">Row</span><span class="p">(</span><span class="n">features</span><span class="o">=</span><span class="n">Vectors</span><span class="o">.</span><span class="n">dense</span><span class="p">(</span><span class="o">-</span><span class="mf">1.0</span><span class="p">))])</span><span class="o">.</span><span class="n">toDF</span><span class="p">()</span>
<span class="gp">>>> </span><span class="n">result</span> <span class="o">=</span> <span class="n">model</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">test0</span><span class="p">)</span><span class="o">.</span><span class="n">head</span><span class="p">()</span>
<span class="gp">>>> </span><span class="n">result</span><span class="o">.</span><span class="n">prediction</span>
<span class="go">0.0</span>
<span class="gp">>>> </span><span class="n">result</span><span class="o">.</span><span class="n">probability</span>
<span class="go">DenseVector([0.99..., 0.00...])</span>
<span class="gp">>>> </span><span class="n">result</span><span class="o">.</span><span class="n">rawPrediction</span>
<span class="go">DenseVector([8.22..., -8.22...])</span>
<span class="gp">>>> </span><span class="n">test1</span> <span class="o">=</span> <span class="n">sc</span><span class="o">.</span><span class="n">parallelize</span><span class="p">([</span><span class="n">Row</span><span class="p">(</span><span class="n">features</span><span class="o">=</span><span class="n">Vectors</span><span class="o">.</span><span class="n">sparse</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="p">[</span><span class="mf">1.0</span><span class="p">]))])</span><span class="o">.</span><span class="n">toDF</span><span class="p">()</span>
<span class="gp">>>> </span><span class="n">model</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">test1</span><span class="p">)</span><span class="o">.</span><span class="n">head</span><span class="p">()</span><span class="o">.</span><span class="n">prediction</span>
<span class="go">1.0</span>
<span class="gp">>>> </span><span class="n">lr</span><span class="o">.</span><span class="n">setParams</span><span class="p">(</span><span class="s">"vector"</span><span class="p">)</span>
<span class="gt">Traceback (most recent call last):</span>
<span class="o">...</span>
<span class="gr">TypeError</span>: <span class="n">Method setParams forces keyword arguments.</span>
</pre></div>
</div>
<dl class="method">
<dt id="pyspark.ml.classification.LogisticRegression.copy">
<code class="descname">copy</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.LogisticRegression.copy" title="Permalink to this definition">¶</a></dt>
<dd><p>Creates a copy of this instance with the same uid and some
extra params. The default implementation creates a
shallow copy using <code class="xref py py-func docutils literal"><span class="pre">copy.copy()</span></code>, and then copies the
embedded and extra parameters over and returns the copy.
Subclasses should override this method if the default approach
is not sufficient.
:param extra: Extra parameters to copy to the new instance
:return: Copy of this instance</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.classification.LogisticRegression.elasticNetParam">
<code class="descname">elasticNetParam</code><em class="property"> = Param(parent='undefined', name='elasticNetParam', doc='the ElasticNet mixing parameter, in range [0, 1]. For alpha = 0, the penalty is an L2 penalty. For alpha = 1, it is an L1 penalty.')</em><a class="headerlink" href="#pyspark.ml.classification.LogisticRegression.elasticNetParam" title="Permalink to this definition">¶</a></dt>
<dd><p>param for the ElasticNet mixing parameter, in range [0, 1]. For alpha = 0, the penalty</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.LogisticRegression.explainParam">
<code class="descname">explainParam</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.LogisticRegression.explainParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Explains a single param and returns its name, doc, and optional
default value and user-supplied value in a string.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.LogisticRegression.explainParams">
<code class="descname">explainParams</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.LogisticRegression.explainParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns the documentation of all params with their optionally
default values and user-supplied values.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.LogisticRegression.extractParamMap">
<code class="descname">extractParamMap</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.LogisticRegression.extractParamMap" title="Permalink to this definition">¶</a></dt>
<dd><p>Extracts the embedded default param values and user-supplied
values, and then merges them with extra values from input into
a flat param map, where the latter value is used if there exist
conflicts, i.e., with ordering: default param values <
user-supplied values < extra.
:param extra: extra param values
:return: merged param map</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.classification.LogisticRegression.featuresCol">
<code class="descname">featuresCol</code><em class="property"> = Param(parent='undefined', name='featuresCol', doc='features column name')</em><a class="headerlink" href="#pyspark.ml.classification.LogisticRegression.featuresCol" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.LogisticRegression.fit">
<code class="descname">fit</code><span class="sig-paren">(</span><em>dataset</em>, <em>params=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.LogisticRegression.fit" title="Permalink to this definition">¶</a></dt>
<dd><p>Fits a model to the input dataset with optional parameters.</p>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple">
<li><strong>dataset</strong> – input dataset, which is an instance of
<a class="reference internal" href="pyspark.sql.html#pyspark.sql.DataFrame" title="pyspark.sql.DataFrame"><code class="xref py py-class docutils literal"><span class="pre">pyspark.sql.DataFrame</span></code></a></li>
<li><strong>params</strong> – an optional param map that overrides embedded
params. If a list/tuple of param maps is given,
this calls fit on each param map and returns a
list of models.</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body"><p class="first last">fitted model(s)</p>
</td>
</tr>
</tbody>
</table>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.classification.LogisticRegression.fitIntercept">
<code class="descname">fitIntercept</code><em class="property"> = Param(parent='undefined', name='fitIntercept', doc='whether to fit an intercept term.')</em><a class="headerlink" href="#pyspark.ml.classification.LogisticRegression.fitIntercept" title="Permalink to this definition">¶</a></dt>
<dd><p>param for whether to fit an intercept term.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.LogisticRegression.getElasticNetParam">
<code class="descname">getElasticNetParam</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/classification.html#LogisticRegression.getElasticNetParam"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.classification.LogisticRegression.getElasticNetParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of elasticNetParam or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.LogisticRegression.getFeaturesCol">
<code class="descname">getFeaturesCol</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.LogisticRegression.getFeaturesCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of featuresCol or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.LogisticRegression.getFitIntercept">
<code class="descname">getFitIntercept</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/classification.html#LogisticRegression.getFitIntercept"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.classification.LogisticRegression.getFitIntercept" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of fitIntercept or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.LogisticRegression.getLabelCol">
<code class="descname">getLabelCol</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.LogisticRegression.getLabelCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of labelCol or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.LogisticRegression.getMaxIter">
<code class="descname">getMaxIter</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.LogisticRegression.getMaxIter" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of maxIter or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.LogisticRegression.getOrDefault">
<code class="descname">getOrDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.LogisticRegression.getOrDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of a param in the user-supplied param map or its
default value. Raises an error if neither is set.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.LogisticRegression.getParam">
<code class="descname">getParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.LogisticRegression.getParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets a param by its name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.LogisticRegression.getPredictionCol">
<code class="descname">getPredictionCol</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.LogisticRegression.getPredictionCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of predictionCol or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.LogisticRegression.getProbabilityCol">
<code class="descname">getProbabilityCol</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.LogisticRegression.getProbabilityCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of probabilityCol or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.LogisticRegression.getRawPredictionCol">
<code class="descname">getRawPredictionCol</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.LogisticRegression.getRawPredictionCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of rawPredictionCol or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.LogisticRegression.getRegParam">
<code class="descname">getRegParam</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.LogisticRegression.getRegParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of regParam or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.LogisticRegression.getThreshold">
<code class="descname">getThreshold</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/classification.html#LogisticRegression.getThreshold"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.classification.LogisticRegression.getThreshold" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of threshold or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.LogisticRegression.getThresholds">
<code class="descname">getThresholds</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/classification.html#LogisticRegression.getThresholds"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.classification.LogisticRegression.getThresholds" title="Permalink to this definition">¶</a></dt>
<dd><p>If <a class="reference internal" href="#pyspark.ml.classification.LogisticRegression.thresholds" title="pyspark.ml.classification.LogisticRegression.thresholds"><code class="xref py py-attr docutils literal"><span class="pre">thresholds</span></code></a> is set, return its value.
Otherwise, if <a class="reference internal" href="#pyspark.ml.classification.LogisticRegression.threshold" title="pyspark.ml.classification.LogisticRegression.threshold"><code class="xref py py-attr docutils literal"><span class="pre">threshold</span></code></a> is set, return the equivalent thresholds for binary
classification: (1-threshold, threshold).
If neither are set, throw an error.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.LogisticRegression.getTol">
<code class="descname">getTol</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.LogisticRegression.getTol" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of tol or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.LogisticRegression.hasDefault">
<code class="descname">hasDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.LogisticRegression.hasDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param has a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.LogisticRegression.hasParam">
<code class="descname">hasParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.LogisticRegression.hasParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Tests whether this instance contains a param with a given
(string) name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.LogisticRegression.isDefined">
<code class="descname">isDefined</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.LogisticRegression.isDefined" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user or has
a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.LogisticRegression.isSet">
<code class="descname">isSet</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.LogisticRegression.isSet" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.classification.LogisticRegression.labelCol">
<code class="descname">labelCol</code><em class="property"> = Param(parent='undefined', name='labelCol', doc='label column name')</em><a class="headerlink" href="#pyspark.ml.classification.LogisticRegression.labelCol" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.classification.LogisticRegression.maxIter">
<code class="descname">maxIter</code><em class="property"> = Param(parent='undefined', name='maxIter', doc='max number of iterations (>= 0)')</em><a class="headerlink" href="#pyspark.ml.classification.LogisticRegression.maxIter" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.classification.LogisticRegression.params">
<code class="descname">params</code><a class="headerlink" href="#pyspark.ml.classification.LogisticRegression.params" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns all params ordered by name. The default implementation
uses <code class="xref py py-func docutils literal"><span class="pre">dir()</span></code> to get all attributes of type
<code class="xref py py-class docutils literal"><span class="pre">Param</span></code>.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.classification.LogisticRegression.predictionCol">
<code class="descname">predictionCol</code><em class="property"> = Param(parent='undefined', name='predictionCol', doc='prediction column name')</em><a class="headerlink" href="#pyspark.ml.classification.LogisticRegression.predictionCol" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.classification.LogisticRegression.probabilityCol">
<code class="descname">probabilityCol</code><em class="property"> = Param(parent='undefined', name='probabilityCol', doc='Column name for predicted class conditional probabilities. Note: Not all models output well-calibrated probability estimates! These probabilities should be treated as confidences, not precise probabilities.')</em><a class="headerlink" href="#pyspark.ml.classification.LogisticRegression.probabilityCol" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.classification.LogisticRegression.rawPredictionCol">
<code class="descname">rawPredictionCol</code><em class="property"> = Param(parent='undefined', name='rawPredictionCol', doc='raw prediction (a.k.a. confidence) column name')</em><a class="headerlink" href="#pyspark.ml.classification.LogisticRegression.rawPredictionCol" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.classification.LogisticRegression.regParam">
<code class="descname">regParam</code><em class="property"> = Param(parent='undefined', name='regParam', doc='regularization parameter (>= 0)')</em><a class="headerlink" href="#pyspark.ml.classification.LogisticRegression.regParam" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.LogisticRegression.setElasticNetParam">
<code class="descname">setElasticNetParam</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/classification.html#LogisticRegression.setElasticNetParam"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.classification.LogisticRegression.setElasticNetParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.classification.LogisticRegression.elasticNetParam" title="pyspark.ml.classification.LogisticRegression.elasticNetParam"><code class="xref py py-attr docutils literal"><span class="pre">elasticNetParam</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.LogisticRegression.setFeaturesCol">
<code class="descname">setFeaturesCol</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.LogisticRegression.setFeaturesCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.classification.LogisticRegression.featuresCol" title="pyspark.ml.classification.LogisticRegression.featuresCol"><code class="xref py py-attr docutils literal"><span class="pre">featuresCol</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.LogisticRegression.setFitIntercept">
<code class="descname">setFitIntercept</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/classification.html#LogisticRegression.setFitIntercept"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.classification.LogisticRegression.setFitIntercept" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.classification.LogisticRegression.fitIntercept" title="pyspark.ml.classification.LogisticRegression.fitIntercept"><code class="xref py py-attr docutils literal"><span class="pre">fitIntercept</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.LogisticRegression.setLabelCol">
<code class="descname">setLabelCol</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.LogisticRegression.setLabelCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.classification.LogisticRegression.labelCol" title="pyspark.ml.classification.LogisticRegression.labelCol"><code class="xref py py-attr docutils literal"><span class="pre">labelCol</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.LogisticRegression.setMaxIter">
<code class="descname">setMaxIter</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.LogisticRegression.setMaxIter" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.classification.LogisticRegression.maxIter" title="pyspark.ml.classification.LogisticRegression.maxIter"><code class="xref py py-attr docutils literal"><span class="pre">maxIter</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.LogisticRegression.setParams">
<code class="descname">setParams</code><span class="sig-paren">(</span><em>self</em>, <em>featuresCol="features"</em>, <em>labelCol="label"</em>, <em>predictionCol="prediction"</em>, <em>maxIter=100</em>, <em>regParam=0.1</em>, <em>elasticNetParam=0.0</em>, <em>tol=1e-6</em>, <em>fitIntercept=True</em>, <em>threshold=0.5</em>, <em>thresholds=None</em>, <em>probabilityCol="probability"</em>, <em>rawPredictionCol="rawPrediction"</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/classification.html#LogisticRegression.setParams"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.classification.LogisticRegression.setParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets params for logistic regression.
If the threshold and thresholds Params are both set, they must be equivalent.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.LogisticRegression.setPredictionCol">
<code class="descname">setPredictionCol</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.LogisticRegression.setPredictionCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.classification.LogisticRegression.predictionCol" title="pyspark.ml.classification.LogisticRegression.predictionCol"><code class="xref py py-attr docutils literal"><span class="pre">predictionCol</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.LogisticRegression.setProbabilityCol">
<code class="descname">setProbabilityCol</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.LogisticRegression.setProbabilityCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.classification.LogisticRegression.probabilityCol" title="pyspark.ml.classification.LogisticRegression.probabilityCol"><code class="xref py py-attr docutils literal"><span class="pre">probabilityCol</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.LogisticRegression.setRawPredictionCol">
<code class="descname">setRawPredictionCol</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.LogisticRegression.setRawPredictionCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.classification.LogisticRegression.rawPredictionCol" title="pyspark.ml.classification.LogisticRegression.rawPredictionCol"><code class="xref py py-attr docutils literal"><span class="pre">rawPredictionCol</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.LogisticRegression.setRegParam">
<code class="descname">setRegParam</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.LogisticRegression.setRegParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.classification.LogisticRegression.regParam" title="pyspark.ml.classification.LogisticRegression.regParam"><code class="xref py py-attr docutils literal"><span class="pre">regParam</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.LogisticRegression.setThreshold">
<code class="descname">setThreshold</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/classification.html#LogisticRegression.setThreshold"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.classification.LogisticRegression.setThreshold" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.classification.LogisticRegression.threshold" title="pyspark.ml.classification.LogisticRegression.threshold"><code class="xref py py-attr docutils literal"><span class="pre">threshold</span></code></a>.
Clears value of <a class="reference internal" href="#pyspark.ml.classification.LogisticRegression.thresholds" title="pyspark.ml.classification.LogisticRegression.thresholds"><code class="xref py py-attr docutils literal"><span class="pre">thresholds</span></code></a> if it has been set.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.LogisticRegression.setThresholds">
<code class="descname">setThresholds</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/classification.html#LogisticRegression.setThresholds"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.classification.LogisticRegression.setThresholds" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.classification.LogisticRegression.thresholds" title="pyspark.ml.classification.LogisticRegression.thresholds"><code class="xref py py-attr docutils literal"><span class="pre">thresholds</span></code></a>.
Clears value of <a class="reference internal" href="#pyspark.ml.classification.LogisticRegression.threshold" title="pyspark.ml.classification.LogisticRegression.threshold"><code class="xref py py-attr docutils literal"><span class="pre">threshold</span></code></a> if it has been set.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.LogisticRegression.setTol">
<code class="descname">setTol</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.LogisticRegression.setTol" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.classification.LogisticRegression.tol" title="pyspark.ml.classification.LogisticRegression.tol"><code class="xref py py-attr docutils literal"><span class="pre">tol</span></code></a>.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.classification.LogisticRegression.threshold">
<code class="descname">threshold</code><em class="property"> = Param(parent='undefined', name='threshold', doc='Threshold in binary classification prediction, in range [0, 1]. If threshold and thresholds are both set, they must match.')</em><a class="headerlink" href="#pyspark.ml.classification.LogisticRegression.threshold" title="Permalink to this definition">¶</a></dt>
<dd><p>param for threshold in binary classification, in range [0, 1].</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.classification.LogisticRegression.thresholds">
<code class="descname">thresholds</code><em class="property"> = Param(parent='undefined', name='thresholds', doc="Thresholds in multi-class classification to adjust the probability of predicting each class. Array must have length equal to the number of classes, with values >= 0. The class with largest value p/t is predicted, where p is the original probability of that class and t is the class' threshold.")</em><a class="headerlink" href="#pyspark.ml.classification.LogisticRegression.thresholds" title="Permalink to this definition">¶</a></dt>
<dd><p>param for thresholds or cutoffs in binary or multiclass classification</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.classification.LogisticRegression.tol">
<code class="descname">tol</code><em class="property"> = Param(parent='undefined', name='tol', doc='the convergence tolerance for iterative algorithms')</em><a class="headerlink" href="#pyspark.ml.classification.LogisticRegression.tol" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
</dd></dl>
<dl class="class">
<dt id="pyspark.ml.classification.LogisticRegressionModel">
<em class="property">class </em><code class="descclassname">pyspark.ml.classification.</code><code class="descname">LogisticRegressionModel</code><span class="sig-paren">(</span><em>java_model</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/classification.html#LogisticRegressionModel"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.classification.LogisticRegressionModel" title="Permalink to this definition">¶</a></dt>
<dd><p>Model fitted by LogisticRegression.</p>
<dl class="method">
<dt id="pyspark.ml.classification.LogisticRegressionModel.copy">
<code class="descname">copy</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.LogisticRegressionModel.copy" title="Permalink to this definition">¶</a></dt>
<dd><p>Creates a copy of this instance with the same uid and some
extra params. This implementation first calls Params.copy and
then make a copy of the companion Java model with extra params.
So both the Python wrapper and the Java model get copied.
:param extra: Extra parameters to copy to the new instance
:return: Copy of this instance</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.LogisticRegressionModel.explainParam">
<code class="descname">explainParam</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.LogisticRegressionModel.explainParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Explains a single param and returns its name, doc, and optional
default value and user-supplied value in a string.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.LogisticRegressionModel.explainParams">
<code class="descname">explainParams</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.LogisticRegressionModel.explainParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns the documentation of all params with their optionally
default values and user-supplied values.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.LogisticRegressionModel.extractParamMap">
<code class="descname">extractParamMap</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.LogisticRegressionModel.extractParamMap" title="Permalink to this definition">¶</a></dt>
<dd><p>Extracts the embedded default param values and user-supplied
values, and then merges them with extra values from input into
a flat param map, where the latter value is used if there exist
conflicts, i.e., with ordering: default param values <
user-supplied values < extra.
:param extra: extra param values
:return: merged param map</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.LogisticRegressionModel.getOrDefault">
<code class="descname">getOrDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.LogisticRegressionModel.getOrDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of a param in the user-supplied param map or its
default value. Raises an error if neither is set.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.LogisticRegressionModel.getParam">
<code class="descname">getParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.LogisticRegressionModel.getParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets a param by its name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.LogisticRegressionModel.hasDefault">
<code class="descname">hasDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.LogisticRegressionModel.hasDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param has a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.LogisticRegressionModel.hasParam">
<code class="descname">hasParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.LogisticRegressionModel.hasParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Tests whether this instance contains a param with a given
(string) name.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.classification.LogisticRegressionModel.intercept">
<code class="descname">intercept</code><a class="headerlink" href="#pyspark.ml.classification.LogisticRegressionModel.intercept" title="Permalink to this definition">¶</a></dt>
<dd><p>Model intercept.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.LogisticRegressionModel.isDefined">
<code class="descname">isDefined</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.LogisticRegressionModel.isDefined" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user or has
a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.LogisticRegressionModel.isSet">
<code class="descname">isSet</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.LogisticRegressionModel.isSet" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.classification.LogisticRegressionModel.params">
<code class="descname">params</code><a class="headerlink" href="#pyspark.ml.classification.LogisticRegressionModel.params" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns all params ordered by name. The default implementation
uses <code class="xref py py-func docutils literal"><span class="pre">dir()</span></code> to get all attributes of type
<code class="xref py py-class docutils literal"><span class="pre">Param</span></code>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.LogisticRegressionModel.transform">
<code class="descname">transform</code><span class="sig-paren">(</span><em>dataset</em>, <em>params=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.LogisticRegressionModel.transform" title="Permalink to this definition">¶</a></dt>
<dd><p>Transforms the input dataset with optional parameters.</p>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple">
<li><strong>dataset</strong> – input dataset, which is an instance of
<a class="reference internal" href="pyspark.sql.html#pyspark.sql.DataFrame" title="pyspark.sql.DataFrame"><code class="xref py py-class docutils literal"><span class="pre">pyspark.sql.DataFrame</span></code></a></li>
<li><strong>params</strong> – an optional param map that overrides embedded
params.</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body"><p class="first last">transformed dataset</p>
</td>
</tr>
</tbody>
</table>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.classification.LogisticRegressionModel.weights">
<code class="descname">weights</code><a class="headerlink" href="#pyspark.ml.classification.LogisticRegressionModel.weights" title="Permalink to this definition">¶</a></dt>
<dd><p>Model weights.</p>
</dd></dl>
</dd></dl>
<dl class="class">
<dt id="pyspark.ml.classification.DecisionTreeClassifier">
<em class="property">class </em><code class="descclassname">pyspark.ml.classification.</code><code class="descname">DecisionTreeClassifier</code><span class="sig-paren">(</span><em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/classification.html#DecisionTreeClassifier"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.classification.DecisionTreeClassifier" title="Permalink to this definition">¶</a></dt>
<dd><p><cite>http://en.wikipedia.org/wiki/Decision_tree_learning Decision tree</cite>
learning algorithm for classification.
It supports both binary and multiclass labels, as well as both continuous and categorical
features.</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="kn">from</span> <span class="nn">pyspark.mllib.linalg</span> <span class="kn">import</span> <span class="n">Vectors</span>
<span class="gp">>>> </span><span class="kn">from</span> <span class="nn">pyspark.ml.feature</span> <span class="kn">import</span> <span class="n">StringIndexer</span>
<span class="gp">>>> </span><span class="n">df</span> <span class="o">=</span> <span class="n">sqlContext</span><span class="o">.</span><span class="n">createDataFrame</span><span class="p">([</span>
<span class="gp">... </span> <span class="p">(</span><span class="mf">1.0</span><span class="p">,</span> <span class="n">Vectors</span><span class="o">.</span><span class="n">dense</span><span class="p">(</span><span class="mf">1.0</span><span class="p">)),</span>
<span class="gp">... </span> <span class="p">(</span><span class="mf">0.0</span><span class="p">,</span> <span class="n">Vectors</span><span class="o">.</span><span class="n">sparse</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="p">[],</span> <span class="p">[]))],</span> <span class="p">[</span><span class="s">"label"</span><span class="p">,</span> <span class="s">"features"</span><span class="p">])</span>
<span class="gp">>>> </span><span class="n">stringIndexer</span> <span class="o">=</span> <span class="n">StringIndexer</span><span class="p">(</span><span class="n">inputCol</span><span class="o">=</span><span class="s">"label"</span><span class="p">,</span> <span class="n">outputCol</span><span class="o">=</span><span class="s">"indexed"</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">si_model</span> <span class="o">=</span> <span class="n">stringIndexer</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">df</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">td</span> <span class="o">=</span> <span class="n">si_model</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">df</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">dt</span> <span class="o">=</span> <span class="n">DecisionTreeClassifier</span><span class="p">(</span><span class="n">maxDepth</span><span class="o">=</span><span class="mi">2</span><span class="p">,</span> <span class="n">labelCol</span><span class="o">=</span><span class="s">"indexed"</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">model</span> <span class="o">=</span> <span class="n">dt</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">td</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">model</span><span class="o">.</span><span class="n">numNodes</span>
<span class="go">3</span>
<span class="gp">>>> </span><span class="n">model</span><span class="o">.</span><span class="n">depth</span>
<span class="go">1</span>
<span class="gp">>>> </span><span class="n">test0</span> <span class="o">=</span> <span class="n">sqlContext</span><span class="o">.</span><span class="n">createDataFrame</span><span class="p">([(</span><span class="n">Vectors</span><span class="o">.</span><span class="n">dense</span><span class="p">(</span><span class="o">-</span><span class="mf">1.0</span><span class="p">),)],</span> <span class="p">[</span><span class="s">"features"</span><span class="p">])</span>
<span class="gp">>>> </span><span class="n">result</span> <span class="o">=</span> <span class="n">model</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">test0</span><span class="p">)</span><span class="o">.</span><span class="n">head</span><span class="p">()</span>
<span class="gp">>>> </span><span class="n">result</span><span class="o">.</span><span class="n">prediction</span>
<span class="go">0.0</span>
<span class="gp">>>> </span><span class="n">result</span><span class="o">.</span><span class="n">probability</span>
<span class="go">DenseVector([1.0, 0.0])</span>
<span class="gp">>>> </span><span class="n">result</span><span class="o">.</span><span class="n">rawPrediction</span>
<span class="go">DenseVector([1.0, 0.0])</span>
<span class="gp">>>> </span><span class="n">test1</span> <span class="o">=</span> <span class="n">sqlContext</span><span class="o">.</span><span class="n">createDataFrame</span><span class="p">([(</span><span class="n">Vectors</span><span class="o">.</span><span class="n">sparse</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="p">[</span><span class="mf">1.0</span><span class="p">]),)],</span> <span class="p">[</span><span class="s">"features"</span><span class="p">])</span>
<span class="gp">>>> </span><span class="n">model</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">test1</span><span class="p">)</span><span class="o">.</span><span class="n">head</span><span class="p">()</span><span class="o">.</span><span class="n">prediction</span>
<span class="go">1.0</span>
</pre></div>
</div>
<dl class="attribute">
<dt id="pyspark.ml.classification.DecisionTreeClassifier.cacheNodeIds">
<code class="descname">cacheNodeIds</code><em class="property"> = Param(parent='undefined', name='cacheNodeIds', doc='If false, the algorithm will pass trees to executors to match instances with nodes. If true, the algorithm will cache node IDs for each instance. Caching can speed up training of deeper trees.')</em><a class="headerlink" href="#pyspark.ml.classification.DecisionTreeClassifier.cacheNodeIds" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.classification.DecisionTreeClassifier.checkpointInterval">
<code class="descname">checkpointInterval</code><em class="property"> = Param(parent='undefined', name='checkpointInterval', doc='checkpoint interval (>= 1)')</em><a class="headerlink" href="#pyspark.ml.classification.DecisionTreeClassifier.checkpointInterval" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.DecisionTreeClassifier.copy">
<code class="descname">copy</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.DecisionTreeClassifier.copy" title="Permalink to this definition">¶</a></dt>
<dd><p>Creates a copy of this instance with the same uid and some
extra params. The default implementation creates a
shallow copy using <code class="xref py py-func docutils literal"><span class="pre">copy.copy()</span></code>, and then copies the
embedded and extra parameters over and returns the copy.
Subclasses should override this method if the default approach
is not sufficient.
:param extra: Extra parameters to copy to the new instance
:return: Copy of this instance</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.DecisionTreeClassifier.explainParam">
<code class="descname">explainParam</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.DecisionTreeClassifier.explainParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Explains a single param and returns its name, doc, and optional
default value and user-supplied value in a string.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.DecisionTreeClassifier.explainParams">
<code class="descname">explainParams</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.DecisionTreeClassifier.explainParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns the documentation of all params with their optionally
default values and user-supplied values.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.DecisionTreeClassifier.extractParamMap">
<code class="descname">extractParamMap</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.DecisionTreeClassifier.extractParamMap" title="Permalink to this definition">¶</a></dt>
<dd><p>Extracts the embedded default param values and user-supplied
values, and then merges them with extra values from input into
a flat param map, where the latter value is used if there exist
conflicts, i.e., with ordering: default param values <
user-supplied values < extra.
:param extra: extra param values
:return: merged param map</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.classification.DecisionTreeClassifier.featuresCol">
<code class="descname">featuresCol</code><em class="property"> = Param(parent='undefined', name='featuresCol', doc='features column name')</em><a class="headerlink" href="#pyspark.ml.classification.DecisionTreeClassifier.featuresCol" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.DecisionTreeClassifier.fit">
<code class="descname">fit</code><span class="sig-paren">(</span><em>dataset</em>, <em>params=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.DecisionTreeClassifier.fit" title="Permalink to this definition">¶</a></dt>
<dd><p>Fits a model to the input dataset with optional parameters.</p>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple">
<li><strong>dataset</strong> – input dataset, which is an instance of
<a class="reference internal" href="pyspark.sql.html#pyspark.sql.DataFrame" title="pyspark.sql.DataFrame"><code class="xref py py-class docutils literal"><span class="pre">pyspark.sql.DataFrame</span></code></a></li>
<li><strong>params</strong> – an optional param map that overrides embedded
params. If a list/tuple of param maps is given,
this calls fit on each param map and returns a
list of models.</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body"><p class="first last">fitted model(s)</p>
</td>
</tr>
</tbody>
</table>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.DecisionTreeClassifier.getCacheNodeIds">
<code class="descname">getCacheNodeIds</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.DecisionTreeClassifier.getCacheNodeIds" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of cacheNodeIds or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.DecisionTreeClassifier.getCheckpointInterval">
<code class="descname">getCheckpointInterval</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.DecisionTreeClassifier.getCheckpointInterval" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of checkpointInterval or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.DecisionTreeClassifier.getFeaturesCol">
<code class="descname">getFeaturesCol</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.DecisionTreeClassifier.getFeaturesCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of featuresCol or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.DecisionTreeClassifier.getImpurity">
<code class="descname">getImpurity</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/classification.html#DecisionTreeClassifier.getImpurity"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.classification.DecisionTreeClassifier.getImpurity" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of impurity or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.DecisionTreeClassifier.getLabelCol">
<code class="descname">getLabelCol</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.DecisionTreeClassifier.getLabelCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of labelCol or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.DecisionTreeClassifier.getMaxBins">
<code class="descname">getMaxBins</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.DecisionTreeClassifier.getMaxBins" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of maxBins or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.DecisionTreeClassifier.getMaxDepth">
<code class="descname">getMaxDepth</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.DecisionTreeClassifier.getMaxDepth" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of maxDepth or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.DecisionTreeClassifier.getMaxMemoryInMB">
<code class="descname">getMaxMemoryInMB</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.DecisionTreeClassifier.getMaxMemoryInMB" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of maxMemoryInMB or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.DecisionTreeClassifier.getMinInfoGain">
<code class="descname">getMinInfoGain</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.DecisionTreeClassifier.getMinInfoGain" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of minInfoGain or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.DecisionTreeClassifier.getMinInstancesPerNode">
<code class="descname">getMinInstancesPerNode</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.DecisionTreeClassifier.getMinInstancesPerNode" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of minInstancesPerNode or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.DecisionTreeClassifier.getOrDefault">
<code class="descname">getOrDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.DecisionTreeClassifier.getOrDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of a param in the user-supplied param map or its
default value. Raises an error if neither is set.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.DecisionTreeClassifier.getParam">
<code class="descname">getParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.DecisionTreeClassifier.getParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets a param by its name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.DecisionTreeClassifier.getPredictionCol">
<code class="descname">getPredictionCol</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.DecisionTreeClassifier.getPredictionCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of predictionCol or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.DecisionTreeClassifier.getProbabilityCol">
<code class="descname">getProbabilityCol</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.DecisionTreeClassifier.getProbabilityCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of probabilityCol or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.DecisionTreeClassifier.getRawPredictionCol">
<code class="descname">getRawPredictionCol</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.DecisionTreeClassifier.getRawPredictionCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of rawPredictionCol or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.DecisionTreeClassifier.hasDefault">
<code class="descname">hasDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.DecisionTreeClassifier.hasDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param has a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.DecisionTreeClassifier.hasParam">
<code class="descname">hasParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.DecisionTreeClassifier.hasParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Tests whether this instance contains a param with a given
(string) name.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.classification.DecisionTreeClassifier.impurity">
<code class="descname">impurity</code><em class="property"> = Param(parent='undefined', name='impurity', doc='Criterion used for information gain calculation (case-insensitive). Supported options: entropy, gini')</em><a class="headerlink" href="#pyspark.ml.classification.DecisionTreeClassifier.impurity" title="Permalink to this definition">¶</a></dt>
<dd><p>param for Criterion used for information gain calculation (case-insensitive).</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.DecisionTreeClassifier.isDefined">
<code class="descname">isDefined</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.DecisionTreeClassifier.isDefined" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user or has
a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.DecisionTreeClassifier.isSet">
<code class="descname">isSet</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.DecisionTreeClassifier.isSet" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.classification.DecisionTreeClassifier.labelCol">
<code class="descname">labelCol</code><em class="property"> = Param(parent='undefined', name='labelCol', doc='label column name')</em><a class="headerlink" href="#pyspark.ml.classification.DecisionTreeClassifier.labelCol" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.classification.DecisionTreeClassifier.maxBins">
<code class="descname">maxBins</code><em class="property"> = Param(parent='undefined', name='maxBins', doc='Max number of bins for discretizing continuous features. Must be >=2 and >= number of categories for any categorical feature.')</em><a class="headerlink" href="#pyspark.ml.classification.DecisionTreeClassifier.maxBins" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.classification.DecisionTreeClassifier.maxDepth">
<code class="descname">maxDepth</code><em class="property"> = Param(parent='undefined', name='maxDepth', doc='Maximum depth of the tree. (>= 0) E.g., depth 0 means 1 leaf node; depth 1 means 1 internal node + 2 leaf nodes.')</em><a class="headerlink" href="#pyspark.ml.classification.DecisionTreeClassifier.maxDepth" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.classification.DecisionTreeClassifier.maxMemoryInMB">
<code class="descname">maxMemoryInMB</code><em class="property"> = Param(parent='undefined', name='maxMemoryInMB', doc='Maximum memory in MB allocated to histogram aggregation.')</em><a class="headerlink" href="#pyspark.ml.classification.DecisionTreeClassifier.maxMemoryInMB" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.classification.DecisionTreeClassifier.minInfoGain">
<code class="descname">minInfoGain</code><em class="property"> = Param(parent='undefined', name='minInfoGain', doc='Minimum information gain for a split to be considered at a tree node.')</em><a class="headerlink" href="#pyspark.ml.classification.DecisionTreeClassifier.minInfoGain" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.classification.DecisionTreeClassifier.minInstancesPerNode">
<code class="descname">minInstancesPerNode</code><em class="property"> = Param(parent='undefined', name='minInstancesPerNode', doc='Minimum number of instances each child must have after split. If a split causes the left or right child to have fewer than minInstancesPerNode, the split will be discarded as invalid. Should be >= 1.')</em><a class="headerlink" href="#pyspark.ml.classification.DecisionTreeClassifier.minInstancesPerNode" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.classification.DecisionTreeClassifier.params">
<code class="descname">params</code><a class="headerlink" href="#pyspark.ml.classification.DecisionTreeClassifier.params" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns all params ordered by name. The default implementation
uses <code class="xref py py-func docutils literal"><span class="pre">dir()</span></code> to get all attributes of type
<code class="xref py py-class docutils literal"><span class="pre">Param</span></code>.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.classification.DecisionTreeClassifier.predictionCol">
<code class="descname">predictionCol</code><em class="property"> = Param(parent='undefined', name='predictionCol', doc='prediction column name')</em><a class="headerlink" href="#pyspark.ml.classification.DecisionTreeClassifier.predictionCol" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.classification.DecisionTreeClassifier.probabilityCol">
<code class="descname">probabilityCol</code><em class="property"> = Param(parent='undefined', name='probabilityCol', doc='Column name for predicted class conditional probabilities. Note: Not all models output well-calibrated probability estimates! These probabilities should be treated as confidences, not precise probabilities.')</em><a class="headerlink" href="#pyspark.ml.classification.DecisionTreeClassifier.probabilityCol" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.classification.DecisionTreeClassifier.rawPredictionCol">
<code class="descname">rawPredictionCol</code><em class="property"> = Param(parent='undefined', name='rawPredictionCol', doc='raw prediction (a.k.a. confidence) column name')</em><a class="headerlink" href="#pyspark.ml.classification.DecisionTreeClassifier.rawPredictionCol" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.DecisionTreeClassifier.setCacheNodeIds">
<code class="descname">setCacheNodeIds</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.DecisionTreeClassifier.setCacheNodeIds" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.classification.DecisionTreeClassifier.cacheNodeIds" title="pyspark.ml.classification.DecisionTreeClassifier.cacheNodeIds"><code class="xref py py-attr docutils literal"><span class="pre">cacheNodeIds</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.DecisionTreeClassifier.setCheckpointInterval">
<code class="descname">setCheckpointInterval</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.DecisionTreeClassifier.setCheckpointInterval" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.classification.DecisionTreeClassifier.checkpointInterval" title="pyspark.ml.classification.DecisionTreeClassifier.checkpointInterval"><code class="xref py py-attr docutils literal"><span class="pre">checkpointInterval</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.DecisionTreeClassifier.setFeaturesCol">
<code class="descname">setFeaturesCol</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.DecisionTreeClassifier.setFeaturesCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.classification.DecisionTreeClassifier.featuresCol" title="pyspark.ml.classification.DecisionTreeClassifier.featuresCol"><code class="xref py py-attr docutils literal"><span class="pre">featuresCol</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.DecisionTreeClassifier.setImpurity">
<code class="descname">setImpurity</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/classification.html#DecisionTreeClassifier.setImpurity"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.classification.DecisionTreeClassifier.setImpurity" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.classification.DecisionTreeClassifier.impurity" title="pyspark.ml.classification.DecisionTreeClassifier.impurity"><code class="xref py py-attr docutils literal"><span class="pre">impurity</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.DecisionTreeClassifier.setLabelCol">
<code class="descname">setLabelCol</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.DecisionTreeClassifier.setLabelCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.classification.DecisionTreeClassifier.labelCol" title="pyspark.ml.classification.DecisionTreeClassifier.labelCol"><code class="xref py py-attr docutils literal"><span class="pre">labelCol</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.DecisionTreeClassifier.setMaxBins">
<code class="descname">setMaxBins</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.DecisionTreeClassifier.setMaxBins" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.classification.DecisionTreeClassifier.maxBins" title="pyspark.ml.classification.DecisionTreeClassifier.maxBins"><code class="xref py py-attr docutils literal"><span class="pre">maxBins</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.DecisionTreeClassifier.setMaxDepth">
<code class="descname">setMaxDepth</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.DecisionTreeClassifier.setMaxDepth" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.classification.DecisionTreeClassifier.maxDepth" title="pyspark.ml.classification.DecisionTreeClassifier.maxDepth"><code class="xref py py-attr docutils literal"><span class="pre">maxDepth</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.DecisionTreeClassifier.setMaxMemoryInMB">
<code class="descname">setMaxMemoryInMB</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.DecisionTreeClassifier.setMaxMemoryInMB" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.classification.DecisionTreeClassifier.maxMemoryInMB" title="pyspark.ml.classification.DecisionTreeClassifier.maxMemoryInMB"><code class="xref py py-attr docutils literal"><span class="pre">maxMemoryInMB</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.DecisionTreeClassifier.setMinInfoGain">
<code class="descname">setMinInfoGain</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.DecisionTreeClassifier.setMinInfoGain" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.classification.DecisionTreeClassifier.minInfoGain" title="pyspark.ml.classification.DecisionTreeClassifier.minInfoGain"><code class="xref py py-attr docutils literal"><span class="pre">minInfoGain</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.DecisionTreeClassifier.setMinInstancesPerNode">
<code class="descname">setMinInstancesPerNode</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.DecisionTreeClassifier.setMinInstancesPerNode" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.classification.DecisionTreeClassifier.minInstancesPerNode" title="pyspark.ml.classification.DecisionTreeClassifier.minInstancesPerNode"><code class="xref py py-attr docutils literal"><span class="pre">minInstancesPerNode</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.DecisionTreeClassifier.setParams">
<code class="descname">setParams</code><span class="sig-paren">(</span><em>self</em>, <em>featuresCol="features"</em>, <em>labelCol="label"</em>, <em>predictionCol="prediction"</em>, <em>probabilityCol="probability"</em>, <em>rawPredictionCol="rawPrediction"</em>, <em>maxDepth=5</em>, <em>maxBins=32</em>, <em>minInstancesPerNode=1</em>, <em>minInfoGain=0.0</em>, <em>maxMemoryInMB=256</em>, <em>cacheNodeIds=False</em>, <em>checkpointInterval=10</em>, <em>impurity="gini"</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/classification.html#DecisionTreeClassifier.setParams"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.classification.DecisionTreeClassifier.setParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets params for the DecisionTreeClassifier.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.DecisionTreeClassifier.setPredictionCol">
<code class="descname">setPredictionCol</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.DecisionTreeClassifier.setPredictionCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.classification.DecisionTreeClassifier.predictionCol" title="pyspark.ml.classification.DecisionTreeClassifier.predictionCol"><code class="xref py py-attr docutils literal"><span class="pre">predictionCol</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.DecisionTreeClassifier.setProbabilityCol">
<code class="descname">setProbabilityCol</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.DecisionTreeClassifier.setProbabilityCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.classification.DecisionTreeClassifier.probabilityCol" title="pyspark.ml.classification.DecisionTreeClassifier.probabilityCol"><code class="xref py py-attr docutils literal"><span class="pre">probabilityCol</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.DecisionTreeClassifier.setRawPredictionCol">
<code class="descname">setRawPredictionCol</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.DecisionTreeClassifier.setRawPredictionCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.classification.DecisionTreeClassifier.rawPredictionCol" title="pyspark.ml.classification.DecisionTreeClassifier.rawPredictionCol"><code class="xref py py-attr docutils literal"><span class="pre">rawPredictionCol</span></code></a>.</p>
</dd></dl>
</dd></dl>
<dl class="class">
<dt id="pyspark.ml.classification.DecisionTreeClassificationModel">
<em class="property">class </em><code class="descclassname">pyspark.ml.classification.</code><code class="descname">DecisionTreeClassificationModel</code><span class="sig-paren">(</span><em>java_model</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/classification.html#DecisionTreeClassificationModel"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.classification.DecisionTreeClassificationModel" title="Permalink to this definition">¶</a></dt>
<dd><p>Model fitted by DecisionTreeClassifier.</p>
<dl class="method">
<dt id="pyspark.ml.classification.DecisionTreeClassificationModel.copy">
<code class="descname">copy</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.DecisionTreeClassificationModel.copy" title="Permalink to this definition">¶</a></dt>
<dd><p>Creates a copy of this instance with the same uid and some
extra params. This implementation first calls Params.copy and
then make a copy of the companion Java model with extra params.
So both the Python wrapper and the Java model get copied.
:param extra: Extra parameters to copy to the new instance
:return: Copy of this instance</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.classification.DecisionTreeClassificationModel.depth">
<code class="descname">depth</code><a class="headerlink" href="#pyspark.ml.classification.DecisionTreeClassificationModel.depth" title="Permalink to this definition">¶</a></dt>
<dd><p>Return depth of the decision tree.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.DecisionTreeClassificationModel.explainParam">
<code class="descname">explainParam</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.DecisionTreeClassificationModel.explainParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Explains a single param and returns its name, doc, and optional
default value and user-supplied value in a string.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.DecisionTreeClassificationModel.explainParams">
<code class="descname">explainParams</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.DecisionTreeClassificationModel.explainParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns the documentation of all params with their optionally
default values and user-supplied values.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.DecisionTreeClassificationModel.extractParamMap">
<code class="descname">extractParamMap</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.DecisionTreeClassificationModel.extractParamMap" title="Permalink to this definition">¶</a></dt>
<dd><p>Extracts the embedded default param values and user-supplied
values, and then merges them with extra values from input into
a flat param map, where the latter value is used if there exist
conflicts, i.e., with ordering: default param values <
user-supplied values < extra.
:param extra: extra param values
:return: merged param map</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.DecisionTreeClassificationModel.getOrDefault">
<code class="descname">getOrDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.DecisionTreeClassificationModel.getOrDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of a param in the user-supplied param map or its
default value. Raises an error if neither is set.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.DecisionTreeClassificationModel.getParam">
<code class="descname">getParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.DecisionTreeClassificationModel.getParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets a param by its name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.DecisionTreeClassificationModel.hasDefault">
<code class="descname">hasDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.DecisionTreeClassificationModel.hasDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param has a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.DecisionTreeClassificationModel.hasParam">
<code class="descname">hasParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.DecisionTreeClassificationModel.hasParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Tests whether this instance contains a param with a given
(string) name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.DecisionTreeClassificationModel.isDefined">
<code class="descname">isDefined</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.DecisionTreeClassificationModel.isDefined" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user or has
a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.DecisionTreeClassificationModel.isSet">
<code class="descname">isSet</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.DecisionTreeClassificationModel.isSet" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.classification.DecisionTreeClassificationModel.numNodes">
<code class="descname">numNodes</code><a class="headerlink" href="#pyspark.ml.classification.DecisionTreeClassificationModel.numNodes" title="Permalink to this definition">¶</a></dt>
<dd><p>Return number of nodes of the decision tree.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.classification.DecisionTreeClassificationModel.params">
<code class="descname">params</code><a class="headerlink" href="#pyspark.ml.classification.DecisionTreeClassificationModel.params" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns all params ordered by name. The default implementation
uses <code class="xref py py-func docutils literal"><span class="pre">dir()</span></code> to get all attributes of type
<code class="xref py py-class docutils literal"><span class="pre">Param</span></code>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.DecisionTreeClassificationModel.transform">
<code class="descname">transform</code><span class="sig-paren">(</span><em>dataset</em>, <em>params=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.DecisionTreeClassificationModel.transform" title="Permalink to this definition">¶</a></dt>
<dd><p>Transforms the input dataset with optional parameters.</p>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple">
<li><strong>dataset</strong> – input dataset, which is an instance of
<a class="reference internal" href="pyspark.sql.html#pyspark.sql.DataFrame" title="pyspark.sql.DataFrame"><code class="xref py py-class docutils literal"><span class="pre">pyspark.sql.DataFrame</span></code></a></li>
<li><strong>params</strong> – an optional param map that overrides embedded
params.</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body"><p class="first last">transformed dataset</p>
</td>
</tr>
</tbody>
</table>
</dd></dl>
</dd></dl>
<dl class="class">
<dt id="pyspark.ml.classification.GBTClassifier">
<em class="property">class </em><code class="descclassname">pyspark.ml.classification.</code><code class="descname">GBTClassifier</code><span class="sig-paren">(</span><em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/classification.html#GBTClassifier"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.classification.GBTClassifier" title="Permalink to this definition">¶</a></dt>
<dd><p><cite>http://en.wikipedia.org/wiki/Gradient_boosting Gradient-Boosted Trees (GBTs)</cite>
learning algorithm for classification.
It supports binary labels, as well as both continuous and categorical features.
Note: Multiclass labels are not currently supported.</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="kn">from</span> <span class="nn">numpy</span> <span class="kn">import</span> <span class="n">allclose</span>
<span class="gp">>>> </span><span class="kn">from</span> <span class="nn">pyspark.mllib.linalg</span> <span class="kn">import</span> <span class="n">Vectors</span>
<span class="gp">>>> </span><span class="kn">from</span> <span class="nn">pyspark.ml.feature</span> <span class="kn">import</span> <span class="n">StringIndexer</span>
<span class="gp">>>> </span><span class="n">df</span> <span class="o">=</span> <span class="n">sqlContext</span><span class="o">.</span><span class="n">createDataFrame</span><span class="p">([</span>
<span class="gp">... </span> <span class="p">(</span><span class="mf">1.0</span><span class="p">,</span> <span class="n">Vectors</span><span class="o">.</span><span class="n">dense</span><span class="p">(</span><span class="mf">1.0</span><span class="p">)),</span>
<span class="gp">... </span> <span class="p">(</span><span class="mf">0.0</span><span class="p">,</span> <span class="n">Vectors</span><span class="o">.</span><span class="n">sparse</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="p">[],</span> <span class="p">[]))],</span> <span class="p">[</span><span class="s">"label"</span><span class="p">,</span> <span class="s">"features"</span><span class="p">])</span>
<span class="gp">>>> </span><span class="n">stringIndexer</span> <span class="o">=</span> <span class="n">StringIndexer</span><span class="p">(</span><span class="n">inputCol</span><span class="o">=</span><span class="s">"label"</span><span class="p">,</span> <span class="n">outputCol</span><span class="o">=</span><span class="s">"indexed"</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">si_model</span> <span class="o">=</span> <span class="n">stringIndexer</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">df</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">td</span> <span class="o">=</span> <span class="n">si_model</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">df</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">gbt</span> <span class="o">=</span> <span class="n">GBTClassifier</span><span class="p">(</span><span class="n">maxIter</span><span class="o">=</span><span class="mi">5</span><span class="p">,</span> <span class="n">maxDepth</span><span class="o">=</span><span class="mi">2</span><span class="p">,</span> <span class="n">labelCol</span><span class="o">=</span><span class="s">"indexed"</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">model</span> <span class="o">=</span> <span class="n">gbt</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">td</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">allclose</span><span class="p">(</span><span class="n">model</span><span class="o">.</span><span class="n">treeWeights</span><span class="p">,</span> <span class="p">[</span><span class="mf">1.0</span><span class="p">,</span> <span class="mf">0.1</span><span class="p">,</span> <span class="mf">0.1</span><span class="p">,</span> <span class="mf">0.1</span><span class="p">,</span> <span class="mf">0.1</span><span class="p">])</span>
<span class="go">True</span>
<span class="gp">>>> </span><span class="n">test0</span> <span class="o">=</span> <span class="n">sqlContext</span><span class="o">.</span><span class="n">createDataFrame</span><span class="p">([(</span><span class="n">Vectors</span><span class="o">.</span><span class="n">dense</span><span class="p">(</span><span class="o">-</span><span class="mf">1.0</span><span class="p">),)],</span> <span class="p">[</span><span class="s">"features"</span><span class="p">])</span>
<span class="gp">>>> </span><span class="n">model</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">test0</span><span class="p">)</span><span class="o">.</span><span class="n">head</span><span class="p">()</span><span class="o">.</span><span class="n">prediction</span>
<span class="go">0.0</span>
<span class="gp">>>> </span><span class="n">test1</span> <span class="o">=</span> <span class="n">sqlContext</span><span class="o">.</span><span class="n">createDataFrame</span><span class="p">([(</span><span class="n">Vectors</span><span class="o">.</span><span class="n">sparse</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="p">[</span><span class="mf">1.0</span><span class="p">]),)],</span> <span class="p">[</span><span class="s">"features"</span><span class="p">])</span>
<span class="gp">>>> </span><span class="n">model</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">test1</span><span class="p">)</span><span class="o">.</span><span class="n">head</span><span class="p">()</span><span class="o">.</span><span class="n">prediction</span>
<span class="go">1.0</span>
</pre></div>
</div>
<dl class="attribute">
<dt id="pyspark.ml.classification.GBTClassifier.cacheNodeIds">
<code class="descname">cacheNodeIds</code><em class="property"> = Param(parent='undefined', name='cacheNodeIds', doc='If false, the algorithm will pass trees to executors to match instances with nodes. If true, the algorithm will cache node IDs for each instance. Caching can speed up training of deeper trees.')</em><a class="headerlink" href="#pyspark.ml.classification.GBTClassifier.cacheNodeIds" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.classification.GBTClassifier.checkpointInterval">
<code class="descname">checkpointInterval</code><em class="property"> = Param(parent='undefined', name='checkpointInterval', doc='checkpoint interval (>= 1)')</em><a class="headerlink" href="#pyspark.ml.classification.GBTClassifier.checkpointInterval" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.GBTClassifier.copy">
<code class="descname">copy</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.GBTClassifier.copy" title="Permalink to this definition">¶</a></dt>
<dd><p>Creates a copy of this instance with the same uid and some
extra params. The default implementation creates a
shallow copy using <code class="xref py py-func docutils literal"><span class="pre">copy.copy()</span></code>, and then copies the
embedded and extra parameters over and returns the copy.
Subclasses should override this method if the default approach
is not sufficient.
:param extra: Extra parameters to copy to the new instance
:return: Copy of this instance</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.GBTClassifier.explainParam">
<code class="descname">explainParam</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.GBTClassifier.explainParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Explains a single param and returns its name, doc, and optional
default value and user-supplied value in a string.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.GBTClassifier.explainParams">
<code class="descname">explainParams</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.GBTClassifier.explainParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns the documentation of all params with their optionally
default values and user-supplied values.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.GBTClassifier.extractParamMap">
<code class="descname">extractParamMap</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.GBTClassifier.extractParamMap" title="Permalink to this definition">¶</a></dt>
<dd><p>Extracts the embedded default param values and user-supplied
values, and then merges them with extra values from input into
a flat param map, where the latter value is used if there exist
conflicts, i.e., with ordering: default param values <
user-supplied values < extra.
:param extra: extra param values
:return: merged param map</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.classification.GBTClassifier.featuresCol">
<code class="descname">featuresCol</code><em class="property"> = Param(parent='undefined', name='featuresCol', doc='features column name')</em><a class="headerlink" href="#pyspark.ml.classification.GBTClassifier.featuresCol" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.GBTClassifier.fit">
<code class="descname">fit</code><span class="sig-paren">(</span><em>dataset</em>, <em>params=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.GBTClassifier.fit" title="Permalink to this definition">¶</a></dt>
<dd><p>Fits a model to the input dataset with optional parameters.</p>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple">
<li><strong>dataset</strong> – input dataset, which is an instance of
<a class="reference internal" href="pyspark.sql.html#pyspark.sql.DataFrame" title="pyspark.sql.DataFrame"><code class="xref py py-class docutils literal"><span class="pre">pyspark.sql.DataFrame</span></code></a></li>
<li><strong>params</strong> – an optional param map that overrides embedded
params. If a list/tuple of param maps is given,
this calls fit on each param map and returns a
list of models.</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body"><p class="first last">fitted model(s)</p>
</td>
</tr>
</tbody>
</table>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.GBTClassifier.getCacheNodeIds">
<code class="descname">getCacheNodeIds</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.GBTClassifier.getCacheNodeIds" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of cacheNodeIds or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.GBTClassifier.getCheckpointInterval">
<code class="descname">getCheckpointInterval</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.GBTClassifier.getCheckpointInterval" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of checkpointInterval or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.GBTClassifier.getFeaturesCol">
<code class="descname">getFeaturesCol</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.GBTClassifier.getFeaturesCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of featuresCol or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.GBTClassifier.getLabelCol">
<code class="descname">getLabelCol</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.GBTClassifier.getLabelCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of labelCol or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.GBTClassifier.getLossType">
<code class="descname">getLossType</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/classification.html#GBTClassifier.getLossType"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.classification.GBTClassifier.getLossType" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of lossType or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.GBTClassifier.getMaxBins">
<code class="descname">getMaxBins</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.GBTClassifier.getMaxBins" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of maxBins or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.GBTClassifier.getMaxDepth">
<code class="descname">getMaxDepth</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.GBTClassifier.getMaxDepth" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of maxDepth or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.GBTClassifier.getMaxIter">
<code class="descname">getMaxIter</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.GBTClassifier.getMaxIter" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of maxIter or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.GBTClassifier.getMaxMemoryInMB">
<code class="descname">getMaxMemoryInMB</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.GBTClassifier.getMaxMemoryInMB" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of maxMemoryInMB or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.GBTClassifier.getMinInfoGain">
<code class="descname">getMinInfoGain</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.GBTClassifier.getMinInfoGain" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of minInfoGain or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.GBTClassifier.getMinInstancesPerNode">
<code class="descname">getMinInstancesPerNode</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.GBTClassifier.getMinInstancesPerNode" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of minInstancesPerNode or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.GBTClassifier.getOrDefault">
<code class="descname">getOrDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.GBTClassifier.getOrDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of a param in the user-supplied param map or its
default value. Raises an error if neither is set.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.GBTClassifier.getParam">
<code class="descname">getParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.GBTClassifier.getParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets a param by its name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.GBTClassifier.getPredictionCol">
<code class="descname">getPredictionCol</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.GBTClassifier.getPredictionCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of predictionCol or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.GBTClassifier.getStepSize">
<code class="descname">getStepSize</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/classification.html#GBTClassifier.getStepSize"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.classification.GBTClassifier.getStepSize" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of stepSize or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.GBTClassifier.getSubsamplingRate">
<code class="descname">getSubsamplingRate</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/classification.html#GBTClassifier.getSubsamplingRate"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.classification.GBTClassifier.getSubsamplingRate" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of subsamplingRate or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.GBTClassifier.hasDefault">
<code class="descname">hasDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.GBTClassifier.hasDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param has a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.GBTClassifier.hasParam">
<code class="descname">hasParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.GBTClassifier.hasParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Tests whether this instance contains a param with a given
(string) name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.GBTClassifier.isDefined">
<code class="descname">isDefined</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.GBTClassifier.isDefined" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user or has
a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.GBTClassifier.isSet">
<code class="descname">isSet</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.GBTClassifier.isSet" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.classification.GBTClassifier.labelCol">
<code class="descname">labelCol</code><em class="property"> = Param(parent='undefined', name='labelCol', doc='label column name')</em><a class="headerlink" href="#pyspark.ml.classification.GBTClassifier.labelCol" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.classification.GBTClassifier.lossType">
<code class="descname">lossType</code><em class="property"> = Param(parent='undefined', name='lossType', doc='Loss function which GBT tries to minimize (case-insensitive). Supported options: logistic')</em><a class="headerlink" href="#pyspark.ml.classification.GBTClassifier.lossType" title="Permalink to this definition">¶</a></dt>
<dd><p>param for Loss function which GBT tries to minimize (case-insensitive).</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.classification.GBTClassifier.maxBins">
<code class="descname">maxBins</code><em class="property"> = Param(parent='undefined', name='maxBins', doc='Max number of bins for discretizing continuous features. Must be >=2 and >= number of categories for any categorical feature.')</em><a class="headerlink" href="#pyspark.ml.classification.GBTClassifier.maxBins" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.classification.GBTClassifier.maxDepth">
<code class="descname">maxDepth</code><em class="property"> = Param(parent='undefined', name='maxDepth', doc='Maximum depth of the tree. (>= 0) E.g., depth 0 means 1 leaf node; depth 1 means 1 internal node + 2 leaf nodes.')</em><a class="headerlink" href="#pyspark.ml.classification.GBTClassifier.maxDepth" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.classification.GBTClassifier.maxIter">
<code class="descname">maxIter</code><em class="property"> = Param(parent='undefined', name='maxIter', doc='max number of iterations (>= 0)')</em><a class="headerlink" href="#pyspark.ml.classification.GBTClassifier.maxIter" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.classification.GBTClassifier.maxMemoryInMB">
<code class="descname">maxMemoryInMB</code><em class="property"> = Param(parent='undefined', name='maxMemoryInMB', doc='Maximum memory in MB allocated to histogram aggregation.')</em><a class="headerlink" href="#pyspark.ml.classification.GBTClassifier.maxMemoryInMB" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.classification.GBTClassifier.minInfoGain">
<code class="descname">minInfoGain</code><em class="property"> = Param(parent='undefined', name='minInfoGain', doc='Minimum information gain for a split to be considered at a tree node.')</em><a class="headerlink" href="#pyspark.ml.classification.GBTClassifier.minInfoGain" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.classification.GBTClassifier.minInstancesPerNode">
<code class="descname">minInstancesPerNode</code><em class="property"> = Param(parent='undefined', name='minInstancesPerNode', doc='Minimum number of instances each child must have after split. If a split causes the left or right child to have fewer than minInstancesPerNode, the split will be discarded as invalid. Should be >= 1.')</em><a class="headerlink" href="#pyspark.ml.classification.GBTClassifier.minInstancesPerNode" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.classification.GBTClassifier.params">
<code class="descname">params</code><a class="headerlink" href="#pyspark.ml.classification.GBTClassifier.params" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns all params ordered by name. The default implementation
uses <code class="xref py py-func docutils literal"><span class="pre">dir()</span></code> to get all attributes of type
<code class="xref py py-class docutils literal"><span class="pre">Param</span></code>.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.classification.GBTClassifier.predictionCol">
<code class="descname">predictionCol</code><em class="property"> = Param(parent='undefined', name='predictionCol', doc='prediction column name')</em><a class="headerlink" href="#pyspark.ml.classification.GBTClassifier.predictionCol" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.GBTClassifier.setCacheNodeIds">
<code class="descname">setCacheNodeIds</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.GBTClassifier.setCacheNodeIds" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.classification.GBTClassifier.cacheNodeIds" title="pyspark.ml.classification.GBTClassifier.cacheNodeIds"><code class="xref py py-attr docutils literal"><span class="pre">cacheNodeIds</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.GBTClassifier.setCheckpointInterval">
<code class="descname">setCheckpointInterval</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.GBTClassifier.setCheckpointInterval" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.classification.GBTClassifier.checkpointInterval" title="pyspark.ml.classification.GBTClassifier.checkpointInterval"><code class="xref py py-attr docutils literal"><span class="pre">checkpointInterval</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.GBTClassifier.setFeaturesCol">
<code class="descname">setFeaturesCol</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.GBTClassifier.setFeaturesCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.classification.GBTClassifier.featuresCol" title="pyspark.ml.classification.GBTClassifier.featuresCol"><code class="xref py py-attr docutils literal"><span class="pre">featuresCol</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.GBTClassifier.setLabelCol">
<code class="descname">setLabelCol</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.GBTClassifier.setLabelCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.classification.GBTClassifier.labelCol" title="pyspark.ml.classification.GBTClassifier.labelCol"><code class="xref py py-attr docutils literal"><span class="pre">labelCol</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.GBTClassifier.setLossType">
<code class="descname">setLossType</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/classification.html#GBTClassifier.setLossType"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.classification.GBTClassifier.setLossType" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.classification.GBTClassifier.lossType" title="pyspark.ml.classification.GBTClassifier.lossType"><code class="xref py py-attr docutils literal"><span class="pre">lossType</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.GBTClassifier.setMaxBins">
<code class="descname">setMaxBins</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.GBTClassifier.setMaxBins" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.classification.GBTClassifier.maxBins" title="pyspark.ml.classification.GBTClassifier.maxBins"><code class="xref py py-attr docutils literal"><span class="pre">maxBins</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.GBTClassifier.setMaxDepth">
<code class="descname">setMaxDepth</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.GBTClassifier.setMaxDepth" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.classification.GBTClassifier.maxDepth" title="pyspark.ml.classification.GBTClassifier.maxDepth"><code class="xref py py-attr docutils literal"><span class="pre">maxDepth</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.GBTClassifier.setMaxIter">
<code class="descname">setMaxIter</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.GBTClassifier.setMaxIter" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.classification.GBTClassifier.maxIter" title="pyspark.ml.classification.GBTClassifier.maxIter"><code class="xref py py-attr docutils literal"><span class="pre">maxIter</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.GBTClassifier.setMaxMemoryInMB">
<code class="descname">setMaxMemoryInMB</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.GBTClassifier.setMaxMemoryInMB" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.classification.GBTClassifier.maxMemoryInMB" title="pyspark.ml.classification.GBTClassifier.maxMemoryInMB"><code class="xref py py-attr docutils literal"><span class="pre">maxMemoryInMB</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.GBTClassifier.setMinInfoGain">
<code class="descname">setMinInfoGain</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.GBTClassifier.setMinInfoGain" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.classification.GBTClassifier.minInfoGain" title="pyspark.ml.classification.GBTClassifier.minInfoGain"><code class="xref py py-attr docutils literal"><span class="pre">minInfoGain</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.GBTClassifier.setMinInstancesPerNode">
<code class="descname">setMinInstancesPerNode</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.GBTClassifier.setMinInstancesPerNode" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.classification.GBTClassifier.minInstancesPerNode" title="pyspark.ml.classification.GBTClassifier.minInstancesPerNode"><code class="xref py py-attr docutils literal"><span class="pre">minInstancesPerNode</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.GBTClassifier.setParams">
<code class="descname">setParams</code><span class="sig-paren">(</span><em>self</em>, <em>featuresCol="features"</em>, <em>labelCol="label"</em>, <em>predictionCol="prediction"</em>, <em>maxDepth=5</em>, <em>maxBins=32</em>, <em>minInstancesPerNode=1</em>, <em>minInfoGain=0.0</em>, <em>maxMemoryInMB=256</em>, <em>cacheNodeIds=False</em>, <em>checkpointInterval=10</em>, <em>lossType="logistic"</em>, <em>maxIter=20</em>, <em>stepSize=0.1</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/classification.html#GBTClassifier.setParams"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.classification.GBTClassifier.setParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets params for Gradient Boosted Tree Classification.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.GBTClassifier.setPredictionCol">
<code class="descname">setPredictionCol</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.GBTClassifier.setPredictionCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.classification.GBTClassifier.predictionCol" title="pyspark.ml.classification.GBTClassifier.predictionCol"><code class="xref py py-attr docutils literal"><span class="pre">predictionCol</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.GBTClassifier.setStepSize">
<code class="descname">setStepSize</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/classification.html#GBTClassifier.setStepSize"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.classification.GBTClassifier.setStepSize" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.classification.GBTClassifier.stepSize" title="pyspark.ml.classification.GBTClassifier.stepSize"><code class="xref py py-attr docutils literal"><span class="pre">stepSize</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.GBTClassifier.setSubsamplingRate">
<code class="descname">setSubsamplingRate</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/classification.html#GBTClassifier.setSubsamplingRate"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.classification.GBTClassifier.setSubsamplingRate" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.classification.GBTClassifier.subsamplingRate" title="pyspark.ml.classification.GBTClassifier.subsamplingRate"><code class="xref py py-attr docutils literal"><span class="pre">subsamplingRate</span></code></a>.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.classification.GBTClassifier.stepSize">
<code class="descname">stepSize</code><em class="property"> = Param(parent='undefined', name='stepSize', doc='Step size (a.k.a. learning rate) in interval (0, 1] for shrinking the contribution of each estimator')</em><a class="headerlink" href="#pyspark.ml.classification.GBTClassifier.stepSize" title="Permalink to this definition">¶</a></dt>
<dd><p>Step size (a.k.a. learning rate) in interval (0, 1] for shrinking the contribution of</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.classification.GBTClassifier.subsamplingRate">
<code class="descname">subsamplingRate</code><em class="property"> = Param(parent='undefined', name='subsamplingRate', doc='Fraction of the training data used for learning each decision tree, in range (0, 1].')</em><a class="headerlink" href="#pyspark.ml.classification.GBTClassifier.subsamplingRate" title="Permalink to this definition">¶</a></dt>
<dd><p>Fraction of the training data used for learning each decision tree, in range (0, 1].</p>
</dd></dl>
</dd></dl>
<dl class="class">
<dt id="pyspark.ml.classification.GBTClassificationModel">
<em class="property">class </em><code class="descclassname">pyspark.ml.classification.</code><code class="descname">GBTClassificationModel</code><span class="sig-paren">(</span><em>java_model</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/classification.html#GBTClassificationModel"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.classification.GBTClassificationModel" title="Permalink to this definition">¶</a></dt>
<dd><p>Model fitted by GBTClassifier.</p>
<dl class="method">
<dt id="pyspark.ml.classification.GBTClassificationModel.copy">
<code class="descname">copy</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.GBTClassificationModel.copy" title="Permalink to this definition">¶</a></dt>
<dd><p>Creates a copy of this instance with the same uid and some
extra params. This implementation first calls Params.copy and
then make a copy of the companion Java model with extra params.
So both the Python wrapper and the Java model get copied.
:param extra: Extra parameters to copy to the new instance
:return: Copy of this instance</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.GBTClassificationModel.explainParam">
<code class="descname">explainParam</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.GBTClassificationModel.explainParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Explains a single param and returns its name, doc, and optional
default value and user-supplied value in a string.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.GBTClassificationModel.explainParams">
<code class="descname">explainParams</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.GBTClassificationModel.explainParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns the documentation of all params with their optionally
default values and user-supplied values.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.GBTClassificationModel.extractParamMap">
<code class="descname">extractParamMap</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.GBTClassificationModel.extractParamMap" title="Permalink to this definition">¶</a></dt>
<dd><p>Extracts the embedded default param values and user-supplied
values, and then merges them with extra values from input into
a flat param map, where the latter value is used if there exist
conflicts, i.e., with ordering: default param values <
user-supplied values < extra.
:param extra: extra param values
:return: merged param map</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.GBTClassificationModel.getOrDefault">
<code class="descname">getOrDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.GBTClassificationModel.getOrDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of a param in the user-supplied param map or its
default value. Raises an error if neither is set.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.GBTClassificationModel.getParam">
<code class="descname">getParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.GBTClassificationModel.getParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets a param by its name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.GBTClassificationModel.hasDefault">
<code class="descname">hasDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.GBTClassificationModel.hasDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param has a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.GBTClassificationModel.hasParam">
<code class="descname">hasParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.GBTClassificationModel.hasParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Tests whether this instance contains a param with a given
(string) name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.GBTClassificationModel.isDefined">
<code class="descname">isDefined</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.GBTClassificationModel.isDefined" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user or has
a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.GBTClassificationModel.isSet">
<code class="descname">isSet</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.GBTClassificationModel.isSet" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.classification.GBTClassificationModel.params">
<code class="descname">params</code><a class="headerlink" href="#pyspark.ml.classification.GBTClassificationModel.params" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns all params ordered by name. The default implementation
uses <code class="xref py py-func docutils literal"><span class="pre">dir()</span></code> to get all attributes of type
<code class="xref py py-class docutils literal"><span class="pre">Param</span></code>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.GBTClassificationModel.transform">
<code class="descname">transform</code><span class="sig-paren">(</span><em>dataset</em>, <em>params=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.GBTClassificationModel.transform" title="Permalink to this definition">¶</a></dt>
<dd><p>Transforms the input dataset with optional parameters.</p>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple">
<li><strong>dataset</strong> – input dataset, which is an instance of
<a class="reference internal" href="pyspark.sql.html#pyspark.sql.DataFrame" title="pyspark.sql.DataFrame"><code class="xref py py-class docutils literal"><span class="pre">pyspark.sql.DataFrame</span></code></a></li>
<li><strong>params</strong> – an optional param map that overrides embedded
params.</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body"><p class="first last">transformed dataset</p>
</td>
</tr>
</tbody>
</table>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.classification.GBTClassificationModel.treeWeights">
<code class="descname">treeWeights</code><a class="headerlink" href="#pyspark.ml.classification.GBTClassificationModel.treeWeights" title="Permalink to this definition">¶</a></dt>
<dd><p>Return the weights for each tree</p>
</dd></dl>
</dd></dl>
<dl class="class">
<dt id="pyspark.ml.classification.RandomForestClassifier">
<em class="property">class </em><code class="descclassname">pyspark.ml.classification.</code><code class="descname">RandomForestClassifier</code><span class="sig-paren">(</span><em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/classification.html#RandomForestClassifier"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.classification.RandomForestClassifier" title="Permalink to this definition">¶</a></dt>
<dd><p><cite>http://en.wikipedia.org/wiki/Random_forest Random Forest</cite>
learning algorithm for classification.
It supports both binary and multiclass labels, as well as both continuous and categorical
features.</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="kn">import</span> <span class="nn">numpy</span>
<span class="gp">>>> </span><span class="kn">from</span> <span class="nn">numpy</span> <span class="kn">import</span> <span class="n">allclose</span>
<span class="gp">>>> </span><span class="kn">from</span> <span class="nn">pyspark.mllib.linalg</span> <span class="kn">import</span> <span class="n">Vectors</span>
<span class="gp">>>> </span><span class="kn">from</span> <span class="nn">pyspark.ml.feature</span> <span class="kn">import</span> <span class="n">StringIndexer</span>
<span class="gp">>>> </span><span class="n">df</span> <span class="o">=</span> <span class="n">sqlContext</span><span class="o">.</span><span class="n">createDataFrame</span><span class="p">([</span>
<span class="gp">... </span> <span class="p">(</span><span class="mf">1.0</span><span class="p">,</span> <span class="n">Vectors</span><span class="o">.</span><span class="n">dense</span><span class="p">(</span><span class="mf">1.0</span><span class="p">)),</span>
<span class="gp">... </span> <span class="p">(</span><span class="mf">0.0</span><span class="p">,</span> <span class="n">Vectors</span><span class="o">.</span><span class="n">sparse</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="p">[],</span> <span class="p">[]))],</span> <span class="p">[</span><span class="s">"label"</span><span class="p">,</span> <span class="s">"features"</span><span class="p">])</span>
<span class="gp">>>> </span><span class="n">stringIndexer</span> <span class="o">=</span> <span class="n">StringIndexer</span><span class="p">(</span><span class="n">inputCol</span><span class="o">=</span><span class="s">"label"</span><span class="p">,</span> <span class="n">outputCol</span><span class="o">=</span><span class="s">"indexed"</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">si_model</span> <span class="o">=</span> <span class="n">stringIndexer</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">df</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">td</span> <span class="o">=</span> <span class="n">si_model</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">df</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">rf</span> <span class="o">=</span> <span class="n">RandomForestClassifier</span><span class="p">(</span><span class="n">numTrees</span><span class="o">=</span><span class="mi">3</span><span class="p">,</span> <span class="n">maxDepth</span><span class="o">=</span><span class="mi">2</span><span class="p">,</span> <span class="n">labelCol</span><span class="o">=</span><span class="s">"indexed"</span><span class="p">,</span> <span class="n">seed</span><span class="o">=</span><span class="mi">42</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">model</span> <span class="o">=</span> <span class="n">rf</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">td</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">allclose</span><span class="p">(</span><span class="n">model</span><span class="o">.</span><span class="n">treeWeights</span><span class="p">,</span> <span class="p">[</span><span class="mf">1.0</span><span class="p">,</span> <span class="mf">1.0</span><span class="p">,</span> <span class="mf">1.0</span><span class="p">])</span>
<span class="go">True</span>
<span class="gp">>>> </span><span class="n">test0</span> <span class="o">=</span> <span class="n">sqlContext</span><span class="o">.</span><span class="n">createDataFrame</span><span class="p">([(</span><span class="n">Vectors</span><span class="o">.</span><span class="n">dense</span><span class="p">(</span><span class="o">-</span><span class="mf">1.0</span><span class="p">),)],</span> <span class="p">[</span><span class="s">"features"</span><span class="p">])</span>
<span class="gp">>>> </span><span class="n">result</span> <span class="o">=</span> <span class="n">model</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">test0</span><span class="p">)</span><span class="o">.</span><span class="n">head</span><span class="p">()</span>
<span class="gp">>>> </span><span class="n">result</span><span class="o">.</span><span class="n">prediction</span>
<span class="go">0.0</span>
<span class="gp">>>> </span><span class="n">numpy</span><span class="o">.</span><span class="n">argmax</span><span class="p">(</span><span class="n">result</span><span class="o">.</span><span class="n">probability</span><span class="p">)</span>
<span class="go">0</span>
<span class="gp">>>> </span><span class="n">numpy</span><span class="o">.</span><span class="n">argmax</span><span class="p">(</span><span class="n">result</span><span class="o">.</span><span class="n">rawPrediction</span><span class="p">)</span>
<span class="go">0</span>
<span class="gp">>>> </span><span class="n">test1</span> <span class="o">=</span> <span class="n">sqlContext</span><span class="o">.</span><span class="n">createDataFrame</span><span class="p">([(</span><span class="n">Vectors</span><span class="o">.</span><span class="n">sparse</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="p">[</span><span class="mf">1.0</span><span class="p">]),)],</span> <span class="p">[</span><span class="s">"features"</span><span class="p">])</span>
<span class="gp">>>> </span><span class="n">model</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">test1</span><span class="p">)</span><span class="o">.</span><span class="n">head</span><span class="p">()</span><span class="o">.</span><span class="n">prediction</span>
<span class="go">1.0</span>
</pre></div>
</div>
<dl class="attribute">
<dt id="pyspark.ml.classification.RandomForestClassifier.cacheNodeIds">
<code class="descname">cacheNodeIds</code><em class="property"> = Param(parent='undefined', name='cacheNodeIds', doc='If false, the algorithm will pass trees to executors to match instances with nodes. If true, the algorithm will cache node IDs for each instance. Caching can speed up training of deeper trees.')</em><a class="headerlink" href="#pyspark.ml.classification.RandomForestClassifier.cacheNodeIds" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.classification.RandomForestClassifier.checkpointInterval">
<code class="descname">checkpointInterval</code><em class="property"> = Param(parent='undefined', name='checkpointInterval', doc='checkpoint interval (>= 1)')</em><a class="headerlink" href="#pyspark.ml.classification.RandomForestClassifier.checkpointInterval" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.RandomForestClassifier.copy">
<code class="descname">copy</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.RandomForestClassifier.copy" title="Permalink to this definition">¶</a></dt>
<dd><p>Creates a copy of this instance with the same uid and some
extra params. The default implementation creates a
shallow copy using <code class="xref py py-func docutils literal"><span class="pre">copy.copy()</span></code>, and then copies the
embedded and extra parameters over and returns the copy.
Subclasses should override this method if the default approach
is not sufficient.
:param extra: Extra parameters to copy to the new instance
:return: Copy of this instance</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.RandomForestClassifier.explainParam">
<code class="descname">explainParam</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.RandomForestClassifier.explainParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Explains a single param and returns its name, doc, and optional
default value and user-supplied value in a string.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.RandomForestClassifier.explainParams">
<code class="descname">explainParams</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.RandomForestClassifier.explainParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns the documentation of all params with their optionally
default values and user-supplied values.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.RandomForestClassifier.extractParamMap">
<code class="descname">extractParamMap</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.RandomForestClassifier.extractParamMap" title="Permalink to this definition">¶</a></dt>
<dd><p>Extracts the embedded default param values and user-supplied
values, and then merges them with extra values from input into
a flat param map, where the latter value is used if there exist
conflicts, i.e., with ordering: default param values <
user-supplied values < extra.
:param extra: extra param values
:return: merged param map</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.classification.RandomForestClassifier.featureSubsetStrategy">
<code class="descname">featureSubsetStrategy</code><em class="property"> = Param(parent='undefined', name='featureSubsetStrategy', doc='The number of features to consider for splits at each tree node. Supported options: auto, all, onethird, sqrt, log2')</em><a class="headerlink" href="#pyspark.ml.classification.RandomForestClassifier.featureSubsetStrategy" title="Permalink to this definition">¶</a></dt>
<dd><p>param for The number of features to consider for splits at each tree node</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.classification.RandomForestClassifier.featuresCol">
<code class="descname">featuresCol</code><em class="property"> = Param(parent='undefined', name='featuresCol', doc='features column name')</em><a class="headerlink" href="#pyspark.ml.classification.RandomForestClassifier.featuresCol" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.RandomForestClassifier.fit">
<code class="descname">fit</code><span class="sig-paren">(</span><em>dataset</em>, <em>params=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.RandomForestClassifier.fit" title="Permalink to this definition">¶</a></dt>
<dd><p>Fits a model to the input dataset with optional parameters.</p>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple">
<li><strong>dataset</strong> – input dataset, which is an instance of
<a class="reference internal" href="pyspark.sql.html#pyspark.sql.DataFrame" title="pyspark.sql.DataFrame"><code class="xref py py-class docutils literal"><span class="pre">pyspark.sql.DataFrame</span></code></a></li>
<li><strong>params</strong> – an optional param map that overrides embedded
params. If a list/tuple of param maps is given,
this calls fit on each param map and returns a
list of models.</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body"><p class="first last">fitted model(s)</p>
</td>
</tr>
</tbody>
</table>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.RandomForestClassifier.getCacheNodeIds">
<code class="descname">getCacheNodeIds</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.RandomForestClassifier.getCacheNodeIds" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of cacheNodeIds or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.RandomForestClassifier.getCheckpointInterval">
<code class="descname">getCheckpointInterval</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.RandomForestClassifier.getCheckpointInterval" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of checkpointInterval or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.RandomForestClassifier.getFeatureSubsetStrategy">
<code class="descname">getFeatureSubsetStrategy</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/classification.html#RandomForestClassifier.getFeatureSubsetStrategy"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.classification.RandomForestClassifier.getFeatureSubsetStrategy" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of featureSubsetStrategy or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.RandomForestClassifier.getFeaturesCol">
<code class="descname">getFeaturesCol</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.RandomForestClassifier.getFeaturesCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of featuresCol or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.RandomForestClassifier.getImpurity">
<code class="descname">getImpurity</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/classification.html#RandomForestClassifier.getImpurity"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.classification.RandomForestClassifier.getImpurity" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of impurity or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.RandomForestClassifier.getLabelCol">
<code class="descname">getLabelCol</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.RandomForestClassifier.getLabelCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of labelCol or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.RandomForestClassifier.getMaxBins">
<code class="descname">getMaxBins</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.RandomForestClassifier.getMaxBins" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of maxBins or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.RandomForestClassifier.getMaxDepth">
<code class="descname">getMaxDepth</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.RandomForestClassifier.getMaxDepth" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of maxDepth or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.RandomForestClassifier.getMaxMemoryInMB">
<code class="descname">getMaxMemoryInMB</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.RandomForestClassifier.getMaxMemoryInMB" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of maxMemoryInMB or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.RandomForestClassifier.getMinInfoGain">
<code class="descname">getMinInfoGain</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.RandomForestClassifier.getMinInfoGain" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of minInfoGain or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.RandomForestClassifier.getMinInstancesPerNode">
<code class="descname">getMinInstancesPerNode</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.RandomForestClassifier.getMinInstancesPerNode" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of minInstancesPerNode or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.RandomForestClassifier.getNumTrees">
<code class="descname">getNumTrees</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/classification.html#RandomForestClassifier.getNumTrees"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.classification.RandomForestClassifier.getNumTrees" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of numTrees or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.RandomForestClassifier.getOrDefault">
<code class="descname">getOrDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.RandomForestClassifier.getOrDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of a param in the user-supplied param map or its
default value. Raises an error if neither is set.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.RandomForestClassifier.getParam">
<code class="descname">getParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.RandomForestClassifier.getParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets a param by its name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.RandomForestClassifier.getPredictionCol">
<code class="descname">getPredictionCol</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.RandomForestClassifier.getPredictionCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of predictionCol or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.RandomForestClassifier.getProbabilityCol">
<code class="descname">getProbabilityCol</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.RandomForestClassifier.getProbabilityCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of probabilityCol or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.RandomForestClassifier.getRawPredictionCol">
<code class="descname">getRawPredictionCol</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.RandomForestClassifier.getRawPredictionCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of rawPredictionCol or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.RandomForestClassifier.getSeed">
<code class="descname">getSeed</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.RandomForestClassifier.getSeed" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of seed or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.RandomForestClassifier.getSubsamplingRate">
<code class="descname">getSubsamplingRate</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/classification.html#RandomForestClassifier.getSubsamplingRate"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.classification.RandomForestClassifier.getSubsamplingRate" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of subsamplingRate or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.RandomForestClassifier.hasDefault">
<code class="descname">hasDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.RandomForestClassifier.hasDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param has a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.RandomForestClassifier.hasParam">
<code class="descname">hasParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.RandomForestClassifier.hasParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Tests whether this instance contains a param with a given
(string) name.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.classification.RandomForestClassifier.impurity">
<code class="descname">impurity</code><em class="property"> = Param(parent='undefined', name='impurity', doc='Criterion used for information gain calculation (case-insensitive). Supported options: entropy, gini')</em><a class="headerlink" href="#pyspark.ml.classification.RandomForestClassifier.impurity" title="Permalink to this definition">¶</a></dt>
<dd><p>param for Criterion used for information gain calculation (case-insensitive).</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.RandomForestClassifier.isDefined">
<code class="descname">isDefined</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.RandomForestClassifier.isDefined" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user or has
a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.RandomForestClassifier.isSet">
<code class="descname">isSet</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.RandomForestClassifier.isSet" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.classification.RandomForestClassifier.labelCol">
<code class="descname">labelCol</code><em class="property"> = Param(parent='undefined', name='labelCol', doc='label column name')</em><a class="headerlink" href="#pyspark.ml.classification.RandomForestClassifier.labelCol" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.classification.RandomForestClassifier.maxBins">
<code class="descname">maxBins</code><em class="property"> = Param(parent='undefined', name='maxBins', doc='Max number of bins for discretizing continuous features. Must be >=2 and >= number of categories for any categorical feature.')</em><a class="headerlink" href="#pyspark.ml.classification.RandomForestClassifier.maxBins" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.classification.RandomForestClassifier.maxDepth">
<code class="descname">maxDepth</code><em class="property"> = Param(parent='undefined', name='maxDepth', doc='Maximum depth of the tree. (>= 0) E.g., depth 0 means 1 leaf node; depth 1 means 1 internal node + 2 leaf nodes.')</em><a class="headerlink" href="#pyspark.ml.classification.RandomForestClassifier.maxDepth" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.classification.RandomForestClassifier.maxMemoryInMB">
<code class="descname">maxMemoryInMB</code><em class="property"> = Param(parent='undefined', name='maxMemoryInMB', doc='Maximum memory in MB allocated to histogram aggregation.')</em><a class="headerlink" href="#pyspark.ml.classification.RandomForestClassifier.maxMemoryInMB" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.classification.RandomForestClassifier.minInfoGain">
<code class="descname">minInfoGain</code><em class="property"> = Param(parent='undefined', name='minInfoGain', doc='Minimum information gain for a split to be considered at a tree node.')</em><a class="headerlink" href="#pyspark.ml.classification.RandomForestClassifier.minInfoGain" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.classification.RandomForestClassifier.minInstancesPerNode">
<code class="descname">minInstancesPerNode</code><em class="property"> = Param(parent='undefined', name='minInstancesPerNode', doc='Minimum number of instances each child must have after split. If a split causes the left or right child to have fewer than minInstancesPerNode, the split will be discarded as invalid. Should be >= 1.')</em><a class="headerlink" href="#pyspark.ml.classification.RandomForestClassifier.minInstancesPerNode" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.classification.RandomForestClassifier.numTrees">
<code class="descname">numTrees</code><em class="property"> = Param(parent='undefined', name='numTrees', doc='Number of trees to train (>= 1)')</em><a class="headerlink" href="#pyspark.ml.classification.RandomForestClassifier.numTrees" title="Permalink to this definition">¶</a></dt>
<dd><p>param for Number of trees to train (>= 1)</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.classification.RandomForestClassifier.params">
<code class="descname">params</code><a class="headerlink" href="#pyspark.ml.classification.RandomForestClassifier.params" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns all params ordered by name. The default implementation
uses <code class="xref py py-func docutils literal"><span class="pre">dir()</span></code> to get all attributes of type
<code class="xref py py-class docutils literal"><span class="pre">Param</span></code>.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.classification.RandomForestClassifier.predictionCol">
<code class="descname">predictionCol</code><em class="property"> = Param(parent='undefined', name='predictionCol', doc='prediction column name')</em><a class="headerlink" href="#pyspark.ml.classification.RandomForestClassifier.predictionCol" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.classification.RandomForestClassifier.probabilityCol">
<code class="descname">probabilityCol</code><em class="property"> = Param(parent='undefined', name='probabilityCol', doc='Column name for predicted class conditional probabilities. Note: Not all models output well-calibrated probability estimates! These probabilities should be treated as confidences, not precise probabilities.')</em><a class="headerlink" href="#pyspark.ml.classification.RandomForestClassifier.probabilityCol" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.classification.RandomForestClassifier.rawPredictionCol">
<code class="descname">rawPredictionCol</code><em class="property"> = Param(parent='undefined', name='rawPredictionCol', doc='raw prediction (a.k.a. confidence) column name')</em><a class="headerlink" href="#pyspark.ml.classification.RandomForestClassifier.rawPredictionCol" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.classification.RandomForestClassifier.seed">
<code class="descname">seed</code><em class="property"> = Param(parent='undefined', name='seed', doc='random seed')</em><a class="headerlink" href="#pyspark.ml.classification.RandomForestClassifier.seed" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.RandomForestClassifier.setCacheNodeIds">
<code class="descname">setCacheNodeIds</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.RandomForestClassifier.setCacheNodeIds" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.classification.RandomForestClassifier.cacheNodeIds" title="pyspark.ml.classification.RandomForestClassifier.cacheNodeIds"><code class="xref py py-attr docutils literal"><span class="pre">cacheNodeIds</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.RandomForestClassifier.setCheckpointInterval">
<code class="descname">setCheckpointInterval</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.RandomForestClassifier.setCheckpointInterval" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.classification.RandomForestClassifier.checkpointInterval" title="pyspark.ml.classification.RandomForestClassifier.checkpointInterval"><code class="xref py py-attr docutils literal"><span class="pre">checkpointInterval</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.RandomForestClassifier.setFeatureSubsetStrategy">
<code class="descname">setFeatureSubsetStrategy</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/classification.html#RandomForestClassifier.setFeatureSubsetStrategy"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.classification.RandomForestClassifier.setFeatureSubsetStrategy" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.classification.RandomForestClassifier.featureSubsetStrategy" title="pyspark.ml.classification.RandomForestClassifier.featureSubsetStrategy"><code class="xref py py-attr docutils literal"><span class="pre">featureSubsetStrategy</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.RandomForestClassifier.setFeaturesCol">
<code class="descname">setFeaturesCol</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.RandomForestClassifier.setFeaturesCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.classification.RandomForestClassifier.featuresCol" title="pyspark.ml.classification.RandomForestClassifier.featuresCol"><code class="xref py py-attr docutils literal"><span class="pre">featuresCol</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.RandomForestClassifier.setImpurity">
<code class="descname">setImpurity</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/classification.html#RandomForestClassifier.setImpurity"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.classification.RandomForestClassifier.setImpurity" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.classification.RandomForestClassifier.impurity" title="pyspark.ml.classification.RandomForestClassifier.impurity"><code class="xref py py-attr docutils literal"><span class="pre">impurity</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.RandomForestClassifier.setLabelCol">
<code class="descname">setLabelCol</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.RandomForestClassifier.setLabelCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.classification.RandomForestClassifier.labelCol" title="pyspark.ml.classification.RandomForestClassifier.labelCol"><code class="xref py py-attr docutils literal"><span class="pre">labelCol</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.RandomForestClassifier.setMaxBins">
<code class="descname">setMaxBins</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.RandomForestClassifier.setMaxBins" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.classification.RandomForestClassifier.maxBins" title="pyspark.ml.classification.RandomForestClassifier.maxBins"><code class="xref py py-attr docutils literal"><span class="pre">maxBins</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.RandomForestClassifier.setMaxDepth">
<code class="descname">setMaxDepth</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.RandomForestClassifier.setMaxDepth" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.classification.RandomForestClassifier.maxDepth" title="pyspark.ml.classification.RandomForestClassifier.maxDepth"><code class="xref py py-attr docutils literal"><span class="pre">maxDepth</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.RandomForestClassifier.setMaxMemoryInMB">
<code class="descname">setMaxMemoryInMB</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.RandomForestClassifier.setMaxMemoryInMB" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.classification.RandomForestClassifier.maxMemoryInMB" title="pyspark.ml.classification.RandomForestClassifier.maxMemoryInMB"><code class="xref py py-attr docutils literal"><span class="pre">maxMemoryInMB</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.RandomForestClassifier.setMinInfoGain">
<code class="descname">setMinInfoGain</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.RandomForestClassifier.setMinInfoGain" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.classification.RandomForestClassifier.minInfoGain" title="pyspark.ml.classification.RandomForestClassifier.minInfoGain"><code class="xref py py-attr docutils literal"><span class="pre">minInfoGain</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.RandomForestClassifier.setMinInstancesPerNode">
<code class="descname">setMinInstancesPerNode</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.RandomForestClassifier.setMinInstancesPerNode" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.classification.RandomForestClassifier.minInstancesPerNode" title="pyspark.ml.classification.RandomForestClassifier.minInstancesPerNode"><code class="xref py py-attr docutils literal"><span class="pre">minInstancesPerNode</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.RandomForestClassifier.setNumTrees">
<code class="descname">setNumTrees</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/classification.html#RandomForestClassifier.setNumTrees"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.classification.RandomForestClassifier.setNumTrees" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.classification.RandomForestClassifier.numTrees" title="pyspark.ml.classification.RandomForestClassifier.numTrees"><code class="xref py py-attr docutils literal"><span class="pre">numTrees</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.RandomForestClassifier.setParams">
<code class="descname">setParams</code><span class="sig-paren">(</span><em>self</em>, <em>featuresCol="features"</em>, <em>labelCol="label"</em>, <em>predictionCol="prediction"</em>, <em>probabilityCol="probability"</em>, <em>rawPredictionCol="rawPrediction"</em>, <em>maxDepth=5</em>, <em>maxBins=32</em>, <em>minInstancesPerNode=1</em>, <em>minInfoGain=0.0</em>, <em>maxMemoryInMB=256</em>, <em>cacheNodeIds=False</em>, <em>checkpointInterval=10</em>, <em>seed=None</em>, <em>impurity="gini"</em>, <em>numTrees=20</em>, <em>featureSubsetStrategy="auto"</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/classification.html#RandomForestClassifier.setParams"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.classification.RandomForestClassifier.setParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets params for linear classification.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.RandomForestClassifier.setPredictionCol">
<code class="descname">setPredictionCol</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.RandomForestClassifier.setPredictionCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.classification.RandomForestClassifier.predictionCol" title="pyspark.ml.classification.RandomForestClassifier.predictionCol"><code class="xref py py-attr docutils literal"><span class="pre">predictionCol</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.RandomForestClassifier.setProbabilityCol">
<code class="descname">setProbabilityCol</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.RandomForestClassifier.setProbabilityCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.classification.RandomForestClassifier.probabilityCol" title="pyspark.ml.classification.RandomForestClassifier.probabilityCol"><code class="xref py py-attr docutils literal"><span class="pre">probabilityCol</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.RandomForestClassifier.setRawPredictionCol">
<code class="descname">setRawPredictionCol</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.RandomForestClassifier.setRawPredictionCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.classification.RandomForestClassifier.rawPredictionCol" title="pyspark.ml.classification.RandomForestClassifier.rawPredictionCol"><code class="xref py py-attr docutils literal"><span class="pre">rawPredictionCol</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.RandomForestClassifier.setSeed">
<code class="descname">setSeed</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.RandomForestClassifier.setSeed" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.classification.RandomForestClassifier.seed" title="pyspark.ml.classification.RandomForestClassifier.seed"><code class="xref py py-attr docutils literal"><span class="pre">seed</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.RandomForestClassifier.setSubsamplingRate">
<code class="descname">setSubsamplingRate</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/classification.html#RandomForestClassifier.setSubsamplingRate"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.classification.RandomForestClassifier.setSubsamplingRate" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.classification.RandomForestClassifier.subsamplingRate" title="pyspark.ml.classification.RandomForestClassifier.subsamplingRate"><code class="xref py py-attr docutils literal"><span class="pre">subsamplingRate</span></code></a>.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.classification.RandomForestClassifier.subsamplingRate">
<code class="descname">subsamplingRate</code><em class="property"> = Param(parent='undefined', name='subsamplingRate', doc='Fraction of the training data used for learning each decision tree, in range (0, 1].')</em><a class="headerlink" href="#pyspark.ml.classification.RandomForestClassifier.subsamplingRate" title="Permalink to this definition">¶</a></dt>
<dd><p>param for Fraction of the training data used for learning each decision tree,</p>
</dd></dl>
</dd></dl>
<dl class="class">
<dt id="pyspark.ml.classification.RandomForestClassificationModel">
<em class="property">class </em><code class="descclassname">pyspark.ml.classification.</code><code class="descname">RandomForestClassificationModel</code><span class="sig-paren">(</span><em>java_model</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/classification.html#RandomForestClassificationModel"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.classification.RandomForestClassificationModel" title="Permalink to this definition">¶</a></dt>
<dd><p>Model fitted by RandomForestClassifier.</p>
<dl class="method">
<dt id="pyspark.ml.classification.RandomForestClassificationModel.copy">
<code class="descname">copy</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.RandomForestClassificationModel.copy" title="Permalink to this definition">¶</a></dt>
<dd><p>Creates a copy of this instance with the same uid and some
extra params. This implementation first calls Params.copy and
then make a copy of the companion Java model with extra params.
So both the Python wrapper and the Java model get copied.
:param extra: Extra parameters to copy to the new instance
:return: Copy of this instance</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.RandomForestClassificationModel.explainParam">
<code class="descname">explainParam</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.RandomForestClassificationModel.explainParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Explains a single param and returns its name, doc, and optional
default value and user-supplied value in a string.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.RandomForestClassificationModel.explainParams">
<code class="descname">explainParams</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.RandomForestClassificationModel.explainParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns the documentation of all params with their optionally
default values and user-supplied values.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.RandomForestClassificationModel.extractParamMap">
<code class="descname">extractParamMap</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.RandomForestClassificationModel.extractParamMap" title="Permalink to this definition">¶</a></dt>
<dd><p>Extracts the embedded default param values and user-supplied
values, and then merges them with extra values from input into
a flat param map, where the latter value is used if there exist
conflicts, i.e., with ordering: default param values <
user-supplied values < extra.
:param extra: extra param values
:return: merged param map</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.RandomForestClassificationModel.getOrDefault">
<code class="descname">getOrDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.RandomForestClassificationModel.getOrDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of a param in the user-supplied param map or its
default value. Raises an error if neither is set.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.RandomForestClassificationModel.getParam">
<code class="descname">getParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.RandomForestClassificationModel.getParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets a param by its name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.RandomForestClassificationModel.hasDefault">
<code class="descname">hasDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.RandomForestClassificationModel.hasDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param has a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.RandomForestClassificationModel.hasParam">
<code class="descname">hasParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.RandomForestClassificationModel.hasParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Tests whether this instance contains a param with a given
(string) name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.RandomForestClassificationModel.isDefined">
<code class="descname">isDefined</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.RandomForestClassificationModel.isDefined" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user or has
a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.RandomForestClassificationModel.isSet">
<code class="descname">isSet</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.RandomForestClassificationModel.isSet" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.classification.RandomForestClassificationModel.params">
<code class="descname">params</code><a class="headerlink" href="#pyspark.ml.classification.RandomForestClassificationModel.params" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns all params ordered by name. The default implementation
uses <code class="xref py py-func docutils literal"><span class="pre">dir()</span></code> to get all attributes of type
<code class="xref py py-class docutils literal"><span class="pre">Param</span></code>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.RandomForestClassificationModel.transform">
<code class="descname">transform</code><span class="sig-paren">(</span><em>dataset</em>, <em>params=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.RandomForestClassificationModel.transform" title="Permalink to this definition">¶</a></dt>
<dd><p>Transforms the input dataset with optional parameters.</p>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple">
<li><strong>dataset</strong> – input dataset, which is an instance of
<a class="reference internal" href="pyspark.sql.html#pyspark.sql.DataFrame" title="pyspark.sql.DataFrame"><code class="xref py py-class docutils literal"><span class="pre">pyspark.sql.DataFrame</span></code></a></li>
<li><strong>params</strong> – an optional param map that overrides embedded
params.</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body"><p class="first last">transformed dataset</p>
</td>
</tr>
</tbody>
</table>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.classification.RandomForestClassificationModel.treeWeights">
<code class="descname">treeWeights</code><a class="headerlink" href="#pyspark.ml.classification.RandomForestClassificationModel.treeWeights" title="Permalink to this definition">¶</a></dt>
<dd><p>Return the weights for each tree</p>
</dd></dl>
</dd></dl>
<dl class="class">
<dt id="pyspark.ml.classification.NaiveBayes">
<em class="property">class </em><code class="descclassname">pyspark.ml.classification.</code><code class="descname">NaiveBayes</code><span class="sig-paren">(</span><em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/classification.html#NaiveBayes"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.classification.NaiveBayes" title="Permalink to this definition">¶</a></dt>
<dd><p>Naive Bayes Classifiers.
It supports both Multinomial and Bernoulli NB. Multinomial NB
(<cite>http://nlp.stanford.edu/IR-book/html/htmledition/naive-bayes-text-classification-1.html</cite>)
can handle finitely supported discrete data. For example, by converting documents into
TF-IDF vectors, it can be used for document classification. By making every vector a
binary (0/1) data, it can also be used as Bernoulli NB
(<cite>http://nlp.stanford.edu/IR-book/html/htmledition/the-bernoulli-model-1.html</cite>).
The input feature values must be nonnegative.</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="kn">from</span> <span class="nn">pyspark.sql</span> <span class="kn">import</span> <span class="n">Row</span>
<span class="gp">>>> </span><span class="kn">from</span> <span class="nn">pyspark.mllib.linalg</span> <span class="kn">import</span> <span class="n">Vectors</span>
<span class="gp">>>> </span><span class="n">df</span> <span class="o">=</span> <span class="n">sqlContext</span><span class="o">.</span><span class="n">createDataFrame</span><span class="p">([</span>
<span class="gp">... </span> <span class="n">Row</span><span class="p">(</span><span class="n">label</span><span class="o">=</span><span class="mf">0.0</span><span class="p">,</span> <span class="n">features</span><span class="o">=</span><span class="n">Vectors</span><span class="o">.</span><span class="n">dense</span><span class="p">([</span><span class="mf">0.0</span><span class="p">,</span> <span class="mf">0.0</span><span class="p">])),</span>
<span class="gp">... </span> <span class="n">Row</span><span class="p">(</span><span class="n">label</span><span class="o">=</span><span class="mf">0.0</span><span class="p">,</span> <span class="n">features</span><span class="o">=</span><span class="n">Vectors</span><span class="o">.</span><span class="n">dense</span><span class="p">([</span><span class="mf">0.0</span><span class="p">,</span> <span class="mf">1.0</span><span class="p">])),</span>
<span class="gp">... </span> <span class="n">Row</span><span class="p">(</span><span class="n">label</span><span class="o">=</span><span class="mf">1.0</span><span class="p">,</span> <span class="n">features</span><span class="o">=</span><span class="n">Vectors</span><span class="o">.</span><span class="n">dense</span><span class="p">([</span><span class="mf">1.0</span><span class="p">,</span> <span class="mf">0.0</span><span class="p">]))])</span>
<span class="gp">>>> </span><span class="n">nb</span> <span class="o">=</span> <span class="n">NaiveBayes</span><span class="p">(</span><span class="n">smoothing</span><span class="o">=</span><span class="mf">1.0</span><span class="p">,</span> <span class="n">modelType</span><span class="o">=</span><span class="s">"multinomial"</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">model</span> <span class="o">=</span> <span class="n">nb</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">df</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">model</span><span class="o">.</span><span class="n">pi</span>
<span class="go">DenseVector([-0.51..., -0.91...])</span>
<span class="gp">>>> </span><span class="n">model</span><span class="o">.</span><span class="n">theta</span>
<span class="go">DenseMatrix(2, 2, [-1.09..., -0.40..., -0.40..., -1.09...], 1)</span>
<span class="gp">>>> </span><span class="n">test0</span> <span class="o">=</span> <span class="n">sc</span><span class="o">.</span><span class="n">parallelize</span><span class="p">([</span><span class="n">Row</span><span class="p">(</span><span class="n">features</span><span class="o">=</span><span class="n">Vectors</span><span class="o">.</span><span class="n">dense</span><span class="p">([</span><span class="mf">1.0</span><span class="p">,</span> <span class="mf">0.0</span><span class="p">]))])</span><span class="o">.</span><span class="n">toDF</span><span class="p">()</span>
<span class="gp">>>> </span><span class="n">result</span> <span class="o">=</span> <span class="n">model</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">test0</span><span class="p">)</span><span class="o">.</span><span class="n">head</span><span class="p">()</span>
<span class="gp">>>> </span><span class="n">result</span><span class="o">.</span><span class="n">prediction</span>
<span class="go">1.0</span>
<span class="gp">>>> </span><span class="n">result</span><span class="o">.</span><span class="n">probability</span>
<span class="go">DenseVector([0.42..., 0.57...])</span>
<span class="gp">>>> </span><span class="n">result</span><span class="o">.</span><span class="n">rawPrediction</span>
<span class="go">DenseVector([-1.60..., -1.32...])</span>
<span class="gp">>>> </span><span class="n">test1</span> <span class="o">=</span> <span class="n">sc</span><span class="o">.</span><span class="n">parallelize</span><span class="p">([</span><span class="n">Row</span><span class="p">(</span><span class="n">features</span><span class="o">=</span><span class="n">Vectors</span><span class="o">.</span><span class="n">sparse</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="p">[</span><span class="mf">1.0</span><span class="p">]))])</span><span class="o">.</span><span class="n">toDF</span><span class="p">()</span>
<span class="gp">>>> </span><span class="n">model</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">test1</span><span class="p">)</span><span class="o">.</span><span class="n">head</span><span class="p">()</span><span class="o">.</span><span class="n">prediction</span>
<span class="go">1.0</span>
</pre></div>
</div>
<dl class="method">
<dt id="pyspark.ml.classification.NaiveBayes.copy">
<code class="descname">copy</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.NaiveBayes.copy" title="Permalink to this definition">¶</a></dt>
<dd><p>Creates a copy of this instance with the same uid and some
extra params. The default implementation creates a
shallow copy using <code class="xref py py-func docutils literal"><span class="pre">copy.copy()</span></code>, and then copies the
embedded and extra parameters over and returns the copy.
Subclasses should override this method if the default approach
is not sufficient.
:param extra: Extra parameters to copy to the new instance
:return: Copy of this instance</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.NaiveBayes.explainParam">
<code class="descname">explainParam</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.NaiveBayes.explainParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Explains a single param and returns its name, doc, and optional
default value and user-supplied value in a string.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.NaiveBayes.explainParams">
<code class="descname">explainParams</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.NaiveBayes.explainParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns the documentation of all params with their optionally
default values and user-supplied values.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.NaiveBayes.extractParamMap">
<code class="descname">extractParamMap</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.NaiveBayes.extractParamMap" title="Permalink to this definition">¶</a></dt>
<dd><p>Extracts the embedded default param values and user-supplied
values, and then merges them with extra values from input into
a flat param map, where the latter value is used if there exist
conflicts, i.e., with ordering: default param values <
user-supplied values < extra.
:param extra: extra param values
:return: merged param map</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.classification.NaiveBayes.featuresCol">
<code class="descname">featuresCol</code><em class="property"> = Param(parent='undefined', name='featuresCol', doc='features column name')</em><a class="headerlink" href="#pyspark.ml.classification.NaiveBayes.featuresCol" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.NaiveBayes.fit">
<code class="descname">fit</code><span class="sig-paren">(</span><em>dataset</em>, <em>params=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.NaiveBayes.fit" title="Permalink to this definition">¶</a></dt>
<dd><p>Fits a model to the input dataset with optional parameters.</p>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple">
<li><strong>dataset</strong> – input dataset, which is an instance of
<a class="reference internal" href="pyspark.sql.html#pyspark.sql.DataFrame" title="pyspark.sql.DataFrame"><code class="xref py py-class docutils literal"><span class="pre">pyspark.sql.DataFrame</span></code></a></li>
<li><strong>params</strong> – an optional param map that overrides embedded
params. If a list/tuple of param maps is given,
this calls fit on each param map and returns a
list of models.</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body"><p class="first last">fitted model(s)</p>
</td>
</tr>
</tbody>
</table>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.NaiveBayes.getFeaturesCol">
<code class="descname">getFeaturesCol</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.NaiveBayes.getFeaturesCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of featuresCol or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.NaiveBayes.getLabelCol">
<code class="descname">getLabelCol</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.NaiveBayes.getLabelCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of labelCol or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.NaiveBayes.getModelType">
<code class="descname">getModelType</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/classification.html#NaiveBayes.getModelType"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.classification.NaiveBayes.getModelType" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of modelType or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.NaiveBayes.getOrDefault">
<code class="descname">getOrDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.NaiveBayes.getOrDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of a param in the user-supplied param map or its
default value. Raises an error if neither is set.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.NaiveBayes.getParam">
<code class="descname">getParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.NaiveBayes.getParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets a param by its name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.NaiveBayes.getPredictionCol">
<code class="descname">getPredictionCol</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.NaiveBayes.getPredictionCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of predictionCol or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.NaiveBayes.getProbabilityCol">
<code class="descname">getProbabilityCol</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.NaiveBayes.getProbabilityCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of probabilityCol or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.NaiveBayes.getRawPredictionCol">
<code class="descname">getRawPredictionCol</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.NaiveBayes.getRawPredictionCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of rawPredictionCol or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.NaiveBayes.getSmoothing">
<code class="descname">getSmoothing</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/classification.html#NaiveBayes.getSmoothing"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.classification.NaiveBayes.getSmoothing" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of smoothing or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.NaiveBayes.hasDefault">
<code class="descname">hasDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.NaiveBayes.hasDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param has a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.NaiveBayes.hasParam">
<code class="descname">hasParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.NaiveBayes.hasParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Tests whether this instance contains a param with a given
(string) name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.NaiveBayes.isDefined">
<code class="descname">isDefined</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.NaiveBayes.isDefined" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user or has
a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.NaiveBayes.isSet">
<code class="descname">isSet</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.NaiveBayes.isSet" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.classification.NaiveBayes.labelCol">
<code class="descname">labelCol</code><em class="property"> = Param(parent='undefined', name='labelCol', doc='label column name')</em><a class="headerlink" href="#pyspark.ml.classification.NaiveBayes.labelCol" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.classification.NaiveBayes.modelType">
<code class="descname">modelType</code><em class="property"> = Param(parent='undefined', name='modelType', doc='The model type which is a string (case-sensitive). Supported options: multinomial (default) and bernoulli.')</em><a class="headerlink" href="#pyspark.ml.classification.NaiveBayes.modelType" title="Permalink to this definition">¶</a></dt>
<dd><p>param for the model type.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.classification.NaiveBayes.params">
<code class="descname">params</code><a class="headerlink" href="#pyspark.ml.classification.NaiveBayes.params" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns all params ordered by name. The default implementation
uses <code class="xref py py-func docutils literal"><span class="pre">dir()</span></code> to get all attributes of type
<code class="xref py py-class docutils literal"><span class="pre">Param</span></code>.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.classification.NaiveBayes.predictionCol">
<code class="descname">predictionCol</code><em class="property"> = Param(parent='undefined', name='predictionCol', doc='prediction column name')</em><a class="headerlink" href="#pyspark.ml.classification.NaiveBayes.predictionCol" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.classification.NaiveBayes.probabilityCol">
<code class="descname">probabilityCol</code><em class="property"> = Param(parent='undefined', name='probabilityCol', doc='Column name for predicted class conditional probabilities. Note: Not all models output well-calibrated probability estimates! These probabilities should be treated as confidences, not precise probabilities.')</em><a class="headerlink" href="#pyspark.ml.classification.NaiveBayes.probabilityCol" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.classification.NaiveBayes.rawPredictionCol">
<code class="descname">rawPredictionCol</code><em class="property"> = Param(parent='undefined', name='rawPredictionCol', doc='raw prediction (a.k.a. confidence) column name')</em><a class="headerlink" href="#pyspark.ml.classification.NaiveBayes.rawPredictionCol" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.NaiveBayes.setFeaturesCol">
<code class="descname">setFeaturesCol</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.NaiveBayes.setFeaturesCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.classification.NaiveBayes.featuresCol" title="pyspark.ml.classification.NaiveBayes.featuresCol"><code class="xref py py-attr docutils literal"><span class="pre">featuresCol</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.NaiveBayes.setLabelCol">
<code class="descname">setLabelCol</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.NaiveBayes.setLabelCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.classification.NaiveBayes.labelCol" title="pyspark.ml.classification.NaiveBayes.labelCol"><code class="xref py py-attr docutils literal"><span class="pre">labelCol</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.NaiveBayes.setModelType">
<code class="descname">setModelType</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/classification.html#NaiveBayes.setModelType"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.classification.NaiveBayes.setModelType" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.classification.NaiveBayes.modelType" title="pyspark.ml.classification.NaiveBayes.modelType"><code class="xref py py-attr docutils literal"><span class="pre">modelType</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.NaiveBayes.setParams">
<code class="descname">setParams</code><span class="sig-paren">(</span><em>self</em>, <em>featuresCol="features"</em>, <em>labelCol="label"</em>, <em>predictionCol="prediction"</em>, <em>probabilityCol="probability"</em>, <em>rawPredictionCol="rawPrediction"</em>, <em>smoothing=1.0</em>, <em>modelType="multinomial"</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/classification.html#NaiveBayes.setParams"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.classification.NaiveBayes.setParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets params for Naive Bayes.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.NaiveBayes.setPredictionCol">
<code class="descname">setPredictionCol</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.NaiveBayes.setPredictionCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.classification.NaiveBayes.predictionCol" title="pyspark.ml.classification.NaiveBayes.predictionCol"><code class="xref py py-attr docutils literal"><span class="pre">predictionCol</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.NaiveBayes.setProbabilityCol">
<code class="descname">setProbabilityCol</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.NaiveBayes.setProbabilityCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.classification.NaiveBayes.probabilityCol" title="pyspark.ml.classification.NaiveBayes.probabilityCol"><code class="xref py py-attr docutils literal"><span class="pre">probabilityCol</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.NaiveBayes.setRawPredictionCol">
<code class="descname">setRawPredictionCol</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.NaiveBayes.setRawPredictionCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.classification.NaiveBayes.rawPredictionCol" title="pyspark.ml.classification.NaiveBayes.rawPredictionCol"><code class="xref py py-attr docutils literal"><span class="pre">rawPredictionCol</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.NaiveBayes.setSmoothing">
<code class="descname">setSmoothing</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/classification.html#NaiveBayes.setSmoothing"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.classification.NaiveBayes.setSmoothing" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.classification.NaiveBayes.smoothing" title="pyspark.ml.classification.NaiveBayes.smoothing"><code class="xref py py-attr docutils literal"><span class="pre">smoothing</span></code></a>.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.classification.NaiveBayes.smoothing">
<code class="descname">smoothing</code><em class="property"> = Param(parent='undefined', name='smoothing', doc='The smoothing parameter, should be >= 0, default is 1.0')</em><a class="headerlink" href="#pyspark.ml.classification.NaiveBayes.smoothing" title="Permalink to this definition">¶</a></dt>
<dd><p>param for the smoothing parameter.</p>
</dd></dl>
</dd></dl>
<dl class="class">
<dt id="pyspark.ml.classification.NaiveBayesModel">
<em class="property">class </em><code class="descclassname">pyspark.ml.classification.</code><code class="descname">NaiveBayesModel</code><span class="sig-paren">(</span><em>java_model</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/classification.html#NaiveBayesModel"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.classification.NaiveBayesModel" title="Permalink to this definition">¶</a></dt>
<dd><p>Model fitted by NaiveBayes.</p>
<dl class="method">
<dt id="pyspark.ml.classification.NaiveBayesModel.copy">
<code class="descname">copy</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.NaiveBayesModel.copy" title="Permalink to this definition">¶</a></dt>
<dd><p>Creates a copy of this instance with the same uid and some
extra params. This implementation first calls Params.copy and
then make a copy of the companion Java model with extra params.
So both the Python wrapper and the Java model get copied.
:param extra: Extra parameters to copy to the new instance
:return: Copy of this instance</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.NaiveBayesModel.explainParam">
<code class="descname">explainParam</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.NaiveBayesModel.explainParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Explains a single param and returns its name, doc, and optional
default value and user-supplied value in a string.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.NaiveBayesModel.explainParams">
<code class="descname">explainParams</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.NaiveBayesModel.explainParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns the documentation of all params with their optionally
default values and user-supplied values.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.NaiveBayesModel.extractParamMap">
<code class="descname">extractParamMap</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.NaiveBayesModel.extractParamMap" title="Permalink to this definition">¶</a></dt>
<dd><p>Extracts the embedded default param values and user-supplied
values, and then merges them with extra values from input into
a flat param map, where the latter value is used if there exist
conflicts, i.e., with ordering: default param values <
user-supplied values < extra.
:param extra: extra param values
:return: merged param map</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.NaiveBayesModel.getOrDefault">
<code class="descname">getOrDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.NaiveBayesModel.getOrDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of a param in the user-supplied param map or its
default value. Raises an error if neither is set.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.NaiveBayesModel.getParam">
<code class="descname">getParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.NaiveBayesModel.getParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets a param by its name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.NaiveBayesModel.hasDefault">
<code class="descname">hasDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.NaiveBayesModel.hasDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param has a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.NaiveBayesModel.hasParam">
<code class="descname">hasParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.NaiveBayesModel.hasParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Tests whether this instance contains a param with a given
(string) name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.NaiveBayesModel.isDefined">
<code class="descname">isDefined</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.NaiveBayesModel.isDefined" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user or has
a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.NaiveBayesModel.isSet">
<code class="descname">isSet</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.NaiveBayesModel.isSet" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.classification.NaiveBayesModel.params">
<code class="descname">params</code><a class="headerlink" href="#pyspark.ml.classification.NaiveBayesModel.params" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns all params ordered by name. The default implementation
uses <code class="xref py py-func docutils literal"><span class="pre">dir()</span></code> to get all attributes of type
<code class="xref py py-class docutils literal"><span class="pre">Param</span></code>.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.classification.NaiveBayesModel.pi">
<code class="descname">pi</code><a class="headerlink" href="#pyspark.ml.classification.NaiveBayesModel.pi" title="Permalink to this definition">¶</a></dt>
<dd><p>log of class priors.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.classification.NaiveBayesModel.theta">
<code class="descname">theta</code><a class="headerlink" href="#pyspark.ml.classification.NaiveBayesModel.theta" title="Permalink to this definition">¶</a></dt>
<dd><p>log of class conditional probabilities.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.classification.NaiveBayesModel.transform">
<code class="descname">transform</code><span class="sig-paren">(</span><em>dataset</em>, <em>params=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.classification.NaiveBayesModel.transform" title="Permalink to this definition">¶</a></dt>
<dd><p>Transforms the input dataset with optional parameters.</p>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple">
<li><strong>dataset</strong> – input dataset, which is an instance of
<a class="reference internal" href="pyspark.sql.html#pyspark.sql.DataFrame" title="pyspark.sql.DataFrame"><code class="xref py py-class docutils literal"><span class="pre">pyspark.sql.DataFrame</span></code></a></li>
<li><strong>params</strong> – an optional param map that overrides embedded
params.</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body"><p class="first last">transformed dataset</p>
</td>
</tr>
</tbody>
</table>
</dd></dl>
</dd></dl>
</div>
<div class="section" id="module-pyspark.ml.clustering">
<span id="pyspark-ml-clustering-module"></span><h2>pyspark.ml.clustering module<a class="headerlink" href="#module-pyspark.ml.clustering" title="Permalink to this headline">¶</a></h2>
<dl class="class">
<dt id="pyspark.ml.clustering.KMeans">
<em class="property">class </em><code class="descclassname">pyspark.ml.clustering.</code><code class="descname">KMeans</code><span class="sig-paren">(</span><em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/clustering.html#KMeans"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.clustering.KMeans" title="Permalink to this definition">¶</a></dt>
<dd><p>K-means clustering with support for multiple parallel runs and a k-means++ like initialization
mode (the k-means|| algorithm by Bahmani et al). When multiple concurrent runs are requested,
they are executed together with joint passes over the data for efficiency.</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="kn">from</span> <span class="nn">pyspark.mllib.linalg</span> <span class="kn">import</span> <span class="n">Vectors</span>
<span class="gp">>>> </span><span class="n">data</span> <span class="o">=</span> <span class="p">[(</span><span class="n">Vectors</span><span class="o">.</span><span class="n">dense</span><span class="p">([</span><span class="mf">0.0</span><span class="p">,</span> <span class="mf">0.0</span><span class="p">]),),</span> <span class="p">(</span><span class="n">Vectors</span><span class="o">.</span><span class="n">dense</span><span class="p">([</span><span class="mf">1.0</span><span class="p">,</span> <span class="mf">1.0</span><span class="p">]),),</span>
<span class="gp">... </span> <span class="p">(</span><span class="n">Vectors</span><span class="o">.</span><span class="n">dense</span><span class="p">([</span><span class="mf">9.0</span><span class="p">,</span> <span class="mf">8.0</span><span class="p">]),),</span> <span class="p">(</span><span class="n">Vectors</span><span class="o">.</span><span class="n">dense</span><span class="p">([</span><span class="mf">8.0</span><span class="p">,</span> <span class="mf">9.0</span><span class="p">]),)]</span>
<span class="gp">>>> </span><span class="n">df</span> <span class="o">=</span> <span class="n">sqlContext</span><span class="o">.</span><span class="n">createDataFrame</span><span class="p">(</span><span class="n">data</span><span class="p">,</span> <span class="p">[</span><span class="s">"features"</span><span class="p">])</span>
<span class="gp">>>> </span><span class="n">kmeans</span> <span class="o">=</span> <span class="n">KMeans</span><span class="p">(</span><span class="n">k</span><span class="o">=</span><span class="mi">2</span><span class="p">,</span> <span class="n">seed</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">model</span> <span class="o">=</span> <span class="n">kmeans</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">df</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">centers</span> <span class="o">=</span> <span class="n">model</span><span class="o">.</span><span class="n">clusterCenters</span><span class="p">()</span>
<span class="gp">>>> </span><span class="nb">len</span><span class="p">(</span><span class="n">centers</span><span class="p">)</span>
<span class="go">2</span>
<span class="gp">>>> </span><span class="n">transformed</span> <span class="o">=</span> <span class="n">model</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">df</span><span class="p">)</span><span class="o">.</span><span class="n">select</span><span class="p">(</span><span class="s">"features"</span><span class="p">,</span> <span class="s">"prediction"</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">rows</span> <span class="o">=</span> <span class="n">transformed</span><span class="o">.</span><span class="n">collect</span><span class="p">()</span>
<span class="gp">>>> </span><span class="n">rows</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">prediction</span> <span class="o">==</span> <span class="n">rows</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span><span class="o">.</span><span class="n">prediction</span>
<span class="go">True</span>
<span class="gp">>>> </span><span class="n">rows</span><span class="p">[</span><span class="mi">2</span><span class="p">]</span><span class="o">.</span><span class="n">prediction</span> <span class="o">==</span> <span class="n">rows</span><span class="p">[</span><span class="mi">3</span><span class="p">]</span><span class="o">.</span><span class="n">prediction</span>
<span class="go">True</span>
</pre></div>
</div>
<dl class="method">
<dt id="pyspark.ml.clustering.KMeans.copy">
<code class="descname">copy</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.clustering.KMeans.copy" title="Permalink to this definition">¶</a></dt>
<dd><p>Creates a copy of this instance with the same uid and some
extra params. The default implementation creates a
shallow copy using <code class="xref py py-func docutils literal"><span class="pre">copy.copy()</span></code>, and then copies the
embedded and extra parameters over and returns the copy.
Subclasses should override this method if the default approach
is not sufficient.
:param extra: Extra parameters to copy to the new instance
:return: Copy of this instance</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.clustering.KMeans.explainParam">
<code class="descname">explainParam</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.clustering.KMeans.explainParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Explains a single param and returns its name, doc, and optional
default value and user-supplied value in a string.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.clustering.KMeans.explainParams">
<code class="descname">explainParams</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.clustering.KMeans.explainParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns the documentation of all params with their optionally
default values and user-supplied values.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.clustering.KMeans.extractParamMap">
<code class="descname">extractParamMap</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.clustering.KMeans.extractParamMap" title="Permalink to this definition">¶</a></dt>
<dd><p>Extracts the embedded default param values and user-supplied
values, and then merges them with extra values from input into
a flat param map, where the latter value is used if there exist
conflicts, i.e., with ordering: default param values <
user-supplied values < extra.
:param extra: extra param values
:return: merged param map</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.clustering.KMeans.featuresCol">
<code class="descname">featuresCol</code><em class="property"> = Param(parent='undefined', name='featuresCol', doc='features column name')</em><a class="headerlink" href="#pyspark.ml.clustering.KMeans.featuresCol" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="method">
<dt id="pyspark.ml.clustering.KMeans.fit">
<code class="descname">fit</code><span class="sig-paren">(</span><em>dataset</em>, <em>params=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.clustering.KMeans.fit" title="Permalink to this definition">¶</a></dt>
<dd><p>Fits a model to the input dataset with optional parameters.</p>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple">
<li><strong>dataset</strong> – input dataset, which is an instance of
<a class="reference internal" href="pyspark.sql.html#pyspark.sql.DataFrame" title="pyspark.sql.DataFrame"><code class="xref py py-class docutils literal"><span class="pre">pyspark.sql.DataFrame</span></code></a></li>
<li><strong>params</strong> – an optional param map that overrides embedded
params. If a list/tuple of param maps is given,
this calls fit on each param map and returns a
list of models.</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body"><p class="first last">fitted model(s)</p>
</td>
</tr>
</tbody>
</table>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.clustering.KMeans.getFeaturesCol">
<code class="descname">getFeaturesCol</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.clustering.KMeans.getFeaturesCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of featuresCol or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.clustering.KMeans.getInitMode">
<code class="descname">getInitMode</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/clustering.html#KMeans.getInitMode"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.clustering.KMeans.getInitMode" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of <cite>initMode</cite></p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.clustering.KMeans.getInitSteps">
<code class="descname">getInitSteps</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/clustering.html#KMeans.getInitSteps"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.clustering.KMeans.getInitSteps" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of <cite>initSteps</cite></p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.clustering.KMeans.getK">
<code class="descname">getK</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/clustering.html#KMeans.getK"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.clustering.KMeans.getK" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of <cite>k</cite></p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.clustering.KMeans.getMaxIter">
<code class="descname">getMaxIter</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.clustering.KMeans.getMaxIter" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of maxIter or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.clustering.KMeans.getOrDefault">
<code class="descname">getOrDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.clustering.KMeans.getOrDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of a param in the user-supplied param map or its
default value. Raises an error if neither is set.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.clustering.KMeans.getParam">
<code class="descname">getParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.clustering.KMeans.getParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets a param by its name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.clustering.KMeans.getPredictionCol">
<code class="descname">getPredictionCol</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.clustering.KMeans.getPredictionCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of predictionCol or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.clustering.KMeans.getSeed">
<code class="descname">getSeed</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.clustering.KMeans.getSeed" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of seed or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.clustering.KMeans.getTol">
<code class="descname">getTol</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.clustering.KMeans.getTol" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of tol or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.clustering.KMeans.hasDefault">
<code class="descname">hasDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.clustering.KMeans.hasDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param has a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.clustering.KMeans.hasParam">
<code class="descname">hasParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.clustering.KMeans.hasParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Tests whether this instance contains a param with a given
(string) name.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.clustering.KMeans.initMode">
<code class="descname">initMode</code><em class="property"> = Param(parent='undefined', name='initMode', doc='the initialization algorithm. This can be either "random" to choose random points as initial cluster centers, or "k-means||" to use a parallel variant of k-means++')</em><a class="headerlink" href="#pyspark.ml.clustering.KMeans.initMode" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.clustering.KMeans.initSteps">
<code class="descname">initSteps</code><em class="property"> = Param(parent='undefined', name='initSteps', doc='steps for k-means initialization mode')</em><a class="headerlink" href="#pyspark.ml.clustering.KMeans.initSteps" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="method">
<dt id="pyspark.ml.clustering.KMeans.isDefined">
<code class="descname">isDefined</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.clustering.KMeans.isDefined" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user or has
a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.clustering.KMeans.isSet">
<code class="descname">isSet</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.clustering.KMeans.isSet" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.clustering.KMeans.k">
<code class="descname">k</code><em class="property"> = Param(parent='undefined', name='k', doc='number of clusters to create')</em><a class="headerlink" href="#pyspark.ml.clustering.KMeans.k" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.clustering.KMeans.maxIter">
<code class="descname">maxIter</code><em class="property"> = Param(parent='undefined', name='maxIter', doc='max number of iterations (>= 0)')</em><a class="headerlink" href="#pyspark.ml.clustering.KMeans.maxIter" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.clustering.KMeans.params">
<code class="descname">params</code><a class="headerlink" href="#pyspark.ml.clustering.KMeans.params" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns all params ordered by name. The default implementation
uses <code class="xref py py-func docutils literal"><span class="pre">dir()</span></code> to get all attributes of type
<code class="xref py py-class docutils literal"><span class="pre">Param</span></code>.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.clustering.KMeans.predictionCol">
<code class="descname">predictionCol</code><em class="property"> = Param(parent='undefined', name='predictionCol', doc='prediction column name')</em><a class="headerlink" href="#pyspark.ml.clustering.KMeans.predictionCol" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.clustering.KMeans.seed">
<code class="descname">seed</code><em class="property"> = Param(parent='undefined', name='seed', doc='random seed')</em><a class="headerlink" href="#pyspark.ml.clustering.KMeans.seed" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="method">
<dt id="pyspark.ml.clustering.KMeans.setFeaturesCol">
<code class="descname">setFeaturesCol</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.clustering.KMeans.setFeaturesCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.clustering.KMeans.featuresCol" title="pyspark.ml.clustering.KMeans.featuresCol"><code class="xref py py-attr docutils literal"><span class="pre">featuresCol</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.clustering.KMeans.setInitMode">
<code class="descname">setInitMode</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/clustering.html#KMeans.setInitMode"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.clustering.KMeans.setInitMode" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.clustering.KMeans.initMode" title="pyspark.ml.clustering.KMeans.initMode"><code class="xref py py-attr docutils literal"><span class="pre">initMode</span></code></a>.</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="n">algo</span> <span class="o">=</span> <span class="n">KMeans</span><span class="p">()</span>
<span class="gp">>>> </span><span class="n">algo</span><span class="o">.</span><span class="n">getInitMode</span><span class="p">()</span>
<span class="go">'k-means||'</span>
<span class="gp">>>> </span><span class="n">algo</span> <span class="o">=</span> <span class="n">algo</span><span class="o">.</span><span class="n">setInitMode</span><span class="p">(</span><span class="s">"random"</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">algo</span><span class="o">.</span><span class="n">getInitMode</span><span class="p">()</span>
<span class="go">'random'</span>
</pre></div>
</div>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.clustering.KMeans.setInitSteps">
<code class="descname">setInitSteps</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/clustering.html#KMeans.setInitSteps"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.clustering.KMeans.setInitSteps" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.clustering.KMeans.initSteps" title="pyspark.ml.clustering.KMeans.initSteps"><code class="xref py py-attr docutils literal"><span class="pre">initSteps</span></code></a>.</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="n">algo</span> <span class="o">=</span> <span class="n">KMeans</span><span class="p">()</span><span class="o">.</span><span class="n">setInitSteps</span><span class="p">(</span><span class="mi">10</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">algo</span><span class="o">.</span><span class="n">getInitSteps</span><span class="p">()</span>
<span class="go">10</span>
</pre></div>
</div>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.clustering.KMeans.setK">
<code class="descname">setK</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/clustering.html#KMeans.setK"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.clustering.KMeans.setK" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.clustering.KMeans.k" title="pyspark.ml.clustering.KMeans.k"><code class="xref py py-attr docutils literal"><span class="pre">k</span></code></a>.</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="n">algo</span> <span class="o">=</span> <span class="n">KMeans</span><span class="p">()</span><span class="o">.</span><span class="n">setK</span><span class="p">(</span><span class="mi">10</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">algo</span><span class="o">.</span><span class="n">getK</span><span class="p">()</span>
<span class="go">10</span>
</pre></div>
</div>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.clustering.KMeans.setMaxIter">
<code class="descname">setMaxIter</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.clustering.KMeans.setMaxIter" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.clustering.KMeans.maxIter" title="pyspark.ml.clustering.KMeans.maxIter"><code class="xref py py-attr docutils literal"><span class="pre">maxIter</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.clustering.KMeans.setParams">
<code class="descname">setParams</code><span class="sig-paren">(</span><em>self</em>, <em>featuresCol="features"</em>, <em>predictionCol="prediction"</em>, <em>k=2</em>, <em>initMode="k-means||"</em>, <em>initSteps=5</em>, <em>tol=1e-4</em>, <em>maxIter=20</em>, <em>seed=None</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/clustering.html#KMeans.setParams"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.clustering.KMeans.setParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets params for KMeans.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.clustering.KMeans.setPredictionCol">
<code class="descname">setPredictionCol</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.clustering.KMeans.setPredictionCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.clustering.KMeans.predictionCol" title="pyspark.ml.clustering.KMeans.predictionCol"><code class="xref py py-attr docutils literal"><span class="pre">predictionCol</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.clustering.KMeans.setSeed">
<code class="descname">setSeed</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.clustering.KMeans.setSeed" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.clustering.KMeans.seed" title="pyspark.ml.clustering.KMeans.seed"><code class="xref py py-attr docutils literal"><span class="pre">seed</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.clustering.KMeans.setTol">
<code class="descname">setTol</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.clustering.KMeans.setTol" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.clustering.KMeans.tol" title="pyspark.ml.clustering.KMeans.tol"><code class="xref py py-attr docutils literal"><span class="pre">tol</span></code></a>.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.clustering.KMeans.tol">
<code class="descname">tol</code><em class="property"> = Param(parent='undefined', name='tol', doc='the convergence tolerance for iterative algorithms')</em><a class="headerlink" href="#pyspark.ml.clustering.KMeans.tol" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
</dd></dl>
<dl class="class">
<dt id="pyspark.ml.clustering.KMeansModel">
<em class="property">class </em><code class="descclassname">pyspark.ml.clustering.</code><code class="descname">KMeansModel</code><span class="sig-paren">(</span><em>java_model</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/clustering.html#KMeansModel"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.clustering.KMeansModel" title="Permalink to this definition">¶</a></dt>
<dd><p>Model fitted by KMeans.</p>
<dl class="method">
<dt id="pyspark.ml.clustering.KMeansModel.clusterCenters">
<code class="descname">clusterCenters</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/clustering.html#KMeansModel.clusterCenters"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.clustering.KMeansModel.clusterCenters" title="Permalink to this definition">¶</a></dt>
<dd><p>Get the cluster centers, represented as a list of NumPy arrays.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.clustering.KMeansModel.copy">
<code class="descname">copy</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.clustering.KMeansModel.copy" title="Permalink to this definition">¶</a></dt>
<dd><p>Creates a copy of this instance with the same uid and some
extra params. This implementation first calls Params.copy and
then make a copy of the companion Java model with extra params.
So both the Python wrapper and the Java model get copied.
:param extra: Extra parameters to copy to the new instance
:return: Copy of this instance</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.clustering.KMeansModel.explainParam">
<code class="descname">explainParam</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.clustering.KMeansModel.explainParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Explains a single param and returns its name, doc, and optional
default value and user-supplied value in a string.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.clustering.KMeansModel.explainParams">
<code class="descname">explainParams</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.clustering.KMeansModel.explainParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns the documentation of all params with their optionally
default values and user-supplied values.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.clustering.KMeansModel.extractParamMap">
<code class="descname">extractParamMap</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.clustering.KMeansModel.extractParamMap" title="Permalink to this definition">¶</a></dt>
<dd><p>Extracts the embedded default param values and user-supplied
values, and then merges them with extra values from input into
a flat param map, where the latter value is used if there exist
conflicts, i.e., with ordering: default param values <
user-supplied values < extra.
:param extra: extra param values
:return: merged param map</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.clustering.KMeansModel.getOrDefault">
<code class="descname">getOrDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.clustering.KMeansModel.getOrDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of a param in the user-supplied param map or its
default value. Raises an error if neither is set.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.clustering.KMeansModel.getParam">
<code class="descname">getParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.clustering.KMeansModel.getParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets a param by its name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.clustering.KMeansModel.hasDefault">
<code class="descname">hasDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.clustering.KMeansModel.hasDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param has a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.clustering.KMeansModel.hasParam">
<code class="descname">hasParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.clustering.KMeansModel.hasParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Tests whether this instance contains a param with a given
(string) name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.clustering.KMeansModel.isDefined">
<code class="descname">isDefined</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.clustering.KMeansModel.isDefined" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user or has
a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.clustering.KMeansModel.isSet">
<code class="descname">isSet</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.clustering.KMeansModel.isSet" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.clustering.KMeansModel.params">
<code class="descname">params</code><a class="headerlink" href="#pyspark.ml.clustering.KMeansModel.params" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns all params ordered by name. The default implementation
uses <code class="xref py py-func docutils literal"><span class="pre">dir()</span></code> to get all attributes of type
<code class="xref py py-class docutils literal"><span class="pre">Param</span></code>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.clustering.KMeansModel.transform">
<code class="descname">transform</code><span class="sig-paren">(</span><em>dataset</em>, <em>params=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.clustering.KMeansModel.transform" title="Permalink to this definition">¶</a></dt>
<dd><p>Transforms the input dataset with optional parameters.</p>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple">
<li><strong>dataset</strong> – input dataset, which is an instance of
<a class="reference internal" href="pyspark.sql.html#pyspark.sql.DataFrame" title="pyspark.sql.DataFrame"><code class="xref py py-class docutils literal"><span class="pre">pyspark.sql.DataFrame</span></code></a></li>
<li><strong>params</strong> – an optional param map that overrides embedded
params.</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body"><p class="first last">transformed dataset</p>
</td>
</tr>
</tbody>
</table>
</dd></dl>
</dd></dl>
</div>
<div class="section" id="module-pyspark.ml.recommendation">
<span id="pyspark-ml-recommendation-module"></span><h2>pyspark.ml.recommendation module<a class="headerlink" href="#module-pyspark.ml.recommendation" title="Permalink to this headline">¶</a></h2>
<dl class="class">
<dt id="pyspark.ml.recommendation.ALS">
<em class="property">class </em><code class="descclassname">pyspark.ml.recommendation.</code><code class="descname">ALS</code><span class="sig-paren">(</span><em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/recommendation.html#ALS"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.recommendation.ALS" title="Permalink to this definition">¶</a></dt>
<dd><p>Alternating Least Squares (ALS) matrix factorization.</p>
<p>ALS attempts to estimate the ratings matrix <cite>R</cite> as the product of
two lower-rank matrices, <cite>X</cite> and <cite>Y</cite>, i.e. <cite>X * Yt = R</cite>. Typically
these approximations are called ‘factor’ matrices. The general
approach is iterative. During each iteration, one of the factor
matrices is held constant, while the other is solved for using least
squares. The newly-solved factor matrix is then held constant while
solving for the other factor matrix.</p>
<p>This is a blocked implementation of the ALS factorization algorithm
that groups the two sets of factors (referred to as “users” and
“products”) into blocks and reduces communication by only sending
one copy of each user vector to each product block on each
iteration, and only for the product blocks that need that user’s
feature vector. This is achieved by pre-computing some information
about the ratings matrix to determine the “out-links” of each user
(which blocks of products it will contribute to) and “in-link”
information for each product (which of the feature vectors it
receives from each user block it will depend on). This allows us to
send only an array of feature vectors between each user block and
product block, and have the product block find the users’ ratings
and update the products based on these messages.</p>
<p>For implicit preference data, the algorithm used is based on
“Collaborative Filtering for Implicit Feedback Datasets”, available
at <cite>http://dx.doi.org/10.1109/ICDM.2008.22</cite>, adapted for the blocked
approach used here.</p>
<p>Essentially instead of finding the low-rank approximations to the
rating matrix <cite>R</cite>, this finds the approximations for a preference
matrix <cite>P</cite> where the elements of <cite>P</cite> are 1 if r > 0 and 0 if r <= 0.
The ratings then act as ‘confidence’ values related to strength of
indicated user preferences rather than explicit ratings given to
items.</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="n">df</span> <span class="o">=</span> <span class="n">sqlContext</span><span class="o">.</span><span class="n">createDataFrame</span><span class="p">(</span>
<span class="gp">... </span> <span class="p">[(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mf">4.0</span><span class="p">),</span> <span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mf">2.0</span><span class="p">),</span> <span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mf">3.0</span><span class="p">),</span> <span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mf">4.0</span><span class="p">),</span> <span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mf">1.0</span><span class="p">),</span> <span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mf">5.0</span><span class="p">)],</span>
<span class="gp">... </span> <span class="p">[</span><span class="s">"user"</span><span class="p">,</span> <span class="s">"item"</span><span class="p">,</span> <span class="s">"rating"</span><span class="p">])</span>
<span class="gp">>>> </span><span class="n">als</span> <span class="o">=</span> <span class="n">ALS</span><span class="p">(</span><span class="n">rank</span><span class="o">=</span><span class="mi">10</span><span class="p">,</span> <span class="n">maxIter</span><span class="o">=</span><span class="mi">5</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">model</span> <span class="o">=</span> <span class="n">als</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">df</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">model</span><span class="o">.</span><span class="n">rank</span>
<span class="go">10</span>
<span class="gp">>>> </span><span class="n">model</span><span class="o">.</span><span class="n">userFactors</span><span class="o">.</span><span class="n">orderBy</span><span class="p">(</span><span class="s">"id"</span><span class="p">)</span><span class="o">.</span><span class="n">collect</span><span class="p">()</span>
<span class="go">[Row(id=0, features=[...]), Row(id=1, ...), Row(id=2, ...)]</span>
<span class="gp">>>> </span><span class="n">test</span> <span class="o">=</span> <span class="n">sqlContext</span><span class="o">.</span><span class="n">createDataFrame</span><span class="p">([(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">2</span><span class="p">),</span> <span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">0</span><span class="p">),</span> <span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="mi">0</span><span class="p">)],</span> <span class="p">[</span><span class="s">"user"</span><span class="p">,</span> <span class="s">"item"</span><span class="p">])</span>
<span class="gp">>>> </span><span class="n">predictions</span> <span class="o">=</span> <span class="nb">sorted</span><span class="p">(</span><span class="n">model</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">test</span><span class="p">)</span><span class="o">.</span><span class="n">collect</span><span class="p">(),</span> <span class="n">key</span><span class="o">=</span><span class="k">lambda</span> <span class="n">r</span><span class="p">:</span> <span class="n">r</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span>
<span class="gp">>>> </span><span class="n">predictions</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
<span class="go">Row(user=0, item=2, prediction=0.39...)</span>
<span class="gp">>>> </span><span class="n">predictions</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span>
<span class="go">Row(user=1, item=0, prediction=3.19...)</span>
<span class="gp">>>> </span><span class="n">predictions</span><span class="p">[</span><span class="mi">2</span><span class="p">]</span>
<span class="go">Row(user=2, item=0, prediction=-1.15...)</span>
</pre></div>
</div>
<dl class="attribute">
<dt id="pyspark.ml.recommendation.ALS.alpha">
<code class="descname">alpha</code><em class="property"> = Param(parent='undefined', name='alpha', doc='alpha for implicit preference')</em><a class="headerlink" href="#pyspark.ml.recommendation.ALS.alpha" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.recommendation.ALS.checkpointInterval">
<code class="descname">checkpointInterval</code><em class="property"> = Param(parent='undefined', name='checkpointInterval', doc='checkpoint interval (>= 1)')</em><a class="headerlink" href="#pyspark.ml.recommendation.ALS.checkpointInterval" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="method">
<dt id="pyspark.ml.recommendation.ALS.copy">
<code class="descname">copy</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.recommendation.ALS.copy" title="Permalink to this definition">¶</a></dt>
<dd><p>Creates a copy of this instance with the same uid and some
extra params. The default implementation creates a
shallow copy using <code class="xref py py-func docutils literal"><span class="pre">copy.copy()</span></code>, and then copies the
embedded and extra parameters over and returns the copy.
Subclasses should override this method if the default approach
is not sufficient.
:param extra: Extra parameters to copy to the new instance
:return: Copy of this instance</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.recommendation.ALS.explainParam">
<code class="descname">explainParam</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.recommendation.ALS.explainParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Explains a single param and returns its name, doc, and optional
default value and user-supplied value in a string.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.recommendation.ALS.explainParams">
<code class="descname">explainParams</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.recommendation.ALS.explainParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns the documentation of all params with their optionally
default values and user-supplied values.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.recommendation.ALS.extractParamMap">
<code class="descname">extractParamMap</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.recommendation.ALS.extractParamMap" title="Permalink to this definition">¶</a></dt>
<dd><p>Extracts the embedded default param values and user-supplied
values, and then merges them with extra values from input into
a flat param map, where the latter value is used if there exist
conflicts, i.e., with ordering: default param values <
user-supplied values < extra.
:param extra: extra param values
:return: merged param map</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.recommendation.ALS.fit">
<code class="descname">fit</code><span class="sig-paren">(</span><em>dataset</em>, <em>params=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.recommendation.ALS.fit" title="Permalink to this definition">¶</a></dt>
<dd><p>Fits a model to the input dataset with optional parameters.</p>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple">
<li><strong>dataset</strong> – input dataset, which is an instance of
<a class="reference internal" href="pyspark.sql.html#pyspark.sql.DataFrame" title="pyspark.sql.DataFrame"><code class="xref py py-class docutils literal"><span class="pre">pyspark.sql.DataFrame</span></code></a></li>
<li><strong>params</strong> – an optional param map that overrides embedded
params. If a list/tuple of param maps is given,
this calls fit on each param map and returns a
list of models.</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body"><p class="first last">fitted model(s)</p>
</td>
</tr>
</tbody>
</table>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.recommendation.ALS.getAlpha">
<code class="descname">getAlpha</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/recommendation.html#ALS.getAlpha"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.recommendation.ALS.getAlpha" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of alpha or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.recommendation.ALS.getCheckpointInterval">
<code class="descname">getCheckpointInterval</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.recommendation.ALS.getCheckpointInterval" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of checkpointInterval or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.recommendation.ALS.getImplicitPrefs">
<code class="descname">getImplicitPrefs</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/recommendation.html#ALS.getImplicitPrefs"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.recommendation.ALS.getImplicitPrefs" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of implicitPrefs or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.recommendation.ALS.getItemCol">
<code class="descname">getItemCol</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/recommendation.html#ALS.getItemCol"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.recommendation.ALS.getItemCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of itemCol or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.recommendation.ALS.getMaxIter">
<code class="descname">getMaxIter</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.recommendation.ALS.getMaxIter" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of maxIter or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.recommendation.ALS.getNonnegative">
<code class="descname">getNonnegative</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/recommendation.html#ALS.getNonnegative"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.recommendation.ALS.getNonnegative" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of nonnegative or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.recommendation.ALS.getNumItemBlocks">
<code class="descname">getNumItemBlocks</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/recommendation.html#ALS.getNumItemBlocks"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.recommendation.ALS.getNumItemBlocks" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of numItemBlocks or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.recommendation.ALS.getNumUserBlocks">
<code class="descname">getNumUserBlocks</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/recommendation.html#ALS.getNumUserBlocks"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.recommendation.ALS.getNumUserBlocks" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of numUserBlocks or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.recommendation.ALS.getOrDefault">
<code class="descname">getOrDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.recommendation.ALS.getOrDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of a param in the user-supplied param map or its
default value. Raises an error if neither is set.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.recommendation.ALS.getParam">
<code class="descname">getParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.recommendation.ALS.getParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets a param by its name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.recommendation.ALS.getPredictionCol">
<code class="descname">getPredictionCol</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.recommendation.ALS.getPredictionCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of predictionCol or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.recommendation.ALS.getRank">
<code class="descname">getRank</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/recommendation.html#ALS.getRank"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.recommendation.ALS.getRank" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of rank or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.recommendation.ALS.getRatingCol">
<code class="descname">getRatingCol</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/recommendation.html#ALS.getRatingCol"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.recommendation.ALS.getRatingCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of ratingCol or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.recommendation.ALS.getRegParam">
<code class="descname">getRegParam</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.recommendation.ALS.getRegParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of regParam or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.recommendation.ALS.getSeed">
<code class="descname">getSeed</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.recommendation.ALS.getSeed" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of seed or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.recommendation.ALS.getUserCol">
<code class="descname">getUserCol</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/recommendation.html#ALS.getUserCol"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.recommendation.ALS.getUserCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of userCol or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.recommendation.ALS.hasDefault">
<code class="descname">hasDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.recommendation.ALS.hasDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param has a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.recommendation.ALS.hasParam">
<code class="descname">hasParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.recommendation.ALS.hasParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Tests whether this instance contains a param with a given
(string) name.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.recommendation.ALS.implicitPrefs">
<code class="descname">implicitPrefs</code><em class="property"> = Param(parent='undefined', name='implicitPrefs', doc='whether to use implicit preference')</em><a class="headerlink" href="#pyspark.ml.recommendation.ALS.implicitPrefs" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="method">
<dt id="pyspark.ml.recommendation.ALS.isDefined">
<code class="descname">isDefined</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.recommendation.ALS.isDefined" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user or has
a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.recommendation.ALS.isSet">
<code class="descname">isSet</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.recommendation.ALS.isSet" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.recommendation.ALS.itemCol">
<code class="descname">itemCol</code><em class="property"> = Param(parent='undefined', name='itemCol', doc='column name for item ids')</em><a class="headerlink" href="#pyspark.ml.recommendation.ALS.itemCol" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.recommendation.ALS.maxIter">
<code class="descname">maxIter</code><em class="property"> = Param(parent='undefined', name='maxIter', doc='max number of iterations (>= 0)')</em><a class="headerlink" href="#pyspark.ml.recommendation.ALS.maxIter" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.recommendation.ALS.nonnegative">
<code class="descname">nonnegative</code><em class="property"> = Param(parent='undefined', name='nonnegative', doc='whether to use nonnegative constraint for least squares')</em><a class="headerlink" href="#pyspark.ml.recommendation.ALS.nonnegative" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.recommendation.ALS.numItemBlocks">
<code class="descname">numItemBlocks</code><em class="property"> = Param(parent='undefined', name='numItemBlocks', doc='number of item blocks')</em><a class="headerlink" href="#pyspark.ml.recommendation.ALS.numItemBlocks" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.recommendation.ALS.numUserBlocks">
<code class="descname">numUserBlocks</code><em class="property"> = Param(parent='undefined', name='numUserBlocks', doc='number of user blocks')</em><a class="headerlink" href="#pyspark.ml.recommendation.ALS.numUserBlocks" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.recommendation.ALS.params">
<code class="descname">params</code><a class="headerlink" href="#pyspark.ml.recommendation.ALS.params" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns all params ordered by name. The default implementation
uses <code class="xref py py-func docutils literal"><span class="pre">dir()</span></code> to get all attributes of type
<code class="xref py py-class docutils literal"><span class="pre">Param</span></code>.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.recommendation.ALS.predictionCol">
<code class="descname">predictionCol</code><em class="property"> = Param(parent='undefined', name='predictionCol', doc='prediction column name')</em><a class="headerlink" href="#pyspark.ml.recommendation.ALS.predictionCol" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.recommendation.ALS.rank">
<code class="descname">rank</code><em class="property"> = Param(parent='undefined', name='rank', doc='rank of the factorization')</em><a class="headerlink" href="#pyspark.ml.recommendation.ALS.rank" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.recommendation.ALS.ratingCol">
<code class="descname">ratingCol</code><em class="property"> = Param(parent='undefined', name='ratingCol', doc='column name for ratings')</em><a class="headerlink" href="#pyspark.ml.recommendation.ALS.ratingCol" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.recommendation.ALS.regParam">
<code class="descname">regParam</code><em class="property"> = Param(parent='undefined', name='regParam', doc='regularization parameter (>= 0)')</em><a class="headerlink" href="#pyspark.ml.recommendation.ALS.regParam" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.recommendation.ALS.seed">
<code class="descname">seed</code><em class="property"> = Param(parent='undefined', name='seed', doc='random seed')</em><a class="headerlink" href="#pyspark.ml.recommendation.ALS.seed" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="method">
<dt id="pyspark.ml.recommendation.ALS.setAlpha">
<code class="descname">setAlpha</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/recommendation.html#ALS.setAlpha"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.recommendation.ALS.setAlpha" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.recommendation.ALS.alpha" title="pyspark.ml.recommendation.ALS.alpha"><code class="xref py py-attr docutils literal"><span class="pre">alpha</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.recommendation.ALS.setCheckpointInterval">
<code class="descname">setCheckpointInterval</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.recommendation.ALS.setCheckpointInterval" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.recommendation.ALS.checkpointInterval" title="pyspark.ml.recommendation.ALS.checkpointInterval"><code class="xref py py-attr docutils literal"><span class="pre">checkpointInterval</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.recommendation.ALS.setImplicitPrefs">
<code class="descname">setImplicitPrefs</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/recommendation.html#ALS.setImplicitPrefs"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.recommendation.ALS.setImplicitPrefs" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.recommendation.ALS.implicitPrefs" title="pyspark.ml.recommendation.ALS.implicitPrefs"><code class="xref py py-attr docutils literal"><span class="pre">implicitPrefs</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.recommendation.ALS.setItemCol">
<code class="descname">setItemCol</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/recommendation.html#ALS.setItemCol"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.recommendation.ALS.setItemCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.recommendation.ALS.itemCol" title="pyspark.ml.recommendation.ALS.itemCol"><code class="xref py py-attr docutils literal"><span class="pre">itemCol</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.recommendation.ALS.setMaxIter">
<code class="descname">setMaxIter</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.recommendation.ALS.setMaxIter" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.recommendation.ALS.maxIter" title="pyspark.ml.recommendation.ALS.maxIter"><code class="xref py py-attr docutils literal"><span class="pre">maxIter</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.recommendation.ALS.setNonnegative">
<code class="descname">setNonnegative</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/recommendation.html#ALS.setNonnegative"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.recommendation.ALS.setNonnegative" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.recommendation.ALS.nonnegative" title="pyspark.ml.recommendation.ALS.nonnegative"><code class="xref py py-attr docutils literal"><span class="pre">nonnegative</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.recommendation.ALS.setNumBlocks">
<code class="descname">setNumBlocks</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/recommendation.html#ALS.setNumBlocks"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.recommendation.ALS.setNumBlocks" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets both <a class="reference internal" href="#pyspark.ml.recommendation.ALS.numUserBlocks" title="pyspark.ml.recommendation.ALS.numUserBlocks"><code class="xref py py-attr docutils literal"><span class="pre">numUserBlocks</span></code></a> and <a class="reference internal" href="#pyspark.ml.recommendation.ALS.numItemBlocks" title="pyspark.ml.recommendation.ALS.numItemBlocks"><code class="xref py py-attr docutils literal"><span class="pre">numItemBlocks</span></code></a> to the specific value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.recommendation.ALS.setNumItemBlocks">
<code class="descname">setNumItemBlocks</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/recommendation.html#ALS.setNumItemBlocks"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.recommendation.ALS.setNumItemBlocks" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.recommendation.ALS.numItemBlocks" title="pyspark.ml.recommendation.ALS.numItemBlocks"><code class="xref py py-attr docutils literal"><span class="pre">numItemBlocks</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.recommendation.ALS.setNumUserBlocks">
<code class="descname">setNumUserBlocks</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/recommendation.html#ALS.setNumUserBlocks"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.recommendation.ALS.setNumUserBlocks" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.recommendation.ALS.numUserBlocks" title="pyspark.ml.recommendation.ALS.numUserBlocks"><code class="xref py py-attr docutils literal"><span class="pre">numUserBlocks</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.recommendation.ALS.setParams">
<code class="descname">setParams</code><span class="sig-paren">(</span><em>self</em>, <em>rank=10</em>, <em>maxIter=10</em>, <em>regParam=0.1</em>, <em>numUserBlocks=10</em>, <em>numItemBlocks=10</em>, <em>implicitPrefs=False</em>, <em>alpha=1.0</em>, <em>userCol="user"</em>, <em>itemCol="item"</em>, <em>seed=None</em>, <em>ratingCol="rating"</em>, <em>nonnegative=False</em>, <em>checkpointInterval=10</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/recommendation.html#ALS.setParams"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.recommendation.ALS.setParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets params for ALS.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.recommendation.ALS.setPredictionCol">
<code class="descname">setPredictionCol</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.recommendation.ALS.setPredictionCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.recommendation.ALS.predictionCol" title="pyspark.ml.recommendation.ALS.predictionCol"><code class="xref py py-attr docutils literal"><span class="pre">predictionCol</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.recommendation.ALS.setRank">
<code class="descname">setRank</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/recommendation.html#ALS.setRank"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.recommendation.ALS.setRank" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.recommendation.ALS.rank" title="pyspark.ml.recommendation.ALS.rank"><code class="xref py py-attr docutils literal"><span class="pre">rank</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.recommendation.ALS.setRatingCol">
<code class="descname">setRatingCol</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/recommendation.html#ALS.setRatingCol"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.recommendation.ALS.setRatingCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.recommendation.ALS.ratingCol" title="pyspark.ml.recommendation.ALS.ratingCol"><code class="xref py py-attr docutils literal"><span class="pre">ratingCol</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.recommendation.ALS.setRegParam">
<code class="descname">setRegParam</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.recommendation.ALS.setRegParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.recommendation.ALS.regParam" title="pyspark.ml.recommendation.ALS.regParam"><code class="xref py py-attr docutils literal"><span class="pre">regParam</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.recommendation.ALS.setSeed">
<code class="descname">setSeed</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.recommendation.ALS.setSeed" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.recommendation.ALS.seed" title="pyspark.ml.recommendation.ALS.seed"><code class="xref py py-attr docutils literal"><span class="pre">seed</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.recommendation.ALS.setUserCol">
<code class="descname">setUserCol</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/recommendation.html#ALS.setUserCol"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.recommendation.ALS.setUserCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.recommendation.ALS.userCol" title="pyspark.ml.recommendation.ALS.userCol"><code class="xref py py-attr docutils literal"><span class="pre">userCol</span></code></a>.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.recommendation.ALS.userCol">
<code class="descname">userCol</code><em class="property"> = Param(parent='undefined', name='userCol', doc='column name for user ids')</em><a class="headerlink" href="#pyspark.ml.recommendation.ALS.userCol" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
</dd></dl>
<dl class="class">
<dt id="pyspark.ml.recommendation.ALSModel">
<em class="property">class </em><code class="descclassname">pyspark.ml.recommendation.</code><code class="descname">ALSModel</code><span class="sig-paren">(</span><em>java_model</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/recommendation.html#ALSModel"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.recommendation.ALSModel" title="Permalink to this definition">¶</a></dt>
<dd><p>Model fitted by ALS.</p>
<dl class="method">
<dt id="pyspark.ml.recommendation.ALSModel.copy">
<code class="descname">copy</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.recommendation.ALSModel.copy" title="Permalink to this definition">¶</a></dt>
<dd><p>Creates a copy of this instance with the same uid and some
extra params. This implementation first calls Params.copy and
then make a copy of the companion Java model with extra params.
So both the Python wrapper and the Java model get copied.
:param extra: Extra parameters to copy to the new instance
:return: Copy of this instance</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.recommendation.ALSModel.explainParam">
<code class="descname">explainParam</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.recommendation.ALSModel.explainParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Explains a single param and returns its name, doc, and optional
default value and user-supplied value in a string.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.recommendation.ALSModel.explainParams">
<code class="descname">explainParams</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.recommendation.ALSModel.explainParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns the documentation of all params with their optionally
default values and user-supplied values.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.recommendation.ALSModel.extractParamMap">
<code class="descname">extractParamMap</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.recommendation.ALSModel.extractParamMap" title="Permalink to this definition">¶</a></dt>
<dd><p>Extracts the embedded default param values and user-supplied
values, and then merges them with extra values from input into
a flat param map, where the latter value is used if there exist
conflicts, i.e., with ordering: default param values <
user-supplied values < extra.
:param extra: extra param values
:return: merged param map</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.recommendation.ALSModel.getOrDefault">
<code class="descname">getOrDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.recommendation.ALSModel.getOrDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of a param in the user-supplied param map or its
default value. Raises an error if neither is set.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.recommendation.ALSModel.getParam">
<code class="descname">getParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.recommendation.ALSModel.getParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets a param by its name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.recommendation.ALSModel.hasDefault">
<code class="descname">hasDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.recommendation.ALSModel.hasDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param has a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.recommendation.ALSModel.hasParam">
<code class="descname">hasParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.recommendation.ALSModel.hasParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Tests whether this instance contains a param with a given
(string) name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.recommendation.ALSModel.isDefined">
<code class="descname">isDefined</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.recommendation.ALSModel.isDefined" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user or has
a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.recommendation.ALSModel.isSet">
<code class="descname">isSet</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.recommendation.ALSModel.isSet" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.recommendation.ALSModel.itemFactors">
<code class="descname">itemFactors</code><a class="headerlink" href="#pyspark.ml.recommendation.ALSModel.itemFactors" title="Permalink to this definition">¶</a></dt>
<dd><p>a DataFrame that stores item factors in two columns: <cite>id</cite> and
<cite>features</cite></p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.recommendation.ALSModel.params">
<code class="descname">params</code><a class="headerlink" href="#pyspark.ml.recommendation.ALSModel.params" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns all params ordered by name. The default implementation
uses <code class="xref py py-func docutils literal"><span class="pre">dir()</span></code> to get all attributes of type
<code class="xref py py-class docutils literal"><span class="pre">Param</span></code>.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.recommendation.ALSModel.rank">
<code class="descname">rank</code><a class="headerlink" href="#pyspark.ml.recommendation.ALSModel.rank" title="Permalink to this definition">¶</a></dt>
<dd><p>rank of the matrix factorization model</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.recommendation.ALSModel.transform">
<code class="descname">transform</code><span class="sig-paren">(</span><em>dataset</em>, <em>params=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.recommendation.ALSModel.transform" title="Permalink to this definition">¶</a></dt>
<dd><p>Transforms the input dataset with optional parameters.</p>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple">
<li><strong>dataset</strong> – input dataset, which is an instance of
<a class="reference internal" href="pyspark.sql.html#pyspark.sql.DataFrame" title="pyspark.sql.DataFrame"><code class="xref py py-class docutils literal"><span class="pre">pyspark.sql.DataFrame</span></code></a></li>
<li><strong>params</strong> – an optional param map that overrides embedded
params.</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body"><p class="first last">transformed dataset</p>
</td>
</tr>
</tbody>
</table>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.recommendation.ALSModel.userFactors">
<code class="descname">userFactors</code><a class="headerlink" href="#pyspark.ml.recommendation.ALSModel.userFactors" title="Permalink to this definition">¶</a></dt>
<dd><p>a DataFrame that stores user factors in two columns: <cite>id</cite> and
<cite>features</cite></p>
</dd></dl>
</dd></dl>
</div>
<div class="section" id="module-pyspark.ml.regression">
<span id="pyspark-ml-regression-module"></span><h2>pyspark.ml.regression module<a class="headerlink" href="#module-pyspark.ml.regression" title="Permalink to this headline">¶</a></h2>
<dl class="class">
<dt id="pyspark.ml.regression.DecisionTreeRegressor">
<em class="property">class </em><code class="descclassname">pyspark.ml.regression.</code><code class="descname">DecisionTreeRegressor</code><span class="sig-paren">(</span><em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/regression.html#DecisionTreeRegressor"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.regression.DecisionTreeRegressor" title="Permalink to this definition">¶</a></dt>
<dd><p><cite>http://en.wikipedia.org/wiki/Decision_tree_learning Decision tree</cite>
learning algorithm for regression.
It supports both continuous and categorical features.</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="kn">from</span> <span class="nn">pyspark.mllib.linalg</span> <span class="kn">import</span> <span class="n">Vectors</span>
<span class="gp">>>> </span><span class="n">df</span> <span class="o">=</span> <span class="n">sqlContext</span><span class="o">.</span><span class="n">createDataFrame</span><span class="p">([</span>
<span class="gp">... </span> <span class="p">(</span><span class="mf">1.0</span><span class="p">,</span> <span class="n">Vectors</span><span class="o">.</span><span class="n">dense</span><span class="p">(</span><span class="mf">1.0</span><span class="p">)),</span>
<span class="gp">... </span> <span class="p">(</span><span class="mf">0.0</span><span class="p">,</span> <span class="n">Vectors</span><span class="o">.</span><span class="n">sparse</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="p">[],</span> <span class="p">[]))],</span> <span class="p">[</span><span class="s">"label"</span><span class="p">,</span> <span class="s">"features"</span><span class="p">])</span>
<span class="gp">>>> </span><span class="n">dt</span> <span class="o">=</span> <span class="n">DecisionTreeRegressor</span><span class="p">(</span><span class="n">maxDepth</span><span class="o">=</span><span class="mi">2</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">model</span> <span class="o">=</span> <span class="n">dt</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">df</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">model</span><span class="o">.</span><span class="n">depth</span>
<span class="go">1</span>
<span class="gp">>>> </span><span class="n">model</span><span class="o">.</span><span class="n">numNodes</span>
<span class="go">3</span>
<span class="gp">>>> </span><span class="n">test0</span> <span class="o">=</span> <span class="n">sqlContext</span><span class="o">.</span><span class="n">createDataFrame</span><span class="p">([(</span><span class="n">Vectors</span><span class="o">.</span><span class="n">dense</span><span class="p">(</span><span class="o">-</span><span class="mf">1.0</span><span class="p">),)],</span> <span class="p">[</span><span class="s">"features"</span><span class="p">])</span>
<span class="gp">>>> </span><span class="n">model</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">test0</span><span class="p">)</span><span class="o">.</span><span class="n">head</span><span class="p">()</span><span class="o">.</span><span class="n">prediction</span>
<span class="go">0.0</span>
<span class="gp">>>> </span><span class="n">test1</span> <span class="o">=</span> <span class="n">sqlContext</span><span class="o">.</span><span class="n">createDataFrame</span><span class="p">([(</span><span class="n">Vectors</span><span class="o">.</span><span class="n">sparse</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="p">[</span><span class="mf">1.0</span><span class="p">]),)],</span> <span class="p">[</span><span class="s">"features"</span><span class="p">])</span>
<span class="gp">>>> </span><span class="n">model</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">test1</span><span class="p">)</span><span class="o">.</span><span class="n">head</span><span class="p">()</span><span class="o">.</span><span class="n">prediction</span>
<span class="go">1.0</span>
</pre></div>
</div>
<dl class="attribute">
<dt id="pyspark.ml.regression.DecisionTreeRegressor.cacheNodeIds">
<code class="descname">cacheNodeIds</code><em class="property"> = Param(parent='undefined', name='cacheNodeIds', doc='If false, the algorithm will pass trees to executors to match instances with nodes. If true, the algorithm will cache node IDs for each instance. Caching can speed up training of deeper trees.')</em><a class="headerlink" href="#pyspark.ml.regression.DecisionTreeRegressor.cacheNodeIds" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.regression.DecisionTreeRegressor.checkpointInterval">
<code class="descname">checkpointInterval</code><em class="property"> = Param(parent='undefined', name='checkpointInterval', doc='checkpoint interval (>= 1)')</em><a class="headerlink" href="#pyspark.ml.regression.DecisionTreeRegressor.checkpointInterval" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.DecisionTreeRegressor.copy">
<code class="descname">copy</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.DecisionTreeRegressor.copy" title="Permalink to this definition">¶</a></dt>
<dd><p>Creates a copy of this instance with the same uid and some
extra params. The default implementation creates a
shallow copy using <code class="xref py py-func docutils literal"><span class="pre">copy.copy()</span></code>, and then copies the
embedded and extra parameters over and returns the copy.
Subclasses should override this method if the default approach
is not sufficient.
:param extra: Extra parameters to copy to the new instance
:return: Copy of this instance</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.DecisionTreeRegressor.explainParam">
<code class="descname">explainParam</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.DecisionTreeRegressor.explainParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Explains a single param and returns its name, doc, and optional
default value and user-supplied value in a string.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.DecisionTreeRegressor.explainParams">
<code class="descname">explainParams</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.DecisionTreeRegressor.explainParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns the documentation of all params with their optionally
default values and user-supplied values.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.DecisionTreeRegressor.extractParamMap">
<code class="descname">extractParamMap</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.DecisionTreeRegressor.extractParamMap" title="Permalink to this definition">¶</a></dt>
<dd><p>Extracts the embedded default param values and user-supplied
values, and then merges them with extra values from input into
a flat param map, where the latter value is used if there exist
conflicts, i.e., with ordering: default param values <
user-supplied values < extra.
:param extra: extra param values
:return: merged param map</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.regression.DecisionTreeRegressor.featuresCol">
<code class="descname">featuresCol</code><em class="property"> = Param(parent='undefined', name='featuresCol', doc='features column name')</em><a class="headerlink" href="#pyspark.ml.regression.DecisionTreeRegressor.featuresCol" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.DecisionTreeRegressor.fit">
<code class="descname">fit</code><span class="sig-paren">(</span><em>dataset</em>, <em>params=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.DecisionTreeRegressor.fit" title="Permalink to this definition">¶</a></dt>
<dd><p>Fits a model to the input dataset with optional parameters.</p>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple">
<li><strong>dataset</strong> – input dataset, which is an instance of
<a class="reference internal" href="pyspark.sql.html#pyspark.sql.DataFrame" title="pyspark.sql.DataFrame"><code class="xref py py-class docutils literal"><span class="pre">pyspark.sql.DataFrame</span></code></a></li>
<li><strong>params</strong> – an optional param map that overrides embedded
params. If a list/tuple of param maps is given,
this calls fit on each param map and returns a
list of models.</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body"><p class="first last">fitted model(s)</p>
</td>
</tr>
</tbody>
</table>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.DecisionTreeRegressor.getCacheNodeIds">
<code class="descname">getCacheNodeIds</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.DecisionTreeRegressor.getCacheNodeIds" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of cacheNodeIds or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.DecisionTreeRegressor.getCheckpointInterval">
<code class="descname">getCheckpointInterval</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.DecisionTreeRegressor.getCheckpointInterval" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of checkpointInterval or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.DecisionTreeRegressor.getFeaturesCol">
<code class="descname">getFeaturesCol</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.DecisionTreeRegressor.getFeaturesCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of featuresCol or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.DecisionTreeRegressor.getImpurity">
<code class="descname">getImpurity</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/regression.html#DecisionTreeRegressor.getImpurity"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.regression.DecisionTreeRegressor.getImpurity" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of impurity or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.DecisionTreeRegressor.getLabelCol">
<code class="descname">getLabelCol</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.DecisionTreeRegressor.getLabelCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of labelCol or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.DecisionTreeRegressor.getMaxBins">
<code class="descname">getMaxBins</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.DecisionTreeRegressor.getMaxBins" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of maxBins or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.DecisionTreeRegressor.getMaxDepth">
<code class="descname">getMaxDepth</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.DecisionTreeRegressor.getMaxDepth" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of maxDepth or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.DecisionTreeRegressor.getMaxMemoryInMB">
<code class="descname">getMaxMemoryInMB</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.DecisionTreeRegressor.getMaxMemoryInMB" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of maxMemoryInMB or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.DecisionTreeRegressor.getMinInfoGain">
<code class="descname">getMinInfoGain</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.DecisionTreeRegressor.getMinInfoGain" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of minInfoGain or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.DecisionTreeRegressor.getMinInstancesPerNode">
<code class="descname">getMinInstancesPerNode</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.DecisionTreeRegressor.getMinInstancesPerNode" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of minInstancesPerNode or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.DecisionTreeRegressor.getOrDefault">
<code class="descname">getOrDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.DecisionTreeRegressor.getOrDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of a param in the user-supplied param map or its
default value. Raises an error if neither is set.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.DecisionTreeRegressor.getParam">
<code class="descname">getParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.DecisionTreeRegressor.getParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets a param by its name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.DecisionTreeRegressor.getPredictionCol">
<code class="descname">getPredictionCol</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.DecisionTreeRegressor.getPredictionCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of predictionCol or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.DecisionTreeRegressor.hasDefault">
<code class="descname">hasDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.DecisionTreeRegressor.hasDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param has a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.DecisionTreeRegressor.hasParam">
<code class="descname">hasParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.DecisionTreeRegressor.hasParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Tests whether this instance contains a param with a given
(string) name.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.regression.DecisionTreeRegressor.impurity">
<code class="descname">impurity</code><em class="property"> = Param(parent='undefined', name='impurity', doc='Criterion used for information gain calculation (case-insensitive). Supported options: variance')</em><a class="headerlink" href="#pyspark.ml.regression.DecisionTreeRegressor.impurity" title="Permalink to this definition">¶</a></dt>
<dd><p>param for Criterion used for information gain calculation (case-insensitive).</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.DecisionTreeRegressor.isDefined">
<code class="descname">isDefined</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.DecisionTreeRegressor.isDefined" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user or has
a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.DecisionTreeRegressor.isSet">
<code class="descname">isSet</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.DecisionTreeRegressor.isSet" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.regression.DecisionTreeRegressor.labelCol">
<code class="descname">labelCol</code><em class="property"> = Param(parent='undefined', name='labelCol', doc='label column name')</em><a class="headerlink" href="#pyspark.ml.regression.DecisionTreeRegressor.labelCol" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.regression.DecisionTreeRegressor.maxBins">
<code class="descname">maxBins</code><em class="property"> = Param(parent='undefined', name='maxBins', doc='Max number of bins for discretizing continuous features. Must be >=2 and >= number of categories for any categorical feature.')</em><a class="headerlink" href="#pyspark.ml.regression.DecisionTreeRegressor.maxBins" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.regression.DecisionTreeRegressor.maxDepth">
<code class="descname">maxDepth</code><em class="property"> = Param(parent='undefined', name='maxDepth', doc='Maximum depth of the tree. (>= 0) E.g., depth 0 means 1 leaf node; depth 1 means 1 internal node + 2 leaf nodes.')</em><a class="headerlink" href="#pyspark.ml.regression.DecisionTreeRegressor.maxDepth" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.regression.DecisionTreeRegressor.maxMemoryInMB">
<code class="descname">maxMemoryInMB</code><em class="property"> = Param(parent='undefined', name='maxMemoryInMB', doc='Maximum memory in MB allocated to histogram aggregation.')</em><a class="headerlink" href="#pyspark.ml.regression.DecisionTreeRegressor.maxMemoryInMB" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.regression.DecisionTreeRegressor.minInfoGain">
<code class="descname">minInfoGain</code><em class="property"> = Param(parent='undefined', name='minInfoGain', doc='Minimum information gain for a split to be considered at a tree node.')</em><a class="headerlink" href="#pyspark.ml.regression.DecisionTreeRegressor.minInfoGain" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.regression.DecisionTreeRegressor.minInstancesPerNode">
<code class="descname">minInstancesPerNode</code><em class="property"> = Param(parent='undefined', name='minInstancesPerNode', doc='Minimum number of instances each child must have after split. If a split causes the left or right child to have fewer than minInstancesPerNode, the split will be discarded as invalid. Should be >= 1.')</em><a class="headerlink" href="#pyspark.ml.regression.DecisionTreeRegressor.minInstancesPerNode" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.regression.DecisionTreeRegressor.params">
<code class="descname">params</code><a class="headerlink" href="#pyspark.ml.regression.DecisionTreeRegressor.params" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns all params ordered by name. The default implementation
uses <code class="xref py py-func docutils literal"><span class="pre">dir()</span></code> to get all attributes of type
<code class="xref py py-class docutils literal"><span class="pre">Param</span></code>.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.regression.DecisionTreeRegressor.predictionCol">
<code class="descname">predictionCol</code><em class="property"> = Param(parent='undefined', name='predictionCol', doc='prediction column name')</em><a class="headerlink" href="#pyspark.ml.regression.DecisionTreeRegressor.predictionCol" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.DecisionTreeRegressor.setCacheNodeIds">
<code class="descname">setCacheNodeIds</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.DecisionTreeRegressor.setCacheNodeIds" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.regression.DecisionTreeRegressor.cacheNodeIds" title="pyspark.ml.regression.DecisionTreeRegressor.cacheNodeIds"><code class="xref py py-attr docutils literal"><span class="pre">cacheNodeIds</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.DecisionTreeRegressor.setCheckpointInterval">
<code class="descname">setCheckpointInterval</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.DecisionTreeRegressor.setCheckpointInterval" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.regression.DecisionTreeRegressor.checkpointInterval" title="pyspark.ml.regression.DecisionTreeRegressor.checkpointInterval"><code class="xref py py-attr docutils literal"><span class="pre">checkpointInterval</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.DecisionTreeRegressor.setFeaturesCol">
<code class="descname">setFeaturesCol</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.DecisionTreeRegressor.setFeaturesCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.regression.DecisionTreeRegressor.featuresCol" title="pyspark.ml.regression.DecisionTreeRegressor.featuresCol"><code class="xref py py-attr docutils literal"><span class="pre">featuresCol</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.DecisionTreeRegressor.setImpurity">
<code class="descname">setImpurity</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/regression.html#DecisionTreeRegressor.setImpurity"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.regression.DecisionTreeRegressor.setImpurity" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.regression.DecisionTreeRegressor.impurity" title="pyspark.ml.regression.DecisionTreeRegressor.impurity"><code class="xref py py-attr docutils literal"><span class="pre">impurity</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.DecisionTreeRegressor.setLabelCol">
<code class="descname">setLabelCol</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.DecisionTreeRegressor.setLabelCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.regression.DecisionTreeRegressor.labelCol" title="pyspark.ml.regression.DecisionTreeRegressor.labelCol"><code class="xref py py-attr docutils literal"><span class="pre">labelCol</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.DecisionTreeRegressor.setMaxBins">
<code class="descname">setMaxBins</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.DecisionTreeRegressor.setMaxBins" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.regression.DecisionTreeRegressor.maxBins" title="pyspark.ml.regression.DecisionTreeRegressor.maxBins"><code class="xref py py-attr docutils literal"><span class="pre">maxBins</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.DecisionTreeRegressor.setMaxDepth">
<code class="descname">setMaxDepth</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.DecisionTreeRegressor.setMaxDepth" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.regression.DecisionTreeRegressor.maxDepth" title="pyspark.ml.regression.DecisionTreeRegressor.maxDepth"><code class="xref py py-attr docutils literal"><span class="pre">maxDepth</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.DecisionTreeRegressor.setMaxMemoryInMB">
<code class="descname">setMaxMemoryInMB</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.DecisionTreeRegressor.setMaxMemoryInMB" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.regression.DecisionTreeRegressor.maxMemoryInMB" title="pyspark.ml.regression.DecisionTreeRegressor.maxMemoryInMB"><code class="xref py py-attr docutils literal"><span class="pre">maxMemoryInMB</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.DecisionTreeRegressor.setMinInfoGain">
<code class="descname">setMinInfoGain</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.DecisionTreeRegressor.setMinInfoGain" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.regression.DecisionTreeRegressor.minInfoGain" title="pyspark.ml.regression.DecisionTreeRegressor.minInfoGain"><code class="xref py py-attr docutils literal"><span class="pre">minInfoGain</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.DecisionTreeRegressor.setMinInstancesPerNode">
<code class="descname">setMinInstancesPerNode</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.DecisionTreeRegressor.setMinInstancesPerNode" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.regression.DecisionTreeRegressor.minInstancesPerNode" title="pyspark.ml.regression.DecisionTreeRegressor.minInstancesPerNode"><code class="xref py py-attr docutils literal"><span class="pre">minInstancesPerNode</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.DecisionTreeRegressor.setParams">
<code class="descname">setParams</code><span class="sig-paren">(</span><em>self</em>, <em>featuresCol="features"</em>, <em>labelCol="label"</em>, <em>predictionCol="prediction"</em>, <em>maxDepth=5</em>, <em>maxBins=32</em>, <em>minInstancesPerNode=1</em>, <em>minInfoGain=0.0</em>, <em>maxMemoryInMB=256</em>, <em>cacheNodeIds=False</em>, <em>checkpointInterval=10</em>, <em>impurity="variance"</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/regression.html#DecisionTreeRegressor.setParams"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.regression.DecisionTreeRegressor.setParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets params for the DecisionTreeRegressor.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.DecisionTreeRegressor.setPredictionCol">
<code class="descname">setPredictionCol</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.DecisionTreeRegressor.setPredictionCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.regression.DecisionTreeRegressor.predictionCol" title="pyspark.ml.regression.DecisionTreeRegressor.predictionCol"><code class="xref py py-attr docutils literal"><span class="pre">predictionCol</span></code></a>.</p>
</dd></dl>
</dd></dl>
<dl class="class">
<dt id="pyspark.ml.regression.DecisionTreeRegressionModel">
<em class="property">class </em><code class="descclassname">pyspark.ml.regression.</code><code class="descname">DecisionTreeRegressionModel</code><span class="sig-paren">(</span><em>java_model</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/regression.html#DecisionTreeRegressionModel"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.regression.DecisionTreeRegressionModel" title="Permalink to this definition">¶</a></dt>
<dd><p>Model fitted by DecisionTreeRegressor.</p>
<dl class="method">
<dt id="pyspark.ml.regression.DecisionTreeRegressionModel.copy">
<code class="descname">copy</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.DecisionTreeRegressionModel.copy" title="Permalink to this definition">¶</a></dt>
<dd><p>Creates a copy of this instance with the same uid and some
extra params. This implementation first calls Params.copy and
then make a copy of the companion Java model with extra params.
So both the Python wrapper and the Java model get copied.
:param extra: Extra parameters to copy to the new instance
:return: Copy of this instance</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.regression.DecisionTreeRegressionModel.depth">
<code class="descname">depth</code><a class="headerlink" href="#pyspark.ml.regression.DecisionTreeRegressionModel.depth" title="Permalink to this definition">¶</a></dt>
<dd><p>Return depth of the decision tree.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.DecisionTreeRegressionModel.explainParam">
<code class="descname">explainParam</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.DecisionTreeRegressionModel.explainParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Explains a single param and returns its name, doc, and optional
default value and user-supplied value in a string.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.DecisionTreeRegressionModel.explainParams">
<code class="descname">explainParams</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.DecisionTreeRegressionModel.explainParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns the documentation of all params with their optionally
default values and user-supplied values.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.DecisionTreeRegressionModel.extractParamMap">
<code class="descname">extractParamMap</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.DecisionTreeRegressionModel.extractParamMap" title="Permalink to this definition">¶</a></dt>
<dd><p>Extracts the embedded default param values and user-supplied
values, and then merges them with extra values from input into
a flat param map, where the latter value is used if there exist
conflicts, i.e., with ordering: default param values <
user-supplied values < extra.
:param extra: extra param values
:return: merged param map</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.DecisionTreeRegressionModel.getOrDefault">
<code class="descname">getOrDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.DecisionTreeRegressionModel.getOrDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of a param in the user-supplied param map or its
default value. Raises an error if neither is set.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.DecisionTreeRegressionModel.getParam">
<code class="descname">getParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.DecisionTreeRegressionModel.getParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets a param by its name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.DecisionTreeRegressionModel.hasDefault">
<code class="descname">hasDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.DecisionTreeRegressionModel.hasDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param has a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.DecisionTreeRegressionModel.hasParam">
<code class="descname">hasParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.DecisionTreeRegressionModel.hasParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Tests whether this instance contains a param with a given
(string) name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.DecisionTreeRegressionModel.isDefined">
<code class="descname">isDefined</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.DecisionTreeRegressionModel.isDefined" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user or has
a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.DecisionTreeRegressionModel.isSet">
<code class="descname">isSet</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.DecisionTreeRegressionModel.isSet" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.regression.DecisionTreeRegressionModel.numNodes">
<code class="descname">numNodes</code><a class="headerlink" href="#pyspark.ml.regression.DecisionTreeRegressionModel.numNodes" title="Permalink to this definition">¶</a></dt>
<dd><p>Return number of nodes of the decision tree.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.regression.DecisionTreeRegressionModel.params">
<code class="descname">params</code><a class="headerlink" href="#pyspark.ml.regression.DecisionTreeRegressionModel.params" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns all params ordered by name. The default implementation
uses <code class="xref py py-func docutils literal"><span class="pre">dir()</span></code> to get all attributes of type
<code class="xref py py-class docutils literal"><span class="pre">Param</span></code>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.DecisionTreeRegressionModel.transform">
<code class="descname">transform</code><span class="sig-paren">(</span><em>dataset</em>, <em>params=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.DecisionTreeRegressionModel.transform" title="Permalink to this definition">¶</a></dt>
<dd><p>Transforms the input dataset with optional parameters.</p>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple">
<li><strong>dataset</strong> – input dataset, which is an instance of
<a class="reference internal" href="pyspark.sql.html#pyspark.sql.DataFrame" title="pyspark.sql.DataFrame"><code class="xref py py-class docutils literal"><span class="pre">pyspark.sql.DataFrame</span></code></a></li>
<li><strong>params</strong> – an optional param map that overrides embedded
params.</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body"><p class="first last">transformed dataset</p>
</td>
</tr>
</tbody>
</table>
</dd></dl>
</dd></dl>
<dl class="class">
<dt id="pyspark.ml.regression.GBTRegressor">
<em class="property">class </em><code class="descclassname">pyspark.ml.regression.</code><code class="descname">GBTRegressor</code><span class="sig-paren">(</span><em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/regression.html#GBTRegressor"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.regression.GBTRegressor" title="Permalink to this definition">¶</a></dt>
<dd><p><cite>http://en.wikipedia.org/wiki/Gradient_boosting Gradient-Boosted Trees (GBTs)</cite>
learning algorithm for regression.
It supports both continuous and categorical features.</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="kn">from</span> <span class="nn">numpy</span> <span class="kn">import</span> <span class="n">allclose</span>
<span class="gp">>>> </span><span class="kn">from</span> <span class="nn">pyspark.mllib.linalg</span> <span class="kn">import</span> <span class="n">Vectors</span>
<span class="gp">>>> </span><span class="n">df</span> <span class="o">=</span> <span class="n">sqlContext</span><span class="o">.</span><span class="n">createDataFrame</span><span class="p">([</span>
<span class="gp">... </span> <span class="p">(</span><span class="mf">1.0</span><span class="p">,</span> <span class="n">Vectors</span><span class="o">.</span><span class="n">dense</span><span class="p">(</span><span class="mf">1.0</span><span class="p">)),</span>
<span class="gp">... </span> <span class="p">(</span><span class="mf">0.0</span><span class="p">,</span> <span class="n">Vectors</span><span class="o">.</span><span class="n">sparse</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="p">[],</span> <span class="p">[]))],</span> <span class="p">[</span><span class="s">"label"</span><span class="p">,</span> <span class="s">"features"</span><span class="p">])</span>
<span class="gp">>>> </span><span class="n">gbt</span> <span class="o">=</span> <span class="n">GBTRegressor</span><span class="p">(</span><span class="n">maxIter</span><span class="o">=</span><span class="mi">5</span><span class="p">,</span> <span class="n">maxDepth</span><span class="o">=</span><span class="mi">2</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">model</span> <span class="o">=</span> <span class="n">gbt</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">df</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">allclose</span><span class="p">(</span><span class="n">model</span><span class="o">.</span><span class="n">treeWeights</span><span class="p">,</span> <span class="p">[</span><span class="mf">1.0</span><span class="p">,</span> <span class="mf">0.1</span><span class="p">,</span> <span class="mf">0.1</span><span class="p">,</span> <span class="mf">0.1</span><span class="p">,</span> <span class="mf">0.1</span><span class="p">])</span>
<span class="go">True</span>
<span class="gp">>>> </span><span class="n">test0</span> <span class="o">=</span> <span class="n">sqlContext</span><span class="o">.</span><span class="n">createDataFrame</span><span class="p">([(</span><span class="n">Vectors</span><span class="o">.</span><span class="n">dense</span><span class="p">(</span><span class="o">-</span><span class="mf">1.0</span><span class="p">),)],</span> <span class="p">[</span><span class="s">"features"</span><span class="p">])</span>
<span class="gp">>>> </span><span class="n">model</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">test0</span><span class="p">)</span><span class="o">.</span><span class="n">head</span><span class="p">()</span><span class="o">.</span><span class="n">prediction</span>
<span class="go">0.0</span>
<span class="gp">>>> </span><span class="n">test1</span> <span class="o">=</span> <span class="n">sqlContext</span><span class="o">.</span><span class="n">createDataFrame</span><span class="p">([(</span><span class="n">Vectors</span><span class="o">.</span><span class="n">sparse</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="p">[</span><span class="mf">1.0</span><span class="p">]),)],</span> <span class="p">[</span><span class="s">"features"</span><span class="p">])</span>
<span class="gp">>>> </span><span class="n">model</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">test1</span><span class="p">)</span><span class="o">.</span><span class="n">head</span><span class="p">()</span><span class="o">.</span><span class="n">prediction</span>
<span class="go">1.0</span>
</pre></div>
</div>
<dl class="attribute">
<dt id="pyspark.ml.regression.GBTRegressor.cacheNodeIds">
<code class="descname">cacheNodeIds</code><em class="property"> = Param(parent='undefined', name='cacheNodeIds', doc='If false, the algorithm will pass trees to executors to match instances with nodes. If true, the algorithm will cache node IDs for each instance. Caching can speed up training of deeper trees.')</em><a class="headerlink" href="#pyspark.ml.regression.GBTRegressor.cacheNodeIds" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.regression.GBTRegressor.checkpointInterval">
<code class="descname">checkpointInterval</code><em class="property"> = Param(parent='undefined', name='checkpointInterval', doc='checkpoint interval (>= 1)')</em><a class="headerlink" href="#pyspark.ml.regression.GBTRegressor.checkpointInterval" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.GBTRegressor.copy">
<code class="descname">copy</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.GBTRegressor.copy" title="Permalink to this definition">¶</a></dt>
<dd><p>Creates a copy of this instance with the same uid and some
extra params. The default implementation creates a
shallow copy using <code class="xref py py-func docutils literal"><span class="pre">copy.copy()</span></code>, and then copies the
embedded and extra parameters over and returns the copy.
Subclasses should override this method if the default approach
is not sufficient.
:param extra: Extra parameters to copy to the new instance
:return: Copy of this instance</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.GBTRegressor.explainParam">
<code class="descname">explainParam</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.GBTRegressor.explainParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Explains a single param and returns its name, doc, and optional
default value and user-supplied value in a string.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.GBTRegressor.explainParams">
<code class="descname">explainParams</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.GBTRegressor.explainParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns the documentation of all params with their optionally
default values and user-supplied values.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.GBTRegressor.extractParamMap">
<code class="descname">extractParamMap</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.GBTRegressor.extractParamMap" title="Permalink to this definition">¶</a></dt>
<dd><p>Extracts the embedded default param values and user-supplied
values, and then merges them with extra values from input into
a flat param map, where the latter value is used if there exist
conflicts, i.e., with ordering: default param values <
user-supplied values < extra.
:param extra: extra param values
:return: merged param map</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.regression.GBTRegressor.featuresCol">
<code class="descname">featuresCol</code><em class="property"> = Param(parent='undefined', name='featuresCol', doc='features column name')</em><a class="headerlink" href="#pyspark.ml.regression.GBTRegressor.featuresCol" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.GBTRegressor.fit">
<code class="descname">fit</code><span class="sig-paren">(</span><em>dataset</em>, <em>params=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.GBTRegressor.fit" title="Permalink to this definition">¶</a></dt>
<dd><p>Fits a model to the input dataset with optional parameters.</p>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple">
<li><strong>dataset</strong> – input dataset, which is an instance of
<a class="reference internal" href="pyspark.sql.html#pyspark.sql.DataFrame" title="pyspark.sql.DataFrame"><code class="xref py py-class docutils literal"><span class="pre">pyspark.sql.DataFrame</span></code></a></li>
<li><strong>params</strong> – an optional param map that overrides embedded
params. If a list/tuple of param maps is given,
this calls fit on each param map and returns a
list of models.</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body"><p class="first last">fitted model(s)</p>
</td>
</tr>
</tbody>
</table>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.GBTRegressor.getCacheNodeIds">
<code class="descname">getCacheNodeIds</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.GBTRegressor.getCacheNodeIds" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of cacheNodeIds or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.GBTRegressor.getCheckpointInterval">
<code class="descname">getCheckpointInterval</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.GBTRegressor.getCheckpointInterval" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of checkpointInterval or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.GBTRegressor.getFeaturesCol">
<code class="descname">getFeaturesCol</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.GBTRegressor.getFeaturesCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of featuresCol or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.GBTRegressor.getLabelCol">
<code class="descname">getLabelCol</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.GBTRegressor.getLabelCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of labelCol or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.GBTRegressor.getLossType">
<code class="descname">getLossType</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/regression.html#GBTRegressor.getLossType"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.regression.GBTRegressor.getLossType" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of lossType or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.GBTRegressor.getMaxBins">
<code class="descname">getMaxBins</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.GBTRegressor.getMaxBins" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of maxBins or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.GBTRegressor.getMaxDepth">
<code class="descname">getMaxDepth</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.GBTRegressor.getMaxDepth" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of maxDepth or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.GBTRegressor.getMaxIter">
<code class="descname">getMaxIter</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.GBTRegressor.getMaxIter" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of maxIter or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.GBTRegressor.getMaxMemoryInMB">
<code class="descname">getMaxMemoryInMB</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.GBTRegressor.getMaxMemoryInMB" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of maxMemoryInMB or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.GBTRegressor.getMinInfoGain">
<code class="descname">getMinInfoGain</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.GBTRegressor.getMinInfoGain" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of minInfoGain or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.GBTRegressor.getMinInstancesPerNode">
<code class="descname">getMinInstancesPerNode</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.GBTRegressor.getMinInstancesPerNode" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of minInstancesPerNode or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.GBTRegressor.getOrDefault">
<code class="descname">getOrDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.GBTRegressor.getOrDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of a param in the user-supplied param map or its
default value. Raises an error if neither is set.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.GBTRegressor.getParam">
<code class="descname">getParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.GBTRegressor.getParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets a param by its name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.GBTRegressor.getPredictionCol">
<code class="descname">getPredictionCol</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.GBTRegressor.getPredictionCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of predictionCol or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.GBTRegressor.getStepSize">
<code class="descname">getStepSize</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/regression.html#GBTRegressor.getStepSize"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.regression.GBTRegressor.getStepSize" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of stepSize or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.GBTRegressor.getSubsamplingRate">
<code class="descname">getSubsamplingRate</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/regression.html#GBTRegressor.getSubsamplingRate"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.regression.GBTRegressor.getSubsamplingRate" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of subsamplingRate or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.GBTRegressor.hasDefault">
<code class="descname">hasDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.GBTRegressor.hasDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param has a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.GBTRegressor.hasParam">
<code class="descname">hasParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.GBTRegressor.hasParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Tests whether this instance contains a param with a given
(string) name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.GBTRegressor.isDefined">
<code class="descname">isDefined</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.GBTRegressor.isDefined" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user or has
a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.GBTRegressor.isSet">
<code class="descname">isSet</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.GBTRegressor.isSet" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.regression.GBTRegressor.labelCol">
<code class="descname">labelCol</code><em class="property"> = Param(parent='undefined', name='labelCol', doc='label column name')</em><a class="headerlink" href="#pyspark.ml.regression.GBTRegressor.labelCol" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.regression.GBTRegressor.lossType">
<code class="descname">lossType</code><em class="property"> = Param(parent='undefined', name='lossType', doc='Loss function which GBT tries to minimize (case-insensitive). Supported options: squared, absolute')</em><a class="headerlink" href="#pyspark.ml.regression.GBTRegressor.lossType" title="Permalink to this definition">¶</a></dt>
<dd><p>param for Loss function which GBT tries to minimize (case-insensitive).</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.regression.GBTRegressor.maxBins">
<code class="descname">maxBins</code><em class="property"> = Param(parent='undefined', name='maxBins', doc='Max number of bins for discretizing continuous features. Must be >=2 and >= number of categories for any categorical feature.')</em><a class="headerlink" href="#pyspark.ml.regression.GBTRegressor.maxBins" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.regression.GBTRegressor.maxDepth">
<code class="descname">maxDepth</code><em class="property"> = Param(parent='undefined', name='maxDepth', doc='Maximum depth of the tree. (>= 0) E.g., depth 0 means 1 leaf node; depth 1 means 1 internal node + 2 leaf nodes.')</em><a class="headerlink" href="#pyspark.ml.regression.GBTRegressor.maxDepth" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.regression.GBTRegressor.maxIter">
<code class="descname">maxIter</code><em class="property"> = Param(parent='undefined', name='maxIter', doc='max number of iterations (>= 0)')</em><a class="headerlink" href="#pyspark.ml.regression.GBTRegressor.maxIter" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.regression.GBTRegressor.maxMemoryInMB">
<code class="descname">maxMemoryInMB</code><em class="property"> = Param(parent='undefined', name='maxMemoryInMB', doc='Maximum memory in MB allocated to histogram aggregation.')</em><a class="headerlink" href="#pyspark.ml.regression.GBTRegressor.maxMemoryInMB" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.regression.GBTRegressor.minInfoGain">
<code class="descname">minInfoGain</code><em class="property"> = Param(parent='undefined', name='minInfoGain', doc='Minimum information gain for a split to be considered at a tree node.')</em><a class="headerlink" href="#pyspark.ml.regression.GBTRegressor.minInfoGain" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.regression.GBTRegressor.minInstancesPerNode">
<code class="descname">minInstancesPerNode</code><em class="property"> = Param(parent='undefined', name='minInstancesPerNode', doc='Minimum number of instances each child must have after split. If a split causes the left or right child to have fewer than minInstancesPerNode, the split will be discarded as invalid. Should be >= 1.')</em><a class="headerlink" href="#pyspark.ml.regression.GBTRegressor.minInstancesPerNode" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.regression.GBTRegressor.params">
<code class="descname">params</code><a class="headerlink" href="#pyspark.ml.regression.GBTRegressor.params" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns all params ordered by name. The default implementation
uses <code class="xref py py-func docutils literal"><span class="pre">dir()</span></code> to get all attributes of type
<code class="xref py py-class docutils literal"><span class="pre">Param</span></code>.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.regression.GBTRegressor.predictionCol">
<code class="descname">predictionCol</code><em class="property"> = Param(parent='undefined', name='predictionCol', doc='prediction column name')</em><a class="headerlink" href="#pyspark.ml.regression.GBTRegressor.predictionCol" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.GBTRegressor.setCacheNodeIds">
<code class="descname">setCacheNodeIds</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.GBTRegressor.setCacheNodeIds" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.regression.GBTRegressor.cacheNodeIds" title="pyspark.ml.regression.GBTRegressor.cacheNodeIds"><code class="xref py py-attr docutils literal"><span class="pre">cacheNodeIds</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.GBTRegressor.setCheckpointInterval">
<code class="descname">setCheckpointInterval</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.GBTRegressor.setCheckpointInterval" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.regression.GBTRegressor.checkpointInterval" title="pyspark.ml.regression.GBTRegressor.checkpointInterval"><code class="xref py py-attr docutils literal"><span class="pre">checkpointInterval</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.GBTRegressor.setFeaturesCol">
<code class="descname">setFeaturesCol</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.GBTRegressor.setFeaturesCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.regression.GBTRegressor.featuresCol" title="pyspark.ml.regression.GBTRegressor.featuresCol"><code class="xref py py-attr docutils literal"><span class="pre">featuresCol</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.GBTRegressor.setLabelCol">
<code class="descname">setLabelCol</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.GBTRegressor.setLabelCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.regression.GBTRegressor.labelCol" title="pyspark.ml.regression.GBTRegressor.labelCol"><code class="xref py py-attr docutils literal"><span class="pre">labelCol</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.GBTRegressor.setLossType">
<code class="descname">setLossType</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/regression.html#GBTRegressor.setLossType"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.regression.GBTRegressor.setLossType" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.regression.GBTRegressor.lossType" title="pyspark.ml.regression.GBTRegressor.lossType"><code class="xref py py-attr docutils literal"><span class="pre">lossType</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.GBTRegressor.setMaxBins">
<code class="descname">setMaxBins</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.GBTRegressor.setMaxBins" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.regression.GBTRegressor.maxBins" title="pyspark.ml.regression.GBTRegressor.maxBins"><code class="xref py py-attr docutils literal"><span class="pre">maxBins</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.GBTRegressor.setMaxDepth">
<code class="descname">setMaxDepth</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.GBTRegressor.setMaxDepth" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.regression.GBTRegressor.maxDepth" title="pyspark.ml.regression.GBTRegressor.maxDepth"><code class="xref py py-attr docutils literal"><span class="pre">maxDepth</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.GBTRegressor.setMaxIter">
<code class="descname">setMaxIter</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.GBTRegressor.setMaxIter" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.regression.GBTRegressor.maxIter" title="pyspark.ml.regression.GBTRegressor.maxIter"><code class="xref py py-attr docutils literal"><span class="pre">maxIter</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.GBTRegressor.setMaxMemoryInMB">
<code class="descname">setMaxMemoryInMB</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.GBTRegressor.setMaxMemoryInMB" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.regression.GBTRegressor.maxMemoryInMB" title="pyspark.ml.regression.GBTRegressor.maxMemoryInMB"><code class="xref py py-attr docutils literal"><span class="pre">maxMemoryInMB</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.GBTRegressor.setMinInfoGain">
<code class="descname">setMinInfoGain</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.GBTRegressor.setMinInfoGain" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.regression.GBTRegressor.minInfoGain" title="pyspark.ml.regression.GBTRegressor.minInfoGain"><code class="xref py py-attr docutils literal"><span class="pre">minInfoGain</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.GBTRegressor.setMinInstancesPerNode">
<code class="descname">setMinInstancesPerNode</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.GBTRegressor.setMinInstancesPerNode" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.regression.GBTRegressor.minInstancesPerNode" title="pyspark.ml.regression.GBTRegressor.minInstancesPerNode"><code class="xref py py-attr docutils literal"><span class="pre">minInstancesPerNode</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.GBTRegressor.setParams">
<code class="descname">setParams</code><span class="sig-paren">(</span><em>self</em>, <em>featuresCol="features"</em>, <em>labelCol="label"</em>, <em>predictionCol="prediction"</em>, <em>maxDepth=5</em>, <em>maxBins=32</em>, <em>minInstancesPerNode=1</em>, <em>minInfoGain=0.0</em>, <em>maxMemoryInMB=256</em>, <em>cacheNodeIds=False</em>, <em>checkpointInterval=10</em>, <em>lossType="squared"</em>, <em>maxIter=20</em>, <em>stepSize=0.1</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/regression.html#GBTRegressor.setParams"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.regression.GBTRegressor.setParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets params for Gradient Boosted Tree Regression.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.GBTRegressor.setPredictionCol">
<code class="descname">setPredictionCol</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.GBTRegressor.setPredictionCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.regression.GBTRegressor.predictionCol" title="pyspark.ml.regression.GBTRegressor.predictionCol"><code class="xref py py-attr docutils literal"><span class="pre">predictionCol</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.GBTRegressor.setStepSize">
<code class="descname">setStepSize</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/regression.html#GBTRegressor.setStepSize"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.regression.GBTRegressor.setStepSize" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.regression.GBTRegressor.stepSize" title="pyspark.ml.regression.GBTRegressor.stepSize"><code class="xref py py-attr docutils literal"><span class="pre">stepSize</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.GBTRegressor.setSubsamplingRate">
<code class="descname">setSubsamplingRate</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/regression.html#GBTRegressor.setSubsamplingRate"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.regression.GBTRegressor.setSubsamplingRate" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.regression.GBTRegressor.subsamplingRate" title="pyspark.ml.regression.GBTRegressor.subsamplingRate"><code class="xref py py-attr docutils literal"><span class="pre">subsamplingRate</span></code></a>.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.regression.GBTRegressor.stepSize">
<code class="descname">stepSize</code><em class="property"> = Param(parent='undefined', name='stepSize', doc='Step size (a.k.a. learning rate) in interval (0, 1] for shrinking the contribution of each estimator')</em><a class="headerlink" href="#pyspark.ml.regression.GBTRegressor.stepSize" title="Permalink to this definition">¶</a></dt>
<dd><p>Step size (a.k.a. learning rate) in interval (0, 1] for shrinking the contribution of</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.regression.GBTRegressor.subsamplingRate">
<code class="descname">subsamplingRate</code><em class="property"> = Param(parent='undefined', name='subsamplingRate', doc='Fraction of the training data used for learning each decision tree, in range (0, 1].')</em><a class="headerlink" href="#pyspark.ml.regression.GBTRegressor.subsamplingRate" title="Permalink to this definition">¶</a></dt>
<dd><p>Fraction of the training data used for learning each decision tree, in range (0, 1].</p>
</dd></dl>
</dd></dl>
<dl class="class">
<dt id="pyspark.ml.regression.GBTRegressionModel">
<em class="property">class </em><code class="descclassname">pyspark.ml.regression.</code><code class="descname">GBTRegressionModel</code><span class="sig-paren">(</span><em>java_model</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/regression.html#GBTRegressionModel"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.regression.GBTRegressionModel" title="Permalink to this definition">¶</a></dt>
<dd><p>Model fitted by GBTRegressor.</p>
<dl class="method">
<dt id="pyspark.ml.regression.GBTRegressionModel.copy">
<code class="descname">copy</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.GBTRegressionModel.copy" title="Permalink to this definition">¶</a></dt>
<dd><p>Creates a copy of this instance with the same uid and some
extra params. This implementation first calls Params.copy and
then make a copy of the companion Java model with extra params.
So both the Python wrapper and the Java model get copied.
:param extra: Extra parameters to copy to the new instance
:return: Copy of this instance</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.GBTRegressionModel.explainParam">
<code class="descname">explainParam</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.GBTRegressionModel.explainParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Explains a single param and returns its name, doc, and optional
default value and user-supplied value in a string.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.GBTRegressionModel.explainParams">
<code class="descname">explainParams</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.GBTRegressionModel.explainParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns the documentation of all params with their optionally
default values and user-supplied values.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.GBTRegressionModel.extractParamMap">
<code class="descname">extractParamMap</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.GBTRegressionModel.extractParamMap" title="Permalink to this definition">¶</a></dt>
<dd><p>Extracts the embedded default param values and user-supplied
values, and then merges them with extra values from input into
a flat param map, where the latter value is used if there exist
conflicts, i.e., with ordering: default param values <
user-supplied values < extra.
:param extra: extra param values
:return: merged param map</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.GBTRegressionModel.getOrDefault">
<code class="descname">getOrDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.GBTRegressionModel.getOrDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of a param in the user-supplied param map or its
default value. Raises an error if neither is set.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.GBTRegressionModel.getParam">
<code class="descname">getParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.GBTRegressionModel.getParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets a param by its name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.GBTRegressionModel.hasDefault">
<code class="descname">hasDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.GBTRegressionModel.hasDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param has a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.GBTRegressionModel.hasParam">
<code class="descname">hasParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.GBTRegressionModel.hasParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Tests whether this instance contains a param with a given
(string) name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.GBTRegressionModel.isDefined">
<code class="descname">isDefined</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.GBTRegressionModel.isDefined" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user or has
a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.GBTRegressionModel.isSet">
<code class="descname">isSet</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.GBTRegressionModel.isSet" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.regression.GBTRegressionModel.params">
<code class="descname">params</code><a class="headerlink" href="#pyspark.ml.regression.GBTRegressionModel.params" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns all params ordered by name. The default implementation
uses <code class="xref py py-func docutils literal"><span class="pre">dir()</span></code> to get all attributes of type
<code class="xref py py-class docutils literal"><span class="pre">Param</span></code>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.GBTRegressionModel.transform">
<code class="descname">transform</code><span class="sig-paren">(</span><em>dataset</em>, <em>params=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.GBTRegressionModel.transform" title="Permalink to this definition">¶</a></dt>
<dd><p>Transforms the input dataset with optional parameters.</p>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple">
<li><strong>dataset</strong> – input dataset, which is an instance of
<a class="reference internal" href="pyspark.sql.html#pyspark.sql.DataFrame" title="pyspark.sql.DataFrame"><code class="xref py py-class docutils literal"><span class="pre">pyspark.sql.DataFrame</span></code></a></li>
<li><strong>params</strong> – an optional param map that overrides embedded
params.</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body"><p class="first last">transformed dataset</p>
</td>
</tr>
</tbody>
</table>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.regression.GBTRegressionModel.treeWeights">
<code class="descname">treeWeights</code><a class="headerlink" href="#pyspark.ml.regression.GBTRegressionModel.treeWeights" title="Permalink to this definition">¶</a></dt>
<dd><p>Return the weights for each tree</p>
</dd></dl>
</dd></dl>
<dl class="class">
<dt id="pyspark.ml.regression.LinearRegression">
<em class="property">class </em><code class="descclassname">pyspark.ml.regression.</code><code class="descname">LinearRegression</code><span class="sig-paren">(</span><em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/regression.html#LinearRegression"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.regression.LinearRegression" title="Permalink to this definition">¶</a></dt>
<dd><p>Linear regression.</p>
<p>The learning objective is to minimize the squared error, with regularization.
The specific squared error loss function used is: L = 1/2n ||A weights - y||^2^</p>
<dl class="docutils">
<dt>This support multiple types of regularization:</dt>
<dd><ul class="first last simple">
<li>none (a.k.a. ordinary least squares)</li>
<li>L2 (ridge regression)</li>
<li>L1 (Lasso)</li>
<li>L2 + L1 (elastic net)</li>
</ul>
</dd>
</dl>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="kn">from</span> <span class="nn">pyspark.mllib.linalg</span> <span class="kn">import</span> <span class="n">Vectors</span>
<span class="gp">>>> </span><span class="n">df</span> <span class="o">=</span> <span class="n">sqlContext</span><span class="o">.</span><span class="n">createDataFrame</span><span class="p">([</span>
<span class="gp">... </span> <span class="p">(</span><span class="mf">1.0</span><span class="p">,</span> <span class="n">Vectors</span><span class="o">.</span><span class="n">dense</span><span class="p">(</span><span class="mf">1.0</span><span class="p">)),</span>
<span class="gp">... </span> <span class="p">(</span><span class="mf">0.0</span><span class="p">,</span> <span class="n">Vectors</span><span class="o">.</span><span class="n">sparse</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="p">[],</span> <span class="p">[]))],</span> <span class="p">[</span><span class="s">"label"</span><span class="p">,</span> <span class="s">"features"</span><span class="p">])</span>
<span class="gp">>>> </span><span class="n">lr</span> <span class="o">=</span> <span class="n">LinearRegression</span><span class="p">(</span><span class="n">maxIter</span><span class="o">=</span><span class="mi">5</span><span class="p">,</span> <span class="n">regParam</span><span class="o">=</span><span class="mf">0.0</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">model</span> <span class="o">=</span> <span class="n">lr</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">df</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">test0</span> <span class="o">=</span> <span class="n">sqlContext</span><span class="o">.</span><span class="n">createDataFrame</span><span class="p">([(</span><span class="n">Vectors</span><span class="o">.</span><span class="n">dense</span><span class="p">(</span><span class="o">-</span><span class="mf">1.0</span><span class="p">),)],</span> <span class="p">[</span><span class="s">"features"</span><span class="p">])</span>
<span class="gp">>>> </span><span class="n">model</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">test0</span><span class="p">)</span><span class="o">.</span><span class="n">head</span><span class="p">()</span><span class="o">.</span><span class="n">prediction</span>
<span class="go">-1.0</span>
<span class="gp">>>> </span><span class="n">model</span><span class="o">.</span><span class="n">weights</span>
<span class="go">DenseVector([1.0])</span>
<span class="gp">>>> </span><span class="n">model</span><span class="o">.</span><span class="n">intercept</span>
<span class="go">0.0</span>
<span class="gp">>>> </span><span class="n">test1</span> <span class="o">=</span> <span class="n">sqlContext</span><span class="o">.</span><span class="n">createDataFrame</span><span class="p">([(</span><span class="n">Vectors</span><span class="o">.</span><span class="n">sparse</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="p">[</span><span class="mf">1.0</span><span class="p">]),)],</span> <span class="p">[</span><span class="s">"features"</span><span class="p">])</span>
<span class="gp">>>> </span><span class="n">model</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">test1</span><span class="p">)</span><span class="o">.</span><span class="n">head</span><span class="p">()</span><span class="o">.</span><span class="n">prediction</span>
<span class="go">1.0</span>
<span class="gp">>>> </span><span class="n">lr</span><span class="o">.</span><span class="n">setParams</span><span class="p">(</span><span class="s">"vector"</span><span class="p">)</span>
<span class="gt">Traceback (most recent call last):</span>
<span class="o">...</span>
<span class="gr">TypeError</span>: <span class="n">Method setParams forces keyword arguments.</span>
</pre></div>
</div>
<dl class="method">
<dt id="pyspark.ml.regression.LinearRegression.copy">
<code class="descname">copy</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.LinearRegression.copy" title="Permalink to this definition">¶</a></dt>
<dd><p>Creates a copy of this instance with the same uid and some
extra params. The default implementation creates a
shallow copy using <code class="xref py py-func docutils literal"><span class="pre">copy.copy()</span></code>, and then copies the
embedded and extra parameters over and returns the copy.
Subclasses should override this method if the default approach
is not sufficient.
:param extra: Extra parameters to copy to the new instance
:return: Copy of this instance</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.regression.LinearRegression.elasticNetParam">
<code class="descname">elasticNetParam</code><em class="property"> = Param(parent='undefined', name='elasticNetParam', doc='the ElasticNet mixing parameter, in range [0, 1]. For alpha = 0, the penalty is an L2 penalty. For alpha = 1, it is an L1 penalty.')</em><a class="headerlink" href="#pyspark.ml.regression.LinearRegression.elasticNetParam" title="Permalink to this definition">¶</a></dt>
<dd><p>param for the ElasticNet mixing parameter, in range [0, 1]. For alpha = 0, the penalty</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.LinearRegression.explainParam">
<code class="descname">explainParam</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.LinearRegression.explainParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Explains a single param and returns its name, doc, and optional
default value and user-supplied value in a string.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.LinearRegression.explainParams">
<code class="descname">explainParams</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.LinearRegression.explainParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns the documentation of all params with their optionally
default values and user-supplied values.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.LinearRegression.extractParamMap">
<code class="descname">extractParamMap</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.LinearRegression.extractParamMap" title="Permalink to this definition">¶</a></dt>
<dd><p>Extracts the embedded default param values and user-supplied
values, and then merges them with extra values from input into
a flat param map, where the latter value is used if there exist
conflicts, i.e., with ordering: default param values <
user-supplied values < extra.
:param extra: extra param values
:return: merged param map</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.regression.LinearRegression.featuresCol">
<code class="descname">featuresCol</code><em class="property"> = Param(parent='undefined', name='featuresCol', doc='features column name')</em><a class="headerlink" href="#pyspark.ml.regression.LinearRegression.featuresCol" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.LinearRegression.fit">
<code class="descname">fit</code><span class="sig-paren">(</span><em>dataset</em>, <em>params=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.LinearRegression.fit" title="Permalink to this definition">¶</a></dt>
<dd><p>Fits a model to the input dataset with optional parameters.</p>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple">
<li><strong>dataset</strong> – input dataset, which is an instance of
<a class="reference internal" href="pyspark.sql.html#pyspark.sql.DataFrame" title="pyspark.sql.DataFrame"><code class="xref py py-class docutils literal"><span class="pre">pyspark.sql.DataFrame</span></code></a></li>
<li><strong>params</strong> – an optional param map that overrides embedded
params. If a list/tuple of param maps is given,
this calls fit on each param map and returns a
list of models.</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body"><p class="first last">fitted model(s)</p>
</td>
</tr>
</tbody>
</table>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.LinearRegression.getElasticNetParam">
<code class="descname">getElasticNetParam</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/regression.html#LinearRegression.getElasticNetParam"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.regression.LinearRegression.getElasticNetParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of elasticNetParam or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.LinearRegression.getFeaturesCol">
<code class="descname">getFeaturesCol</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.LinearRegression.getFeaturesCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of featuresCol or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.LinearRegression.getLabelCol">
<code class="descname">getLabelCol</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.LinearRegression.getLabelCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of labelCol or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.LinearRegression.getMaxIter">
<code class="descname">getMaxIter</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.LinearRegression.getMaxIter" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of maxIter or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.LinearRegression.getOrDefault">
<code class="descname">getOrDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.LinearRegression.getOrDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of a param in the user-supplied param map or its
default value. Raises an error if neither is set.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.LinearRegression.getParam">
<code class="descname">getParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.LinearRegression.getParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets a param by its name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.LinearRegression.getPredictionCol">
<code class="descname">getPredictionCol</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.LinearRegression.getPredictionCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of predictionCol or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.LinearRegression.getRegParam">
<code class="descname">getRegParam</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.LinearRegression.getRegParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of regParam or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.LinearRegression.getTol">
<code class="descname">getTol</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.LinearRegression.getTol" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of tol or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.LinearRegression.hasDefault">
<code class="descname">hasDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.LinearRegression.hasDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param has a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.LinearRegression.hasParam">
<code class="descname">hasParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.LinearRegression.hasParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Tests whether this instance contains a param with a given
(string) name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.LinearRegression.isDefined">
<code class="descname">isDefined</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.LinearRegression.isDefined" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user or has
a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.LinearRegression.isSet">
<code class="descname">isSet</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.LinearRegression.isSet" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.regression.LinearRegression.labelCol">
<code class="descname">labelCol</code><em class="property"> = Param(parent='undefined', name='labelCol', doc='label column name')</em><a class="headerlink" href="#pyspark.ml.regression.LinearRegression.labelCol" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.regression.LinearRegression.maxIter">
<code class="descname">maxIter</code><em class="property"> = Param(parent='undefined', name='maxIter', doc='max number of iterations (>= 0)')</em><a class="headerlink" href="#pyspark.ml.regression.LinearRegression.maxIter" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.regression.LinearRegression.params">
<code class="descname">params</code><a class="headerlink" href="#pyspark.ml.regression.LinearRegression.params" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns all params ordered by name. The default implementation
uses <code class="xref py py-func docutils literal"><span class="pre">dir()</span></code> to get all attributes of type
<code class="xref py py-class docutils literal"><span class="pre">Param</span></code>.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.regression.LinearRegression.predictionCol">
<code class="descname">predictionCol</code><em class="property"> = Param(parent='undefined', name='predictionCol', doc='prediction column name')</em><a class="headerlink" href="#pyspark.ml.regression.LinearRegression.predictionCol" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.regression.LinearRegression.regParam">
<code class="descname">regParam</code><em class="property"> = Param(parent='undefined', name='regParam', doc='regularization parameter (>= 0)')</em><a class="headerlink" href="#pyspark.ml.regression.LinearRegression.regParam" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.LinearRegression.setElasticNetParam">
<code class="descname">setElasticNetParam</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/regression.html#LinearRegression.setElasticNetParam"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.regression.LinearRegression.setElasticNetParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.regression.LinearRegression.elasticNetParam" title="pyspark.ml.regression.LinearRegression.elasticNetParam"><code class="xref py py-attr docutils literal"><span class="pre">elasticNetParam</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.LinearRegression.setFeaturesCol">
<code class="descname">setFeaturesCol</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.LinearRegression.setFeaturesCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.regression.LinearRegression.featuresCol" title="pyspark.ml.regression.LinearRegression.featuresCol"><code class="xref py py-attr docutils literal"><span class="pre">featuresCol</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.LinearRegression.setLabelCol">
<code class="descname">setLabelCol</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.LinearRegression.setLabelCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.regression.LinearRegression.labelCol" title="pyspark.ml.regression.LinearRegression.labelCol"><code class="xref py py-attr docutils literal"><span class="pre">labelCol</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.LinearRegression.setMaxIter">
<code class="descname">setMaxIter</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.LinearRegression.setMaxIter" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.regression.LinearRegression.maxIter" title="pyspark.ml.regression.LinearRegression.maxIter"><code class="xref py py-attr docutils literal"><span class="pre">maxIter</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.LinearRegression.setParams">
<code class="descname">setParams</code><span class="sig-paren">(</span><em>self</em>, <em>featuresCol="features"</em>, <em>labelCol="label"</em>, <em>predictionCol="prediction"</em>, <em>maxIter=100</em>, <em>regParam=0.0</em>, <em>elasticNetParam=0.0</em>, <em>tol=1e-6</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/regression.html#LinearRegression.setParams"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.regression.LinearRegression.setParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets params for linear regression.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.LinearRegression.setPredictionCol">
<code class="descname">setPredictionCol</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.LinearRegression.setPredictionCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.regression.LinearRegression.predictionCol" title="pyspark.ml.regression.LinearRegression.predictionCol"><code class="xref py py-attr docutils literal"><span class="pre">predictionCol</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.LinearRegression.setRegParam">
<code class="descname">setRegParam</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.LinearRegression.setRegParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.regression.LinearRegression.regParam" title="pyspark.ml.regression.LinearRegression.regParam"><code class="xref py py-attr docutils literal"><span class="pre">regParam</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.LinearRegression.setTol">
<code class="descname">setTol</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.LinearRegression.setTol" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.regression.LinearRegression.tol" title="pyspark.ml.regression.LinearRegression.tol"><code class="xref py py-attr docutils literal"><span class="pre">tol</span></code></a>.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.regression.LinearRegression.tol">
<code class="descname">tol</code><em class="property"> = Param(parent='undefined', name='tol', doc='the convergence tolerance for iterative algorithms')</em><a class="headerlink" href="#pyspark.ml.regression.LinearRegression.tol" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
</dd></dl>
<dl class="class">
<dt id="pyspark.ml.regression.LinearRegressionModel">
<em class="property">class </em><code class="descclassname">pyspark.ml.regression.</code><code class="descname">LinearRegressionModel</code><span class="sig-paren">(</span><em>java_model</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/regression.html#LinearRegressionModel"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.regression.LinearRegressionModel" title="Permalink to this definition">¶</a></dt>
<dd><p>Model fitted by LinearRegression.</p>
<dl class="method">
<dt id="pyspark.ml.regression.LinearRegressionModel.copy">
<code class="descname">copy</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.LinearRegressionModel.copy" title="Permalink to this definition">¶</a></dt>
<dd><p>Creates a copy of this instance with the same uid and some
extra params. This implementation first calls Params.copy and
then make a copy of the companion Java model with extra params.
So both the Python wrapper and the Java model get copied.
:param extra: Extra parameters to copy to the new instance
:return: Copy of this instance</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.LinearRegressionModel.explainParam">
<code class="descname">explainParam</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.LinearRegressionModel.explainParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Explains a single param and returns its name, doc, and optional
default value and user-supplied value in a string.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.LinearRegressionModel.explainParams">
<code class="descname">explainParams</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.LinearRegressionModel.explainParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns the documentation of all params with their optionally
default values and user-supplied values.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.LinearRegressionModel.extractParamMap">
<code class="descname">extractParamMap</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.LinearRegressionModel.extractParamMap" title="Permalink to this definition">¶</a></dt>
<dd><p>Extracts the embedded default param values and user-supplied
values, and then merges them with extra values from input into
a flat param map, where the latter value is used if there exist
conflicts, i.e., with ordering: default param values <
user-supplied values < extra.
:param extra: extra param values
:return: merged param map</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.LinearRegressionModel.getOrDefault">
<code class="descname">getOrDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.LinearRegressionModel.getOrDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of a param in the user-supplied param map or its
default value. Raises an error if neither is set.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.LinearRegressionModel.getParam">
<code class="descname">getParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.LinearRegressionModel.getParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets a param by its name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.LinearRegressionModel.hasDefault">
<code class="descname">hasDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.LinearRegressionModel.hasDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param has a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.LinearRegressionModel.hasParam">
<code class="descname">hasParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.LinearRegressionModel.hasParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Tests whether this instance contains a param with a given
(string) name.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.regression.LinearRegressionModel.intercept">
<code class="descname">intercept</code><a class="headerlink" href="#pyspark.ml.regression.LinearRegressionModel.intercept" title="Permalink to this definition">¶</a></dt>
<dd><p>Model intercept.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.LinearRegressionModel.isDefined">
<code class="descname">isDefined</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.LinearRegressionModel.isDefined" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user or has
a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.LinearRegressionModel.isSet">
<code class="descname">isSet</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.LinearRegressionModel.isSet" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.regression.LinearRegressionModel.params">
<code class="descname">params</code><a class="headerlink" href="#pyspark.ml.regression.LinearRegressionModel.params" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns all params ordered by name. The default implementation
uses <code class="xref py py-func docutils literal"><span class="pre">dir()</span></code> to get all attributes of type
<code class="xref py py-class docutils literal"><span class="pre">Param</span></code>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.LinearRegressionModel.transform">
<code class="descname">transform</code><span class="sig-paren">(</span><em>dataset</em>, <em>params=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.LinearRegressionModel.transform" title="Permalink to this definition">¶</a></dt>
<dd><p>Transforms the input dataset with optional parameters.</p>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple">
<li><strong>dataset</strong> – input dataset, which is an instance of
<a class="reference internal" href="pyspark.sql.html#pyspark.sql.DataFrame" title="pyspark.sql.DataFrame"><code class="xref py py-class docutils literal"><span class="pre">pyspark.sql.DataFrame</span></code></a></li>
<li><strong>params</strong> – an optional param map that overrides embedded
params.</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body"><p class="first last">transformed dataset</p>
</td>
</tr>
</tbody>
</table>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.regression.LinearRegressionModel.weights">
<code class="descname">weights</code><a class="headerlink" href="#pyspark.ml.regression.LinearRegressionModel.weights" title="Permalink to this definition">¶</a></dt>
<dd><p>Model weights.</p>
</dd></dl>
</dd></dl>
<dl class="class">
<dt id="pyspark.ml.regression.RandomForestRegressor">
<em class="property">class </em><code class="descclassname">pyspark.ml.regression.</code><code class="descname">RandomForestRegressor</code><span class="sig-paren">(</span><em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/regression.html#RandomForestRegressor"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.regression.RandomForestRegressor" title="Permalink to this definition">¶</a></dt>
<dd><p><cite>http://en.wikipedia.org/wiki/Random_forest Random Forest</cite>
learning algorithm for regression.
It supports both continuous and categorical features.</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="kn">from</span> <span class="nn">numpy</span> <span class="kn">import</span> <span class="n">allclose</span>
<span class="gp">>>> </span><span class="kn">from</span> <span class="nn">pyspark.mllib.linalg</span> <span class="kn">import</span> <span class="n">Vectors</span>
<span class="gp">>>> </span><span class="n">df</span> <span class="o">=</span> <span class="n">sqlContext</span><span class="o">.</span><span class="n">createDataFrame</span><span class="p">([</span>
<span class="gp">... </span> <span class="p">(</span><span class="mf">1.0</span><span class="p">,</span> <span class="n">Vectors</span><span class="o">.</span><span class="n">dense</span><span class="p">(</span><span class="mf">1.0</span><span class="p">)),</span>
<span class="gp">... </span> <span class="p">(</span><span class="mf">0.0</span><span class="p">,</span> <span class="n">Vectors</span><span class="o">.</span><span class="n">sparse</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="p">[],</span> <span class="p">[]))],</span> <span class="p">[</span><span class="s">"label"</span><span class="p">,</span> <span class="s">"features"</span><span class="p">])</span>
<span class="gp">>>> </span><span class="n">rf</span> <span class="o">=</span> <span class="n">RandomForestRegressor</span><span class="p">(</span><span class="n">numTrees</span><span class="o">=</span><span class="mi">2</span><span class="p">,</span> <span class="n">maxDepth</span><span class="o">=</span><span class="mi">2</span><span class="p">,</span> <span class="n">seed</span><span class="o">=</span><span class="mi">42</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">model</span> <span class="o">=</span> <span class="n">rf</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">df</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">allclose</span><span class="p">(</span><span class="n">model</span><span class="o">.</span><span class="n">treeWeights</span><span class="p">,</span> <span class="p">[</span><span class="mf">1.0</span><span class="p">,</span> <span class="mf">1.0</span><span class="p">])</span>
<span class="go">True</span>
<span class="gp">>>> </span><span class="n">test0</span> <span class="o">=</span> <span class="n">sqlContext</span><span class="o">.</span><span class="n">createDataFrame</span><span class="p">([(</span><span class="n">Vectors</span><span class="o">.</span><span class="n">dense</span><span class="p">(</span><span class="o">-</span><span class="mf">1.0</span><span class="p">),)],</span> <span class="p">[</span><span class="s">"features"</span><span class="p">])</span>
<span class="gp">>>> </span><span class="n">model</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">test0</span><span class="p">)</span><span class="o">.</span><span class="n">head</span><span class="p">()</span><span class="o">.</span><span class="n">prediction</span>
<span class="go">0.0</span>
<span class="gp">>>> </span><span class="n">test1</span> <span class="o">=</span> <span class="n">sqlContext</span><span class="o">.</span><span class="n">createDataFrame</span><span class="p">([(</span><span class="n">Vectors</span><span class="o">.</span><span class="n">sparse</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="p">[</span><span class="mf">1.0</span><span class="p">]),)],</span> <span class="p">[</span><span class="s">"features"</span><span class="p">])</span>
<span class="gp">>>> </span><span class="n">model</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">test1</span><span class="p">)</span><span class="o">.</span><span class="n">head</span><span class="p">()</span><span class="o">.</span><span class="n">prediction</span>
<span class="go">0.5</span>
</pre></div>
</div>
<dl class="attribute">
<dt id="pyspark.ml.regression.RandomForestRegressor.cacheNodeIds">
<code class="descname">cacheNodeIds</code><em class="property"> = Param(parent='undefined', name='cacheNodeIds', doc='If false, the algorithm will pass trees to executors to match instances with nodes. If true, the algorithm will cache node IDs for each instance. Caching can speed up training of deeper trees.')</em><a class="headerlink" href="#pyspark.ml.regression.RandomForestRegressor.cacheNodeIds" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.regression.RandomForestRegressor.checkpointInterval">
<code class="descname">checkpointInterval</code><em class="property"> = Param(parent='undefined', name='checkpointInterval', doc='checkpoint interval (>= 1)')</em><a class="headerlink" href="#pyspark.ml.regression.RandomForestRegressor.checkpointInterval" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.RandomForestRegressor.copy">
<code class="descname">copy</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.RandomForestRegressor.copy" title="Permalink to this definition">¶</a></dt>
<dd><p>Creates a copy of this instance with the same uid and some
extra params. The default implementation creates a
shallow copy using <code class="xref py py-func docutils literal"><span class="pre">copy.copy()</span></code>, and then copies the
embedded and extra parameters over and returns the copy.
Subclasses should override this method if the default approach
is not sufficient.
:param extra: Extra parameters to copy to the new instance
:return: Copy of this instance</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.RandomForestRegressor.explainParam">
<code class="descname">explainParam</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.RandomForestRegressor.explainParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Explains a single param and returns its name, doc, and optional
default value and user-supplied value in a string.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.RandomForestRegressor.explainParams">
<code class="descname">explainParams</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.RandomForestRegressor.explainParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns the documentation of all params with their optionally
default values and user-supplied values.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.RandomForestRegressor.extractParamMap">
<code class="descname">extractParamMap</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.RandomForestRegressor.extractParamMap" title="Permalink to this definition">¶</a></dt>
<dd><p>Extracts the embedded default param values and user-supplied
values, and then merges them with extra values from input into
a flat param map, where the latter value is used if there exist
conflicts, i.e., with ordering: default param values <
user-supplied values < extra.
:param extra: extra param values
:return: merged param map</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.regression.RandomForestRegressor.featureSubsetStrategy">
<code class="descname">featureSubsetStrategy</code><em class="property"> = Param(parent='undefined', name='featureSubsetStrategy', doc='The number of features to consider for splits at each tree node. Supported options: auto, all, onethird, sqrt, log2')</em><a class="headerlink" href="#pyspark.ml.regression.RandomForestRegressor.featureSubsetStrategy" title="Permalink to this definition">¶</a></dt>
<dd><p>param for The number of features to consider for splits at each tree node</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.regression.RandomForestRegressor.featuresCol">
<code class="descname">featuresCol</code><em class="property"> = Param(parent='undefined', name='featuresCol', doc='features column name')</em><a class="headerlink" href="#pyspark.ml.regression.RandomForestRegressor.featuresCol" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.RandomForestRegressor.fit">
<code class="descname">fit</code><span class="sig-paren">(</span><em>dataset</em>, <em>params=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.RandomForestRegressor.fit" title="Permalink to this definition">¶</a></dt>
<dd><p>Fits a model to the input dataset with optional parameters.</p>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple">
<li><strong>dataset</strong> – input dataset, which is an instance of
<a class="reference internal" href="pyspark.sql.html#pyspark.sql.DataFrame" title="pyspark.sql.DataFrame"><code class="xref py py-class docutils literal"><span class="pre">pyspark.sql.DataFrame</span></code></a></li>
<li><strong>params</strong> – an optional param map that overrides embedded
params. If a list/tuple of param maps is given,
this calls fit on each param map and returns a
list of models.</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body"><p class="first last">fitted model(s)</p>
</td>
</tr>
</tbody>
</table>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.RandomForestRegressor.getCacheNodeIds">
<code class="descname">getCacheNodeIds</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.RandomForestRegressor.getCacheNodeIds" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of cacheNodeIds or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.RandomForestRegressor.getCheckpointInterval">
<code class="descname">getCheckpointInterval</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.RandomForestRegressor.getCheckpointInterval" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of checkpointInterval or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.RandomForestRegressor.getFeatureSubsetStrategy">
<code class="descname">getFeatureSubsetStrategy</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/regression.html#RandomForestRegressor.getFeatureSubsetStrategy"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.regression.RandomForestRegressor.getFeatureSubsetStrategy" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of featureSubsetStrategy or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.RandomForestRegressor.getFeaturesCol">
<code class="descname">getFeaturesCol</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.RandomForestRegressor.getFeaturesCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of featuresCol or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.RandomForestRegressor.getImpurity">
<code class="descname">getImpurity</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/regression.html#RandomForestRegressor.getImpurity"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.regression.RandomForestRegressor.getImpurity" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of impurity or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.RandomForestRegressor.getLabelCol">
<code class="descname">getLabelCol</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.RandomForestRegressor.getLabelCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of labelCol or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.RandomForestRegressor.getMaxBins">
<code class="descname">getMaxBins</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.RandomForestRegressor.getMaxBins" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of maxBins or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.RandomForestRegressor.getMaxDepth">
<code class="descname">getMaxDepth</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.RandomForestRegressor.getMaxDepth" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of maxDepth or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.RandomForestRegressor.getMaxMemoryInMB">
<code class="descname">getMaxMemoryInMB</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.RandomForestRegressor.getMaxMemoryInMB" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of maxMemoryInMB or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.RandomForestRegressor.getMinInfoGain">
<code class="descname">getMinInfoGain</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.RandomForestRegressor.getMinInfoGain" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of minInfoGain or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.RandomForestRegressor.getMinInstancesPerNode">
<code class="descname">getMinInstancesPerNode</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.RandomForestRegressor.getMinInstancesPerNode" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of minInstancesPerNode or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.RandomForestRegressor.getNumTrees">
<code class="descname">getNumTrees</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/regression.html#RandomForestRegressor.getNumTrees"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.regression.RandomForestRegressor.getNumTrees" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of numTrees or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.RandomForestRegressor.getOrDefault">
<code class="descname">getOrDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.RandomForestRegressor.getOrDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of a param in the user-supplied param map or its
default value. Raises an error if neither is set.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.RandomForestRegressor.getParam">
<code class="descname">getParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.RandomForestRegressor.getParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets a param by its name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.RandomForestRegressor.getPredictionCol">
<code class="descname">getPredictionCol</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.RandomForestRegressor.getPredictionCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of predictionCol or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.RandomForestRegressor.getSeed">
<code class="descname">getSeed</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.RandomForestRegressor.getSeed" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of seed or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.RandomForestRegressor.getSubsamplingRate">
<code class="descname">getSubsamplingRate</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/regression.html#RandomForestRegressor.getSubsamplingRate"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.regression.RandomForestRegressor.getSubsamplingRate" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of subsamplingRate or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.RandomForestRegressor.hasDefault">
<code class="descname">hasDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.RandomForestRegressor.hasDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param has a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.RandomForestRegressor.hasParam">
<code class="descname">hasParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.RandomForestRegressor.hasParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Tests whether this instance contains a param with a given
(string) name.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.regression.RandomForestRegressor.impurity">
<code class="descname">impurity</code><em class="property"> = Param(parent='undefined', name='impurity', doc='Criterion used for information gain calculation (case-insensitive). Supported options: variance')</em><a class="headerlink" href="#pyspark.ml.regression.RandomForestRegressor.impurity" title="Permalink to this definition">¶</a></dt>
<dd><p>param for Criterion used for information gain calculation (case-insensitive).</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.RandomForestRegressor.isDefined">
<code class="descname">isDefined</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.RandomForestRegressor.isDefined" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user or has
a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.RandomForestRegressor.isSet">
<code class="descname">isSet</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.RandomForestRegressor.isSet" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.regression.RandomForestRegressor.labelCol">
<code class="descname">labelCol</code><em class="property"> = Param(parent='undefined', name='labelCol', doc='label column name')</em><a class="headerlink" href="#pyspark.ml.regression.RandomForestRegressor.labelCol" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.regression.RandomForestRegressor.maxBins">
<code class="descname">maxBins</code><em class="property"> = Param(parent='undefined', name='maxBins', doc='Max number of bins for discretizing continuous features. Must be >=2 and >= number of categories for any categorical feature.')</em><a class="headerlink" href="#pyspark.ml.regression.RandomForestRegressor.maxBins" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.regression.RandomForestRegressor.maxDepth">
<code class="descname">maxDepth</code><em class="property"> = Param(parent='undefined', name='maxDepth', doc='Maximum depth of the tree. (>= 0) E.g., depth 0 means 1 leaf node; depth 1 means 1 internal node + 2 leaf nodes.')</em><a class="headerlink" href="#pyspark.ml.regression.RandomForestRegressor.maxDepth" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.regression.RandomForestRegressor.maxMemoryInMB">
<code class="descname">maxMemoryInMB</code><em class="property"> = Param(parent='undefined', name='maxMemoryInMB', doc='Maximum memory in MB allocated to histogram aggregation.')</em><a class="headerlink" href="#pyspark.ml.regression.RandomForestRegressor.maxMemoryInMB" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.regression.RandomForestRegressor.minInfoGain">
<code class="descname">minInfoGain</code><em class="property"> = Param(parent='undefined', name='minInfoGain', doc='Minimum information gain for a split to be considered at a tree node.')</em><a class="headerlink" href="#pyspark.ml.regression.RandomForestRegressor.minInfoGain" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.regression.RandomForestRegressor.minInstancesPerNode">
<code class="descname">minInstancesPerNode</code><em class="property"> = Param(parent='undefined', name='minInstancesPerNode', doc='Minimum number of instances each child must have after split. If a split causes the left or right child to have fewer than minInstancesPerNode, the split will be discarded as invalid. Should be >= 1.')</em><a class="headerlink" href="#pyspark.ml.regression.RandomForestRegressor.minInstancesPerNode" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.regression.RandomForestRegressor.numTrees">
<code class="descname">numTrees</code><em class="property"> = Param(parent='undefined', name='numTrees', doc='Number of trees to train (>= 1)')</em><a class="headerlink" href="#pyspark.ml.regression.RandomForestRegressor.numTrees" title="Permalink to this definition">¶</a></dt>
<dd><p>param for Number of trees to train (>= 1)</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.regression.RandomForestRegressor.params">
<code class="descname">params</code><a class="headerlink" href="#pyspark.ml.regression.RandomForestRegressor.params" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns all params ordered by name. The default implementation
uses <code class="xref py py-func docutils literal"><span class="pre">dir()</span></code> to get all attributes of type
<code class="xref py py-class docutils literal"><span class="pre">Param</span></code>.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.regression.RandomForestRegressor.predictionCol">
<code class="descname">predictionCol</code><em class="property"> = Param(parent='undefined', name='predictionCol', doc='prediction column name')</em><a class="headerlink" href="#pyspark.ml.regression.RandomForestRegressor.predictionCol" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.regression.RandomForestRegressor.seed">
<code class="descname">seed</code><em class="property"> = Param(parent='undefined', name='seed', doc='random seed')</em><a class="headerlink" href="#pyspark.ml.regression.RandomForestRegressor.seed" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.RandomForestRegressor.setCacheNodeIds">
<code class="descname">setCacheNodeIds</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.RandomForestRegressor.setCacheNodeIds" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.regression.RandomForestRegressor.cacheNodeIds" title="pyspark.ml.regression.RandomForestRegressor.cacheNodeIds"><code class="xref py py-attr docutils literal"><span class="pre">cacheNodeIds</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.RandomForestRegressor.setCheckpointInterval">
<code class="descname">setCheckpointInterval</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.RandomForestRegressor.setCheckpointInterval" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.regression.RandomForestRegressor.checkpointInterval" title="pyspark.ml.regression.RandomForestRegressor.checkpointInterval"><code class="xref py py-attr docutils literal"><span class="pre">checkpointInterval</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.RandomForestRegressor.setFeatureSubsetStrategy">
<code class="descname">setFeatureSubsetStrategy</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/regression.html#RandomForestRegressor.setFeatureSubsetStrategy"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.regression.RandomForestRegressor.setFeatureSubsetStrategy" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.regression.RandomForestRegressor.featureSubsetStrategy" title="pyspark.ml.regression.RandomForestRegressor.featureSubsetStrategy"><code class="xref py py-attr docutils literal"><span class="pre">featureSubsetStrategy</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.RandomForestRegressor.setFeaturesCol">
<code class="descname">setFeaturesCol</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.RandomForestRegressor.setFeaturesCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.regression.RandomForestRegressor.featuresCol" title="pyspark.ml.regression.RandomForestRegressor.featuresCol"><code class="xref py py-attr docutils literal"><span class="pre">featuresCol</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.RandomForestRegressor.setImpurity">
<code class="descname">setImpurity</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/regression.html#RandomForestRegressor.setImpurity"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.regression.RandomForestRegressor.setImpurity" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.regression.RandomForestRegressor.impurity" title="pyspark.ml.regression.RandomForestRegressor.impurity"><code class="xref py py-attr docutils literal"><span class="pre">impurity</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.RandomForestRegressor.setLabelCol">
<code class="descname">setLabelCol</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.RandomForestRegressor.setLabelCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.regression.RandomForestRegressor.labelCol" title="pyspark.ml.regression.RandomForestRegressor.labelCol"><code class="xref py py-attr docutils literal"><span class="pre">labelCol</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.RandomForestRegressor.setMaxBins">
<code class="descname">setMaxBins</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.RandomForestRegressor.setMaxBins" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.regression.RandomForestRegressor.maxBins" title="pyspark.ml.regression.RandomForestRegressor.maxBins"><code class="xref py py-attr docutils literal"><span class="pre">maxBins</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.RandomForestRegressor.setMaxDepth">
<code class="descname">setMaxDepth</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.RandomForestRegressor.setMaxDepth" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.regression.RandomForestRegressor.maxDepth" title="pyspark.ml.regression.RandomForestRegressor.maxDepth"><code class="xref py py-attr docutils literal"><span class="pre">maxDepth</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.RandomForestRegressor.setMaxMemoryInMB">
<code class="descname">setMaxMemoryInMB</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.RandomForestRegressor.setMaxMemoryInMB" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.regression.RandomForestRegressor.maxMemoryInMB" title="pyspark.ml.regression.RandomForestRegressor.maxMemoryInMB"><code class="xref py py-attr docutils literal"><span class="pre">maxMemoryInMB</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.RandomForestRegressor.setMinInfoGain">
<code class="descname">setMinInfoGain</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.RandomForestRegressor.setMinInfoGain" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.regression.RandomForestRegressor.minInfoGain" title="pyspark.ml.regression.RandomForestRegressor.minInfoGain"><code class="xref py py-attr docutils literal"><span class="pre">minInfoGain</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.RandomForestRegressor.setMinInstancesPerNode">
<code class="descname">setMinInstancesPerNode</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.RandomForestRegressor.setMinInstancesPerNode" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.regression.RandomForestRegressor.minInstancesPerNode" title="pyspark.ml.regression.RandomForestRegressor.minInstancesPerNode"><code class="xref py py-attr docutils literal"><span class="pre">minInstancesPerNode</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.RandomForestRegressor.setNumTrees">
<code class="descname">setNumTrees</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/regression.html#RandomForestRegressor.setNumTrees"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.regression.RandomForestRegressor.setNumTrees" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.regression.RandomForestRegressor.numTrees" title="pyspark.ml.regression.RandomForestRegressor.numTrees"><code class="xref py py-attr docutils literal"><span class="pre">numTrees</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.RandomForestRegressor.setParams">
<code class="descname">setParams</code><span class="sig-paren">(</span><em>self</em>, <em>featuresCol="features"</em>, <em>labelCol="label"</em>, <em>predictionCol="prediction"</em>, <em>maxDepth=5</em>, <em>maxBins=32</em>, <em>minInstancesPerNode=1</em>, <em>minInfoGain=0.0</em>, <em>maxMemoryInMB=256</em>, <em>cacheNodeIds=False</em>, <em>checkpointInterval=10</em>, <em>seed=None</em>, <em>impurity="variance"</em>, <em>numTrees=20</em>, <em>featureSubsetStrategy="auto"</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/regression.html#RandomForestRegressor.setParams"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.regression.RandomForestRegressor.setParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets params for linear regression.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.RandomForestRegressor.setPredictionCol">
<code class="descname">setPredictionCol</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.RandomForestRegressor.setPredictionCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.regression.RandomForestRegressor.predictionCol" title="pyspark.ml.regression.RandomForestRegressor.predictionCol"><code class="xref py py-attr docutils literal"><span class="pre">predictionCol</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.RandomForestRegressor.setSeed">
<code class="descname">setSeed</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.RandomForestRegressor.setSeed" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.regression.RandomForestRegressor.seed" title="pyspark.ml.regression.RandomForestRegressor.seed"><code class="xref py py-attr docutils literal"><span class="pre">seed</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.RandomForestRegressor.setSubsamplingRate">
<code class="descname">setSubsamplingRate</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/regression.html#RandomForestRegressor.setSubsamplingRate"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.regression.RandomForestRegressor.setSubsamplingRate" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.regression.RandomForestRegressor.subsamplingRate" title="pyspark.ml.regression.RandomForestRegressor.subsamplingRate"><code class="xref py py-attr docutils literal"><span class="pre">subsamplingRate</span></code></a>.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.regression.RandomForestRegressor.subsamplingRate">
<code class="descname">subsamplingRate</code><em class="property"> = Param(parent='undefined', name='subsamplingRate', doc='Fraction of the training data used for learning each decision tree, in range (0, 1].')</em><a class="headerlink" href="#pyspark.ml.regression.RandomForestRegressor.subsamplingRate" title="Permalink to this definition">¶</a></dt>
<dd><p>param for Fraction of the training data used for learning each decision tree,</p>
</dd></dl>
</dd></dl>
<dl class="class">
<dt id="pyspark.ml.regression.RandomForestRegressionModel">
<em class="property">class </em><code class="descclassname">pyspark.ml.regression.</code><code class="descname">RandomForestRegressionModel</code><span class="sig-paren">(</span><em>java_model</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/regression.html#RandomForestRegressionModel"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.regression.RandomForestRegressionModel" title="Permalink to this definition">¶</a></dt>
<dd><p>Model fitted by RandomForestRegressor.</p>
<dl class="method">
<dt id="pyspark.ml.regression.RandomForestRegressionModel.copy">
<code class="descname">copy</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.RandomForestRegressionModel.copy" title="Permalink to this definition">¶</a></dt>
<dd><p>Creates a copy of this instance with the same uid and some
extra params. This implementation first calls Params.copy and
then make a copy of the companion Java model with extra params.
So both the Python wrapper and the Java model get copied.
:param extra: Extra parameters to copy to the new instance
:return: Copy of this instance</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.RandomForestRegressionModel.explainParam">
<code class="descname">explainParam</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.RandomForestRegressionModel.explainParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Explains a single param and returns its name, doc, and optional
default value and user-supplied value in a string.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.RandomForestRegressionModel.explainParams">
<code class="descname">explainParams</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.RandomForestRegressionModel.explainParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns the documentation of all params with their optionally
default values and user-supplied values.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.RandomForestRegressionModel.extractParamMap">
<code class="descname">extractParamMap</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.RandomForestRegressionModel.extractParamMap" title="Permalink to this definition">¶</a></dt>
<dd><p>Extracts the embedded default param values and user-supplied
values, and then merges them with extra values from input into
a flat param map, where the latter value is used if there exist
conflicts, i.e., with ordering: default param values <
user-supplied values < extra.
:param extra: extra param values
:return: merged param map</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.RandomForestRegressionModel.getOrDefault">
<code class="descname">getOrDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.RandomForestRegressionModel.getOrDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of a param in the user-supplied param map or its
default value. Raises an error if neither is set.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.RandomForestRegressionModel.getParam">
<code class="descname">getParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.RandomForestRegressionModel.getParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets a param by its name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.RandomForestRegressionModel.hasDefault">
<code class="descname">hasDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.RandomForestRegressionModel.hasDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param has a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.RandomForestRegressionModel.hasParam">
<code class="descname">hasParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.RandomForestRegressionModel.hasParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Tests whether this instance contains a param with a given
(string) name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.RandomForestRegressionModel.isDefined">
<code class="descname">isDefined</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.RandomForestRegressionModel.isDefined" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user or has
a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.RandomForestRegressionModel.isSet">
<code class="descname">isSet</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.RandomForestRegressionModel.isSet" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.regression.RandomForestRegressionModel.params">
<code class="descname">params</code><a class="headerlink" href="#pyspark.ml.regression.RandomForestRegressionModel.params" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns all params ordered by name. The default implementation
uses <code class="xref py py-func docutils literal"><span class="pre">dir()</span></code> to get all attributes of type
<code class="xref py py-class docutils literal"><span class="pre">Param</span></code>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.regression.RandomForestRegressionModel.transform">
<code class="descname">transform</code><span class="sig-paren">(</span><em>dataset</em>, <em>params=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.regression.RandomForestRegressionModel.transform" title="Permalink to this definition">¶</a></dt>
<dd><p>Transforms the input dataset with optional parameters.</p>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple">
<li><strong>dataset</strong> – input dataset, which is an instance of
<a class="reference internal" href="pyspark.sql.html#pyspark.sql.DataFrame" title="pyspark.sql.DataFrame"><code class="xref py py-class docutils literal"><span class="pre">pyspark.sql.DataFrame</span></code></a></li>
<li><strong>params</strong> – an optional param map that overrides embedded
params.</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body"><p class="first last">transformed dataset</p>
</td>
</tr>
</tbody>
</table>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.regression.RandomForestRegressionModel.treeWeights">
<code class="descname">treeWeights</code><a class="headerlink" href="#pyspark.ml.regression.RandomForestRegressionModel.treeWeights" title="Permalink to this definition">¶</a></dt>
<dd><p>Return the weights for each tree</p>
</dd></dl>
</dd></dl>
</div>
<div class="section" id="module-pyspark.ml.tuning">
<span id="pyspark-ml-tuning-module"></span><h2>pyspark.ml.tuning module<a class="headerlink" href="#module-pyspark.ml.tuning" title="Permalink to this headline">¶</a></h2>
<dl class="class">
<dt id="pyspark.ml.tuning.ParamGridBuilder">
<em class="property">class </em><code class="descclassname">pyspark.ml.tuning.</code><code class="descname">ParamGridBuilder</code><a class="reference internal" href="_modules/pyspark/ml/tuning.html#ParamGridBuilder"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.tuning.ParamGridBuilder" title="Permalink to this definition">¶</a></dt>
<dd><p>Builder for a param grid used in grid search-based model selection.</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="kn">from</span> <span class="nn">pyspark.ml.classification</span> <span class="kn">import</span> <span class="n">LogisticRegression</span>
<span class="gp">>>> </span><span class="n">lr</span> <span class="o">=</span> <span class="n">LogisticRegression</span><span class="p">()</span>
<span class="gp">>>> </span><span class="n">output</span> <span class="o">=</span> <span class="n">ParamGridBuilder</span><span class="p">()</span> \
<span class="gp">... </span> <span class="o">.</span><span class="n">baseOn</span><span class="p">({</span><span class="n">lr</span><span class="o">.</span><span class="n">labelCol</span><span class="p">:</span> <span class="s">'l'</span><span class="p">})</span> \
<span class="gp">... </span> <span class="o">.</span><span class="n">baseOn</span><span class="p">([</span><span class="n">lr</span><span class="o">.</span><span class="n">predictionCol</span><span class="p">,</span> <span class="s">'p'</span><span class="p">])</span> \
<span class="gp">... </span> <span class="o">.</span><span class="n">addGrid</span><span class="p">(</span><span class="n">lr</span><span class="o">.</span><span class="n">regParam</span><span class="p">,</span> <span class="p">[</span><span class="mf">1.0</span><span class="p">,</span> <span class="mf">2.0</span><span class="p">])</span> \
<span class="gp">... </span> <span class="o">.</span><span class="n">addGrid</span><span class="p">(</span><span class="n">lr</span><span class="o">.</span><span class="n">maxIter</span><span class="p">,</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">5</span><span class="p">])</span> \
<span class="gp">... </span> <span class="o">.</span><span class="n">build</span><span class="p">()</span>
<span class="gp">>>> </span><span class="n">expected</span> <span class="o">=</span> <span class="p">[</span>
<span class="gp">... </span> <span class="p">{</span><span class="n">lr</span><span class="o">.</span><span class="n">regParam</span><span class="p">:</span> <span class="mf">1.0</span><span class="p">,</span> <span class="n">lr</span><span class="o">.</span><span class="n">maxIter</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span> <span class="n">lr</span><span class="o">.</span><span class="n">labelCol</span><span class="p">:</span> <span class="s">'l'</span><span class="p">,</span> <span class="n">lr</span><span class="o">.</span><span class="n">predictionCol</span><span class="p">:</span> <span class="s">'p'</span><span class="p">},</span>
<span class="gp">... </span> <span class="p">{</span><span class="n">lr</span><span class="o">.</span><span class="n">regParam</span><span class="p">:</span> <span class="mf">2.0</span><span class="p">,</span> <span class="n">lr</span><span class="o">.</span><span class="n">maxIter</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span> <span class="n">lr</span><span class="o">.</span><span class="n">labelCol</span><span class="p">:</span> <span class="s">'l'</span><span class="p">,</span> <span class="n">lr</span><span class="o">.</span><span class="n">predictionCol</span><span class="p">:</span> <span class="s">'p'</span><span class="p">},</span>
<span class="gp">... </span> <span class="p">{</span><span class="n">lr</span><span class="o">.</span><span class="n">regParam</span><span class="p">:</span> <span class="mf">1.0</span><span class="p">,</span> <span class="n">lr</span><span class="o">.</span><span class="n">maxIter</span><span class="p">:</span> <span class="mi">5</span><span class="p">,</span> <span class="n">lr</span><span class="o">.</span><span class="n">labelCol</span><span class="p">:</span> <span class="s">'l'</span><span class="p">,</span> <span class="n">lr</span><span class="o">.</span><span class="n">predictionCol</span><span class="p">:</span> <span class="s">'p'</span><span class="p">},</span>
<span class="gp">... </span> <span class="p">{</span><span class="n">lr</span><span class="o">.</span><span class="n">regParam</span><span class="p">:</span> <span class="mf">2.0</span><span class="p">,</span> <span class="n">lr</span><span class="o">.</span><span class="n">maxIter</span><span class="p">:</span> <span class="mi">5</span><span class="p">,</span> <span class="n">lr</span><span class="o">.</span><span class="n">labelCol</span><span class="p">:</span> <span class="s">'l'</span><span class="p">,</span> <span class="n">lr</span><span class="o">.</span><span class="n">predictionCol</span><span class="p">:</span> <span class="s">'p'</span><span class="p">}]</span>
<span class="gp">>>> </span><span class="nb">len</span><span class="p">(</span><span class="n">output</span><span class="p">)</span> <span class="o">==</span> <span class="nb">len</span><span class="p">(</span><span class="n">expected</span><span class="p">)</span>
<span class="go">True</span>
<span class="gp">>>> </span><span class="nb">all</span><span class="p">([</span><span class="n">m</span> <span class="ow">in</span> <span class="n">expected</span> <span class="k">for</span> <span class="n">m</span> <span class="ow">in</span> <span class="n">output</span><span class="p">])</span>
<span class="go">True</span>
</pre></div>
</div>
<dl class="method">
<dt id="pyspark.ml.tuning.ParamGridBuilder.addGrid">
<code class="descname">addGrid</code><span class="sig-paren">(</span><em>param</em>, <em>values</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/tuning.html#ParamGridBuilder.addGrid"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.tuning.ParamGridBuilder.addGrid" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the given parameters in this grid to fixed values.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.tuning.ParamGridBuilder.baseOn">
<code class="descname">baseOn</code><span class="sig-paren">(</span><em>*args</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/tuning.html#ParamGridBuilder.baseOn"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.tuning.ParamGridBuilder.baseOn" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the given parameters in this grid to fixed values.
Accepts either a parameter dictionary or a list of (parameter, value) pairs.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.tuning.ParamGridBuilder.build">
<code class="descname">build</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/tuning.html#ParamGridBuilder.build"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.tuning.ParamGridBuilder.build" title="Permalink to this definition">¶</a></dt>
<dd><p>Builds and returns all combinations of parameters specified
by the param grid.</p>
</dd></dl>
</dd></dl>
<dl class="class">
<dt id="pyspark.ml.tuning.CrossValidator">
<em class="property">class </em><code class="descclassname">pyspark.ml.tuning.</code><code class="descname">CrossValidator</code><span class="sig-paren">(</span><em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/tuning.html#CrossValidator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.tuning.CrossValidator" title="Permalink to this definition">¶</a></dt>
<dd><p>K-fold cross validation.</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="kn">from</span> <span class="nn">pyspark.ml.classification</span> <span class="kn">import</span> <span class="n">LogisticRegression</span>
<span class="gp">>>> </span><span class="kn">from</span> <span class="nn">pyspark.ml.evaluation</span> <span class="kn">import</span> <span class="n">BinaryClassificationEvaluator</span>
<span class="gp">>>> </span><span class="kn">from</span> <span class="nn">pyspark.mllib.linalg</span> <span class="kn">import</span> <span class="n">Vectors</span>
<span class="gp">>>> </span><span class="n">dataset</span> <span class="o">=</span> <span class="n">sqlContext</span><span class="o">.</span><span class="n">createDataFrame</span><span class="p">(</span>
<span class="gp">... </span> <span class="p">[(</span><span class="n">Vectors</span><span class="o">.</span><span class="n">dense</span><span class="p">([</span><span class="mf">0.0</span><span class="p">]),</span> <span class="mf">0.0</span><span class="p">),</span>
<span class="gp">... </span> <span class="p">(</span><span class="n">Vectors</span><span class="o">.</span><span class="n">dense</span><span class="p">([</span><span class="mf">0.4</span><span class="p">]),</span> <span class="mf">1.0</span><span class="p">),</span>
<span class="gp">... </span> <span class="p">(</span><span class="n">Vectors</span><span class="o">.</span><span class="n">dense</span><span class="p">([</span><span class="mf">0.5</span><span class="p">]),</span> <span class="mf">0.0</span><span class="p">),</span>
<span class="gp">... </span> <span class="p">(</span><span class="n">Vectors</span><span class="o">.</span><span class="n">dense</span><span class="p">([</span><span class="mf">0.6</span><span class="p">]),</span> <span class="mf">1.0</span><span class="p">),</span>
<span class="gp">... </span> <span class="p">(</span><span class="n">Vectors</span><span class="o">.</span><span class="n">dense</span><span class="p">([</span><span class="mf">1.0</span><span class="p">]),</span> <span class="mf">1.0</span><span class="p">)]</span> <span class="o">*</span> <span class="mi">10</span><span class="p">,</span>
<span class="gp">... </span> <span class="p">[</span><span class="s">"features"</span><span class="p">,</span> <span class="s">"label"</span><span class="p">])</span>
<span class="gp">>>> </span><span class="n">lr</span> <span class="o">=</span> <span class="n">LogisticRegression</span><span class="p">()</span>
<span class="gp">>>> </span><span class="n">grid</span> <span class="o">=</span> <span class="n">ParamGridBuilder</span><span class="p">()</span><span class="o">.</span><span class="n">addGrid</span><span class="p">(</span><span class="n">lr</span><span class="o">.</span><span class="n">maxIter</span><span class="p">,</span> <span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="mi">1</span><span class="p">])</span><span class="o">.</span><span class="n">build</span><span class="p">()</span>
<span class="gp">>>> </span><span class="n">evaluator</span> <span class="o">=</span> <span class="n">BinaryClassificationEvaluator</span><span class="p">()</span>
<span class="gp">>>> </span><span class="n">cv</span> <span class="o">=</span> <span class="n">CrossValidator</span><span class="p">(</span><span class="n">estimator</span><span class="o">=</span><span class="n">lr</span><span class="p">,</span> <span class="n">estimatorParamMaps</span><span class="o">=</span><span class="n">grid</span><span class="p">,</span> <span class="n">evaluator</span><span class="o">=</span><span class="n">evaluator</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">cvModel</span> <span class="o">=</span> <span class="n">cv</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">dataset</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">evaluator</span><span class="o">.</span><span class="n">evaluate</span><span class="p">(</span><span class="n">cvModel</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">dataset</span><span class="p">))</span>
<span class="go">0.8333...</span>
</pre></div>
</div>
<dl class="method">
<dt id="pyspark.ml.tuning.CrossValidator.copy">
<code class="descname">copy</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/tuning.html#CrossValidator.copy"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.tuning.CrossValidator.copy" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.tuning.CrossValidator.estimator">
<code class="descname">estimator</code><em class="property"> = Param(parent='undefined', name='estimator', doc='estimator to be cross-validated')</em><a class="headerlink" href="#pyspark.ml.tuning.CrossValidator.estimator" title="Permalink to this definition">¶</a></dt>
<dd><p>param for estimator to be cross-validated</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.tuning.CrossValidator.estimatorParamMaps">
<code class="descname">estimatorParamMaps</code><em class="property"> = Param(parent='undefined', name='estimatorParamMaps', doc='estimator param maps')</em><a class="headerlink" href="#pyspark.ml.tuning.CrossValidator.estimatorParamMaps" title="Permalink to this definition">¶</a></dt>
<dd><p>param for estimator param maps</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.tuning.CrossValidator.evaluator">
<code class="descname">evaluator</code><em class="property"> = Param(parent='undefined', name='evaluator', doc='evaluator used to select hyper-parameters that maximize the cross-validated metric')</em><a class="headerlink" href="#pyspark.ml.tuning.CrossValidator.evaluator" title="Permalink to this definition">¶</a></dt>
<dd><p>param for the evaluator used to select hyper-parameters that
maximize the cross-validated metric</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.tuning.CrossValidator.explainParam">
<code class="descname">explainParam</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.tuning.CrossValidator.explainParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Explains a single param and returns its name, doc, and optional
default value and user-supplied value in a string.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.tuning.CrossValidator.explainParams">
<code class="descname">explainParams</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.tuning.CrossValidator.explainParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns the documentation of all params with their optionally
default values and user-supplied values.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.tuning.CrossValidator.extractParamMap">
<code class="descname">extractParamMap</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.tuning.CrossValidator.extractParamMap" title="Permalink to this definition">¶</a></dt>
<dd><p>Extracts the embedded default param values and user-supplied
values, and then merges them with extra values from input into
a flat param map, where the latter value is used if there exist
conflicts, i.e., with ordering: default param values <
user-supplied values < extra.
:param extra: extra param values
:return: merged param map</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.tuning.CrossValidator.fit">
<code class="descname">fit</code><span class="sig-paren">(</span><em>dataset</em>, <em>params=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.tuning.CrossValidator.fit" title="Permalink to this definition">¶</a></dt>
<dd><p>Fits a model to the input dataset with optional parameters.</p>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple">
<li><strong>dataset</strong> – input dataset, which is an instance of
<a class="reference internal" href="pyspark.sql.html#pyspark.sql.DataFrame" title="pyspark.sql.DataFrame"><code class="xref py py-class docutils literal"><span class="pre">pyspark.sql.DataFrame</span></code></a></li>
<li><strong>params</strong> – an optional param map that overrides embedded
params. If a list/tuple of param maps is given,
this calls fit on each param map and returns a
list of models.</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body"><p class="first last">fitted model(s)</p>
</td>
</tr>
</tbody>
</table>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.tuning.CrossValidator.getEstimator">
<code class="descname">getEstimator</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/tuning.html#CrossValidator.getEstimator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.tuning.CrossValidator.getEstimator" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of estimator or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.tuning.CrossValidator.getEstimatorParamMaps">
<code class="descname">getEstimatorParamMaps</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/tuning.html#CrossValidator.getEstimatorParamMaps"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.tuning.CrossValidator.getEstimatorParamMaps" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of estimatorParamMaps or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.tuning.CrossValidator.getEvaluator">
<code class="descname">getEvaluator</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/tuning.html#CrossValidator.getEvaluator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.tuning.CrossValidator.getEvaluator" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of evaluator or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.tuning.CrossValidator.getNumFolds">
<code class="descname">getNumFolds</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/tuning.html#CrossValidator.getNumFolds"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.tuning.CrossValidator.getNumFolds" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of numFolds or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.tuning.CrossValidator.getOrDefault">
<code class="descname">getOrDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.tuning.CrossValidator.getOrDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of a param in the user-supplied param map or its
default value. Raises an error if neither is set.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.tuning.CrossValidator.getParam">
<code class="descname">getParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.tuning.CrossValidator.getParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets a param by its name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.tuning.CrossValidator.hasDefault">
<code class="descname">hasDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.tuning.CrossValidator.hasDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param has a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.tuning.CrossValidator.hasParam">
<code class="descname">hasParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.tuning.CrossValidator.hasParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Tests whether this instance contains a param with a given
(string) name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.tuning.CrossValidator.isDefined">
<code class="descname">isDefined</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.tuning.CrossValidator.isDefined" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user or has
a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.tuning.CrossValidator.isSet">
<code class="descname">isSet</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.tuning.CrossValidator.isSet" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.tuning.CrossValidator.numFolds">
<code class="descname">numFolds</code><em class="property"> = Param(parent='undefined', name='numFolds', doc='number of folds for cross validation')</em><a class="headerlink" href="#pyspark.ml.tuning.CrossValidator.numFolds" title="Permalink to this definition">¶</a></dt>
<dd><p>param for number of folds for cross validation</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.tuning.CrossValidator.params">
<code class="descname">params</code><a class="headerlink" href="#pyspark.ml.tuning.CrossValidator.params" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns all params ordered by name. The default implementation
uses <code class="xref py py-func docutils literal"><span class="pre">dir()</span></code> to get all attributes of type
<code class="xref py py-class docutils literal"><span class="pre">Param</span></code>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.tuning.CrossValidator.setEstimator">
<code class="descname">setEstimator</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/tuning.html#CrossValidator.setEstimator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.tuning.CrossValidator.setEstimator" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.tuning.CrossValidator.estimator" title="pyspark.ml.tuning.CrossValidator.estimator"><code class="xref py py-attr docutils literal"><span class="pre">estimator</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.tuning.CrossValidator.setEstimatorParamMaps">
<code class="descname">setEstimatorParamMaps</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/tuning.html#CrossValidator.setEstimatorParamMaps"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.tuning.CrossValidator.setEstimatorParamMaps" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.tuning.CrossValidator.estimatorParamMaps" title="pyspark.ml.tuning.CrossValidator.estimatorParamMaps"><code class="xref py py-attr docutils literal"><span class="pre">estimatorParamMaps</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.tuning.CrossValidator.setEvaluator">
<code class="descname">setEvaluator</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/tuning.html#CrossValidator.setEvaluator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.tuning.CrossValidator.setEvaluator" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.tuning.CrossValidator.evaluator" title="pyspark.ml.tuning.CrossValidator.evaluator"><code class="xref py py-attr docutils literal"><span class="pre">evaluator</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.tuning.CrossValidator.setNumFolds">
<code class="descname">setNumFolds</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/tuning.html#CrossValidator.setNumFolds"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.tuning.CrossValidator.setNumFolds" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.tuning.CrossValidator.numFolds" title="pyspark.ml.tuning.CrossValidator.numFolds"><code class="xref py py-attr docutils literal"><span class="pre">numFolds</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.tuning.CrossValidator.setParams">
<code class="descname">setParams</code><span class="sig-paren">(</span><em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/tuning.html#CrossValidator.setParams"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.tuning.CrossValidator.setParams" title="Permalink to this definition">¶</a></dt>
<dd><p>setParams(self, estimator=None, estimatorParamMaps=None, evaluator=None, numFolds=3):
Sets params for cross validator.</p>
</dd></dl>
</dd></dl>
<dl class="class">
<dt id="pyspark.ml.tuning.CrossValidatorModel">
<em class="property">class </em><code class="descclassname">pyspark.ml.tuning.</code><code class="descname">CrossValidatorModel</code><span class="sig-paren">(</span><em>bestModel</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/tuning.html#CrossValidatorModel"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.tuning.CrossValidatorModel" title="Permalink to this definition">¶</a></dt>
<dd><p>Model from k-fold cross validation.</p>
<dl class="attribute">
<dt id="pyspark.ml.tuning.CrossValidatorModel.bestModel">
<code class="descname">bestModel</code><em class="property"> = None</em><a class="headerlink" href="#pyspark.ml.tuning.CrossValidatorModel.bestModel" title="Permalink to this definition">¶</a></dt>
<dd><p>best model from cross validation</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.tuning.CrossValidatorModel.copy">
<code class="descname">copy</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/tuning.html#CrossValidatorModel.copy"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.tuning.CrossValidatorModel.copy" title="Permalink to this definition">¶</a></dt>
<dd><p>Creates a copy of this instance with a randomly generated uid
and some extra params. This copies the underlying bestModel,
creates a deep copy of the embedded paramMap, and
copies the embedded and extra parameters over.
:param extra: Extra parameters to copy to the new instance
:return: Copy of this instance</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.tuning.CrossValidatorModel.explainParam">
<code class="descname">explainParam</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.tuning.CrossValidatorModel.explainParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Explains a single param and returns its name, doc, and optional
default value and user-supplied value in a string.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.tuning.CrossValidatorModel.explainParams">
<code class="descname">explainParams</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.tuning.CrossValidatorModel.explainParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns the documentation of all params with their optionally
default values and user-supplied values.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.tuning.CrossValidatorModel.extractParamMap">
<code class="descname">extractParamMap</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.tuning.CrossValidatorModel.extractParamMap" title="Permalink to this definition">¶</a></dt>
<dd><p>Extracts the embedded default param values and user-supplied
values, and then merges them with extra values from input into
a flat param map, where the latter value is used if there exist
conflicts, i.e., with ordering: default param values <
user-supplied values < extra.
:param extra: extra param values
:return: merged param map</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.tuning.CrossValidatorModel.getOrDefault">
<code class="descname">getOrDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.tuning.CrossValidatorModel.getOrDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of a param in the user-supplied param map or its
default value. Raises an error if neither is set.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.tuning.CrossValidatorModel.getParam">
<code class="descname">getParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.tuning.CrossValidatorModel.getParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets a param by its name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.tuning.CrossValidatorModel.hasDefault">
<code class="descname">hasDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.tuning.CrossValidatorModel.hasDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param has a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.tuning.CrossValidatorModel.hasParam">
<code class="descname">hasParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.tuning.CrossValidatorModel.hasParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Tests whether this instance contains a param with a given
(string) name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.tuning.CrossValidatorModel.isDefined">
<code class="descname">isDefined</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.tuning.CrossValidatorModel.isDefined" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user or has
a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.tuning.CrossValidatorModel.isSet">
<code class="descname">isSet</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.tuning.CrossValidatorModel.isSet" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.tuning.CrossValidatorModel.params">
<code class="descname">params</code><a class="headerlink" href="#pyspark.ml.tuning.CrossValidatorModel.params" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns all params ordered by name. The default implementation
uses <code class="xref py py-func docutils literal"><span class="pre">dir()</span></code> to get all attributes of type
<code class="xref py py-class docutils literal"><span class="pre">Param</span></code>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.tuning.CrossValidatorModel.transform">
<code class="descname">transform</code><span class="sig-paren">(</span><em>dataset</em>, <em>params=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.tuning.CrossValidatorModel.transform" title="Permalink to this definition">¶</a></dt>
<dd><p>Transforms the input dataset with optional parameters.</p>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple">
<li><strong>dataset</strong> – input dataset, which is an instance of
<a class="reference internal" href="pyspark.sql.html#pyspark.sql.DataFrame" title="pyspark.sql.DataFrame"><code class="xref py py-class docutils literal"><span class="pre">pyspark.sql.DataFrame</span></code></a></li>
<li><strong>params</strong> – an optional param map that overrides embedded
params.</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body"><p class="first last">transformed dataset</p>
</td>
</tr>
</tbody>
</table>
</dd></dl>
</dd></dl>
</div>
<div class="section" id="module-pyspark.ml.evaluation">
<span id="pyspark-ml-evaluation-module"></span><h2>pyspark.ml.evaluation module<a class="headerlink" href="#module-pyspark.ml.evaluation" title="Permalink to this headline">¶</a></h2>
<dl class="class">
<dt id="pyspark.ml.evaluation.Evaluator">
<em class="property">class </em><code class="descclassname">pyspark.ml.evaluation.</code><code class="descname">Evaluator</code><a class="reference internal" href="_modules/pyspark/ml/evaluation.html#Evaluator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.evaluation.Evaluator" title="Permalink to this definition">¶</a></dt>
<dd><p>Base class for evaluators that compute metrics from predictions.</p>
<dl class="method">
<dt id="pyspark.ml.evaluation.Evaluator.copy">
<code class="descname">copy</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.evaluation.Evaluator.copy" title="Permalink to this definition">¶</a></dt>
<dd><p>Creates a copy of this instance with the same uid and some
extra params. The default implementation creates a
shallow copy using <code class="xref py py-func docutils literal"><span class="pre">copy.copy()</span></code>, and then copies the
embedded and extra parameters over and returns the copy.
Subclasses should override this method if the default approach
is not sufficient.
:param extra: Extra parameters to copy to the new instance
:return: Copy of this instance</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.evaluation.Evaluator.evaluate">
<code class="descname">evaluate</code><span class="sig-paren">(</span><em>dataset</em>, <em>params=None</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/evaluation.html#Evaluator.evaluate"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.evaluation.Evaluator.evaluate" title="Permalink to this definition">¶</a></dt>
<dd><p>Evaluates the output with optional parameters.</p>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple">
<li><strong>dataset</strong> – a dataset that contains labels/observations and
predictions</li>
<li><strong>params</strong> – an optional param map that overrides embedded
params</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body"><p class="first last">metric</p>
</td>
</tr>
</tbody>
</table>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.evaluation.Evaluator.explainParam">
<code class="descname">explainParam</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.evaluation.Evaluator.explainParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Explains a single param and returns its name, doc, and optional
default value and user-supplied value in a string.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.evaluation.Evaluator.explainParams">
<code class="descname">explainParams</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.evaluation.Evaluator.explainParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns the documentation of all params with their optionally
default values and user-supplied values.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.evaluation.Evaluator.extractParamMap">
<code class="descname">extractParamMap</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.evaluation.Evaluator.extractParamMap" title="Permalink to this definition">¶</a></dt>
<dd><p>Extracts the embedded default param values and user-supplied
values, and then merges them with extra values from input into
a flat param map, where the latter value is used if there exist
conflicts, i.e., with ordering: default param values <
user-supplied values < extra.
:param extra: extra param values
:return: merged param map</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.evaluation.Evaluator.getOrDefault">
<code class="descname">getOrDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.evaluation.Evaluator.getOrDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of a param in the user-supplied param map or its
default value. Raises an error if neither is set.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.evaluation.Evaluator.getParam">
<code class="descname">getParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.evaluation.Evaluator.getParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets a param by its name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.evaluation.Evaluator.hasDefault">
<code class="descname">hasDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.evaluation.Evaluator.hasDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param has a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.evaluation.Evaluator.hasParam">
<code class="descname">hasParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.evaluation.Evaluator.hasParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Tests whether this instance contains a param with a given
(string) name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.evaluation.Evaluator.isDefined">
<code class="descname">isDefined</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.evaluation.Evaluator.isDefined" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user or has
a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.evaluation.Evaluator.isLargerBetter">
<code class="descname">isLargerBetter</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/evaluation.html#Evaluator.isLargerBetter"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.evaluation.Evaluator.isLargerBetter" title="Permalink to this definition">¶</a></dt>
<dd><p>Indicates whether the metric returned by <a class="reference internal" href="#pyspark.ml.evaluation.Evaluator.evaluate" title="pyspark.ml.evaluation.Evaluator.evaluate"><code class="xref py py-meth docutils literal"><span class="pre">evaluate()</span></code></a> should be maximized
(True, default) or minimized (False).
A given evaluator may support multiple metrics which may be maximized or minimized.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.evaluation.Evaluator.isSet">
<code class="descname">isSet</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.evaluation.Evaluator.isSet" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.evaluation.Evaluator.params">
<code class="descname">params</code><a class="headerlink" href="#pyspark.ml.evaluation.Evaluator.params" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns all params ordered by name. The default implementation
uses <code class="xref py py-func docutils literal"><span class="pre">dir()</span></code> to get all attributes of type
<code class="xref py py-class docutils literal"><span class="pre">Param</span></code>.</p>
</dd></dl>
</dd></dl>
<dl class="class">
<dt id="pyspark.ml.evaluation.BinaryClassificationEvaluator">
<em class="property">class </em><code class="descclassname">pyspark.ml.evaluation.</code><code class="descname">BinaryClassificationEvaluator</code><span class="sig-paren">(</span><em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/evaluation.html#BinaryClassificationEvaluator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.evaluation.BinaryClassificationEvaluator" title="Permalink to this definition">¶</a></dt>
<dd><p>Evaluator for binary classification, which expects two input
columns: rawPrediction and label.</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="kn">from</span> <span class="nn">pyspark.mllib.linalg</span> <span class="kn">import</span> <span class="n">Vectors</span>
<span class="gp">>>> </span><span class="n">scoreAndLabels</span> <span class="o">=</span> <span class="nb">map</span><span class="p">(</span><span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="p">(</span><span class="n">Vectors</span><span class="o">.</span><span class="n">dense</span><span class="p">([</span><span class="mf">1.0</span> <span class="o">-</span> <span class="n">x</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="n">x</span><span class="p">[</span><span class="mi">0</span><span class="p">]]),</span> <span class="n">x</span><span class="p">[</span><span class="mi">1</span><span class="p">]),</span>
<span class="gp">... </span> <span class="p">[(</span><span class="mf">0.1</span><span class="p">,</span> <span class="mf">0.0</span><span class="p">),</span> <span class="p">(</span><span class="mf">0.1</span><span class="p">,</span> <span class="mf">1.0</span><span class="p">),</span> <span class="p">(</span><span class="mf">0.4</span><span class="p">,</span> <span class="mf">0.0</span><span class="p">),</span> <span class="p">(</span><span class="mf">0.6</span><span class="p">,</span> <span class="mf">0.0</span><span class="p">),</span> <span class="p">(</span><span class="mf">0.6</span><span class="p">,</span> <span class="mf">1.0</span><span class="p">),</span> <span class="p">(</span><span class="mf">0.6</span><span class="p">,</span> <span class="mf">1.0</span><span class="p">),</span> <span class="p">(</span><span class="mf">0.8</span><span class="p">,</span> <span class="mf">1.0</span><span class="p">)])</span>
<span class="gp">>>> </span><span class="n">dataset</span> <span class="o">=</span> <span class="n">sqlContext</span><span class="o">.</span><span class="n">createDataFrame</span><span class="p">(</span><span class="n">scoreAndLabels</span><span class="p">,</span> <span class="p">[</span><span class="s">"raw"</span><span class="p">,</span> <span class="s">"label"</span><span class="p">])</span>
<span class="gp">...</span>
<span class="gp">>>> </span><span class="n">evaluator</span> <span class="o">=</span> <span class="n">BinaryClassificationEvaluator</span><span class="p">(</span><span class="n">rawPredictionCol</span><span class="o">=</span><span class="s">"raw"</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">evaluator</span><span class="o">.</span><span class="n">evaluate</span><span class="p">(</span><span class="n">dataset</span><span class="p">)</span>
<span class="go">0.70...</span>
<span class="gp">>>> </span><span class="n">evaluator</span><span class="o">.</span><span class="n">evaluate</span><span class="p">(</span><span class="n">dataset</span><span class="p">,</span> <span class="p">{</span><span class="n">evaluator</span><span class="o">.</span><span class="n">metricName</span><span class="p">:</span> <span class="s">"areaUnderPR"</span><span class="p">})</span>
<span class="go">0.83...</span>
</pre></div>
</div>
<dl class="method">
<dt id="pyspark.ml.evaluation.BinaryClassificationEvaluator.copy">
<code class="descname">copy</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.evaluation.BinaryClassificationEvaluator.copy" title="Permalink to this definition">¶</a></dt>
<dd><p>Creates a copy of this instance with the same uid and some
extra params. The default implementation creates a
shallow copy using <code class="xref py py-func docutils literal"><span class="pre">copy.copy()</span></code>, and then copies the
embedded and extra parameters over and returns the copy.
Subclasses should override this method if the default approach
is not sufficient.
:param extra: Extra parameters to copy to the new instance
:return: Copy of this instance</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.evaluation.BinaryClassificationEvaluator.evaluate">
<code class="descname">evaluate</code><span class="sig-paren">(</span><em>dataset</em>, <em>params=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.evaluation.BinaryClassificationEvaluator.evaluate" title="Permalink to this definition">¶</a></dt>
<dd><p>Evaluates the output with optional parameters.</p>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple">
<li><strong>dataset</strong> – a dataset that contains labels/observations and
predictions</li>
<li><strong>params</strong> – an optional param map that overrides embedded
params</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body"><p class="first last">metric</p>
</td>
</tr>
</tbody>
</table>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.evaluation.BinaryClassificationEvaluator.explainParam">
<code class="descname">explainParam</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.evaluation.BinaryClassificationEvaluator.explainParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Explains a single param and returns its name, doc, and optional
default value and user-supplied value in a string.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.evaluation.BinaryClassificationEvaluator.explainParams">
<code class="descname">explainParams</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.evaluation.BinaryClassificationEvaluator.explainParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns the documentation of all params with their optionally
default values and user-supplied values.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.evaluation.BinaryClassificationEvaluator.extractParamMap">
<code class="descname">extractParamMap</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.evaluation.BinaryClassificationEvaluator.extractParamMap" title="Permalink to this definition">¶</a></dt>
<dd><p>Extracts the embedded default param values and user-supplied
values, and then merges them with extra values from input into
a flat param map, where the latter value is used if there exist
conflicts, i.e., with ordering: default param values <
user-supplied values < extra.
:param extra: extra param values
:return: merged param map</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.evaluation.BinaryClassificationEvaluator.getLabelCol">
<code class="descname">getLabelCol</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.evaluation.BinaryClassificationEvaluator.getLabelCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of labelCol or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.evaluation.BinaryClassificationEvaluator.getMetricName">
<code class="descname">getMetricName</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/evaluation.html#BinaryClassificationEvaluator.getMetricName"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.evaluation.BinaryClassificationEvaluator.getMetricName" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of metricName or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.evaluation.BinaryClassificationEvaluator.getOrDefault">
<code class="descname">getOrDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.evaluation.BinaryClassificationEvaluator.getOrDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of a param in the user-supplied param map or its
default value. Raises an error if neither is set.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.evaluation.BinaryClassificationEvaluator.getParam">
<code class="descname">getParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.evaluation.BinaryClassificationEvaluator.getParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets a param by its name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.evaluation.BinaryClassificationEvaluator.getRawPredictionCol">
<code class="descname">getRawPredictionCol</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.evaluation.BinaryClassificationEvaluator.getRawPredictionCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of rawPredictionCol or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.evaluation.BinaryClassificationEvaluator.hasDefault">
<code class="descname">hasDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.evaluation.BinaryClassificationEvaluator.hasDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param has a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.evaluation.BinaryClassificationEvaluator.hasParam">
<code class="descname">hasParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.evaluation.BinaryClassificationEvaluator.hasParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Tests whether this instance contains a param with a given
(string) name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.evaluation.BinaryClassificationEvaluator.isDefined">
<code class="descname">isDefined</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.evaluation.BinaryClassificationEvaluator.isDefined" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user or has
a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.evaluation.BinaryClassificationEvaluator.isLargerBetter">
<code class="descname">isLargerBetter</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.evaluation.BinaryClassificationEvaluator.isLargerBetter" title="Permalink to this definition">¶</a></dt>
<dd><p>Indicates whether the metric returned by <a class="reference internal" href="#pyspark.ml.evaluation.BinaryClassificationEvaluator.evaluate" title="pyspark.ml.evaluation.BinaryClassificationEvaluator.evaluate"><code class="xref py py-meth docutils literal"><span class="pre">evaluate()</span></code></a> should be maximized
(True, default) or minimized (False).
A given evaluator may support multiple metrics which may be maximized or minimized.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.evaluation.BinaryClassificationEvaluator.isSet">
<code class="descname">isSet</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.evaluation.BinaryClassificationEvaluator.isSet" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.evaluation.BinaryClassificationEvaluator.labelCol">
<code class="descname">labelCol</code><em class="property"> = Param(parent='undefined', name='labelCol', doc='label column name')</em><a class="headerlink" href="#pyspark.ml.evaluation.BinaryClassificationEvaluator.labelCol" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.evaluation.BinaryClassificationEvaluator.metricName">
<code class="descname">metricName</code><em class="property"> = Param(parent='undefined', name='metricName', doc='metric name in evaluation (areaUnderROC|areaUnderPR)')</em><a class="headerlink" href="#pyspark.ml.evaluation.BinaryClassificationEvaluator.metricName" title="Permalink to this definition">¶</a></dt>
<dd><p>param for metric name in evaluation (areaUnderROC|areaUnderPR)</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.evaluation.BinaryClassificationEvaluator.params">
<code class="descname">params</code><a class="headerlink" href="#pyspark.ml.evaluation.BinaryClassificationEvaluator.params" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns all params ordered by name. The default implementation
uses <code class="xref py py-func docutils literal"><span class="pre">dir()</span></code> to get all attributes of type
<code class="xref py py-class docutils literal"><span class="pre">Param</span></code>.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.evaluation.BinaryClassificationEvaluator.rawPredictionCol">
<code class="descname">rawPredictionCol</code><em class="property"> = Param(parent='undefined', name='rawPredictionCol', doc='raw prediction (a.k.a. confidence) column name')</em><a class="headerlink" href="#pyspark.ml.evaluation.BinaryClassificationEvaluator.rawPredictionCol" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="method">
<dt id="pyspark.ml.evaluation.BinaryClassificationEvaluator.setLabelCol">
<code class="descname">setLabelCol</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.evaluation.BinaryClassificationEvaluator.setLabelCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.evaluation.BinaryClassificationEvaluator.labelCol" title="pyspark.ml.evaluation.BinaryClassificationEvaluator.labelCol"><code class="xref py py-attr docutils literal"><span class="pre">labelCol</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.evaluation.BinaryClassificationEvaluator.setMetricName">
<code class="descname">setMetricName</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/evaluation.html#BinaryClassificationEvaluator.setMetricName"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.evaluation.BinaryClassificationEvaluator.setMetricName" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.evaluation.BinaryClassificationEvaluator.metricName" title="pyspark.ml.evaluation.BinaryClassificationEvaluator.metricName"><code class="xref py py-attr docutils literal"><span class="pre">metricName</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.evaluation.BinaryClassificationEvaluator.setParams">
<code class="descname">setParams</code><span class="sig-paren">(</span><em>self</em>, <em>rawPredictionCol="rawPrediction"</em>, <em>labelCol="label"</em>, <em>metricName="areaUnderROC"</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/evaluation.html#BinaryClassificationEvaluator.setParams"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.evaluation.BinaryClassificationEvaluator.setParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets params for binary classification evaluator.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.evaluation.BinaryClassificationEvaluator.setRawPredictionCol">
<code class="descname">setRawPredictionCol</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.evaluation.BinaryClassificationEvaluator.setRawPredictionCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.evaluation.BinaryClassificationEvaluator.rawPredictionCol" title="pyspark.ml.evaluation.BinaryClassificationEvaluator.rawPredictionCol"><code class="xref py py-attr docutils literal"><span class="pre">rawPredictionCol</span></code></a>.</p>
</dd></dl>
</dd></dl>
<dl class="class">
<dt id="pyspark.ml.evaluation.RegressionEvaluator">
<em class="property">class </em><code class="descclassname">pyspark.ml.evaluation.</code><code class="descname">RegressionEvaluator</code><span class="sig-paren">(</span><em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/evaluation.html#RegressionEvaluator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.evaluation.RegressionEvaluator" title="Permalink to this definition">¶</a></dt>
<dd><p>Evaluator for Regression, which expects two input
columns: prediction and label.</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="n">scoreAndLabels</span> <span class="o">=</span> <span class="p">[(</span><span class="o">-</span><span class="mf">28.98343821</span><span class="p">,</span> <span class="o">-</span><span class="mf">27.0</span><span class="p">),</span> <span class="p">(</span><span class="mf">20.21491975</span><span class="p">,</span> <span class="mf">21.5</span><span class="p">),</span>
<span class="gp">... </span> <span class="p">(</span><span class="o">-</span><span class="mf">25.98418959</span><span class="p">,</span> <span class="o">-</span><span class="mf">22.0</span><span class="p">),</span> <span class="p">(</span><span class="mf">30.69731842</span><span class="p">,</span> <span class="mf">33.0</span><span class="p">),</span> <span class="p">(</span><span class="mf">74.69283752</span><span class="p">,</span> <span class="mf">71.0</span><span class="p">)]</span>
<span class="gp">>>> </span><span class="n">dataset</span> <span class="o">=</span> <span class="n">sqlContext</span><span class="o">.</span><span class="n">createDataFrame</span><span class="p">(</span><span class="n">scoreAndLabels</span><span class="p">,</span> <span class="p">[</span><span class="s">"raw"</span><span class="p">,</span> <span class="s">"label"</span><span class="p">])</span>
<span class="gp">...</span>
<span class="gp">>>> </span><span class="n">evaluator</span> <span class="o">=</span> <span class="n">RegressionEvaluator</span><span class="p">(</span><span class="n">predictionCol</span><span class="o">=</span><span class="s">"raw"</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">evaluator</span><span class="o">.</span><span class="n">evaluate</span><span class="p">(</span><span class="n">dataset</span><span class="p">)</span>
<span class="go">2.842...</span>
<span class="gp">>>> </span><span class="n">evaluator</span><span class="o">.</span><span class="n">evaluate</span><span class="p">(</span><span class="n">dataset</span><span class="p">,</span> <span class="p">{</span><span class="n">evaluator</span><span class="o">.</span><span class="n">metricName</span><span class="p">:</span> <span class="s">"r2"</span><span class="p">})</span>
<span class="go">0.993...</span>
<span class="gp">>>> </span><span class="n">evaluator</span><span class="o">.</span><span class="n">evaluate</span><span class="p">(</span><span class="n">dataset</span><span class="p">,</span> <span class="p">{</span><span class="n">evaluator</span><span class="o">.</span><span class="n">metricName</span><span class="p">:</span> <span class="s">"mae"</span><span class="p">})</span>
<span class="go">2.649...</span>
</pre></div>
</div>
<dl class="method">
<dt id="pyspark.ml.evaluation.RegressionEvaluator.copy">
<code class="descname">copy</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.evaluation.RegressionEvaluator.copy" title="Permalink to this definition">¶</a></dt>
<dd><p>Creates a copy of this instance with the same uid and some
extra params. The default implementation creates a
shallow copy using <code class="xref py py-func docutils literal"><span class="pre">copy.copy()</span></code>, and then copies the
embedded and extra parameters over and returns the copy.
Subclasses should override this method if the default approach
is not sufficient.
:param extra: Extra parameters to copy to the new instance
:return: Copy of this instance</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.evaluation.RegressionEvaluator.evaluate">
<code class="descname">evaluate</code><span class="sig-paren">(</span><em>dataset</em>, <em>params=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.evaluation.RegressionEvaluator.evaluate" title="Permalink to this definition">¶</a></dt>
<dd><p>Evaluates the output with optional parameters.</p>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple">
<li><strong>dataset</strong> – a dataset that contains labels/observations and
predictions</li>
<li><strong>params</strong> – an optional param map that overrides embedded
params</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body"><p class="first last">metric</p>
</td>
</tr>
</tbody>
</table>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.evaluation.RegressionEvaluator.explainParam">
<code class="descname">explainParam</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.evaluation.RegressionEvaluator.explainParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Explains a single param and returns its name, doc, and optional
default value and user-supplied value in a string.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.evaluation.RegressionEvaluator.explainParams">
<code class="descname">explainParams</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.evaluation.RegressionEvaluator.explainParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns the documentation of all params with their optionally
default values and user-supplied values.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.evaluation.RegressionEvaluator.extractParamMap">
<code class="descname">extractParamMap</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.evaluation.RegressionEvaluator.extractParamMap" title="Permalink to this definition">¶</a></dt>
<dd><p>Extracts the embedded default param values and user-supplied
values, and then merges them with extra values from input into
a flat param map, where the latter value is used if there exist
conflicts, i.e., with ordering: default param values <
user-supplied values < extra.
:param extra: extra param values
:return: merged param map</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.evaluation.RegressionEvaluator.getLabelCol">
<code class="descname">getLabelCol</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.evaluation.RegressionEvaluator.getLabelCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of labelCol or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.evaluation.RegressionEvaluator.getMetricName">
<code class="descname">getMetricName</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/evaluation.html#RegressionEvaluator.getMetricName"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.evaluation.RegressionEvaluator.getMetricName" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of metricName or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.evaluation.RegressionEvaluator.getOrDefault">
<code class="descname">getOrDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.evaluation.RegressionEvaluator.getOrDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of a param in the user-supplied param map or its
default value. Raises an error if neither is set.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.evaluation.RegressionEvaluator.getParam">
<code class="descname">getParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.evaluation.RegressionEvaluator.getParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets a param by its name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.evaluation.RegressionEvaluator.getPredictionCol">
<code class="descname">getPredictionCol</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.evaluation.RegressionEvaluator.getPredictionCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of predictionCol or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.evaluation.RegressionEvaluator.hasDefault">
<code class="descname">hasDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.evaluation.RegressionEvaluator.hasDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param has a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.evaluation.RegressionEvaluator.hasParam">
<code class="descname">hasParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.evaluation.RegressionEvaluator.hasParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Tests whether this instance contains a param with a given
(string) name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.evaluation.RegressionEvaluator.isDefined">
<code class="descname">isDefined</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.evaluation.RegressionEvaluator.isDefined" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user or has
a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.evaluation.RegressionEvaluator.isLargerBetter">
<code class="descname">isLargerBetter</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.evaluation.RegressionEvaluator.isLargerBetter" title="Permalink to this definition">¶</a></dt>
<dd><p>Indicates whether the metric returned by <a class="reference internal" href="#pyspark.ml.evaluation.RegressionEvaluator.evaluate" title="pyspark.ml.evaluation.RegressionEvaluator.evaluate"><code class="xref py py-meth docutils literal"><span class="pre">evaluate()</span></code></a> should be maximized
(True, default) or minimized (False).
A given evaluator may support multiple metrics which may be maximized or minimized.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.evaluation.RegressionEvaluator.isSet">
<code class="descname">isSet</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.evaluation.RegressionEvaluator.isSet" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.evaluation.RegressionEvaluator.labelCol">
<code class="descname">labelCol</code><em class="property"> = Param(parent='undefined', name='labelCol', doc='label column name')</em><a class="headerlink" href="#pyspark.ml.evaluation.RegressionEvaluator.labelCol" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.evaluation.RegressionEvaluator.metricName">
<code class="descname">metricName</code><em class="property"> = Param(parent='undefined', name='metricName', doc='metric name in evaluation (mse|rmse|r2|mae)')</em><a class="headerlink" href="#pyspark.ml.evaluation.RegressionEvaluator.metricName" title="Permalink to this definition">¶</a></dt>
<dd><p>param for metric name in evaluation (mse|rmse|r2|mae)</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.evaluation.RegressionEvaluator.params">
<code class="descname">params</code><a class="headerlink" href="#pyspark.ml.evaluation.RegressionEvaluator.params" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns all params ordered by name. The default implementation
uses <code class="xref py py-func docutils literal"><span class="pre">dir()</span></code> to get all attributes of type
<code class="xref py py-class docutils literal"><span class="pre">Param</span></code>.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.evaluation.RegressionEvaluator.predictionCol">
<code class="descname">predictionCol</code><em class="property"> = Param(parent='undefined', name='predictionCol', doc='prediction column name')</em><a class="headerlink" href="#pyspark.ml.evaluation.RegressionEvaluator.predictionCol" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="method">
<dt id="pyspark.ml.evaluation.RegressionEvaluator.setLabelCol">
<code class="descname">setLabelCol</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.evaluation.RegressionEvaluator.setLabelCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.evaluation.RegressionEvaluator.labelCol" title="pyspark.ml.evaluation.RegressionEvaluator.labelCol"><code class="xref py py-attr docutils literal"><span class="pre">labelCol</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.evaluation.RegressionEvaluator.setMetricName">
<code class="descname">setMetricName</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/evaluation.html#RegressionEvaluator.setMetricName"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.evaluation.RegressionEvaluator.setMetricName" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.evaluation.RegressionEvaluator.metricName" title="pyspark.ml.evaluation.RegressionEvaluator.metricName"><code class="xref py py-attr docutils literal"><span class="pre">metricName</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.evaluation.RegressionEvaluator.setParams">
<code class="descname">setParams</code><span class="sig-paren">(</span><em>self</em>, <em>predictionCol="prediction"</em>, <em>labelCol="label"</em>, <em>metricName="rmse"</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/evaluation.html#RegressionEvaluator.setParams"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.evaluation.RegressionEvaluator.setParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets params for regression evaluator.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.evaluation.RegressionEvaluator.setPredictionCol">
<code class="descname">setPredictionCol</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.evaluation.RegressionEvaluator.setPredictionCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.evaluation.RegressionEvaluator.predictionCol" title="pyspark.ml.evaluation.RegressionEvaluator.predictionCol"><code class="xref py py-attr docutils literal"><span class="pre">predictionCol</span></code></a>.</p>
</dd></dl>
</dd></dl>
<dl class="class">
<dt id="pyspark.ml.evaluation.MulticlassClassificationEvaluator">
<em class="property">class </em><code class="descclassname">pyspark.ml.evaluation.</code><code class="descname">MulticlassClassificationEvaluator</code><span class="sig-paren">(</span><em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/evaluation.html#MulticlassClassificationEvaluator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.evaluation.MulticlassClassificationEvaluator" title="Permalink to this definition">¶</a></dt>
<dd><p>Evaluator for Multiclass Classification, which expects two input
columns: prediction and label.
>>> scoreAndLabels = [(0.0, 0.0), (0.0, 1.0), (0.0, 0.0),
... (1.0, 0.0), (1.0, 1.0), (1.0, 1.0), (1.0, 1.0), (2.0, 2.0), (2.0, 0.0)]
>>> dataset = sqlContext.createDataFrame(scoreAndLabels, [“prediction”, “label”])
...
>>> evaluator = MulticlassClassificationEvaluator(predictionCol=”prediction”)
>>> evaluator.evaluate(dataset)
0.66...
>>> evaluator.evaluate(dataset, {evaluator.metricName: “precision”})
0.66...
>>> evaluator.evaluate(dataset, {evaluator.metricName: “recall”})
0.66...</p>
<dl class="method">
<dt id="pyspark.ml.evaluation.MulticlassClassificationEvaluator.copy">
<code class="descname">copy</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.evaluation.MulticlassClassificationEvaluator.copy" title="Permalink to this definition">¶</a></dt>
<dd><p>Creates a copy of this instance with the same uid and some
extra params. The default implementation creates a
shallow copy using <code class="xref py py-func docutils literal"><span class="pre">copy.copy()</span></code>, and then copies the
embedded and extra parameters over and returns the copy.
Subclasses should override this method if the default approach
is not sufficient.
:param extra: Extra parameters to copy to the new instance
:return: Copy of this instance</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.evaluation.MulticlassClassificationEvaluator.evaluate">
<code class="descname">evaluate</code><span class="sig-paren">(</span><em>dataset</em>, <em>params=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.evaluation.MulticlassClassificationEvaluator.evaluate" title="Permalink to this definition">¶</a></dt>
<dd><p>Evaluates the output with optional parameters.</p>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple">
<li><strong>dataset</strong> – a dataset that contains labels/observations and
predictions</li>
<li><strong>params</strong> – an optional param map that overrides embedded
params</li>
</ul>
</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body"><p class="first last">metric</p>
</td>
</tr>
</tbody>
</table>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.evaluation.MulticlassClassificationEvaluator.explainParam">
<code class="descname">explainParam</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.evaluation.MulticlassClassificationEvaluator.explainParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Explains a single param and returns its name, doc, and optional
default value and user-supplied value in a string.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.evaluation.MulticlassClassificationEvaluator.explainParams">
<code class="descname">explainParams</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.evaluation.MulticlassClassificationEvaluator.explainParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns the documentation of all params with their optionally
default values and user-supplied values.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.evaluation.MulticlassClassificationEvaluator.extractParamMap">
<code class="descname">extractParamMap</code><span class="sig-paren">(</span><em>extra=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.evaluation.MulticlassClassificationEvaluator.extractParamMap" title="Permalink to this definition">¶</a></dt>
<dd><p>Extracts the embedded default param values and user-supplied
values, and then merges them with extra values from input into
a flat param map, where the latter value is used if there exist
conflicts, i.e., with ordering: default param values <
user-supplied values < extra.
:param extra: extra param values
:return: merged param map</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.evaluation.MulticlassClassificationEvaluator.getLabelCol">
<code class="descname">getLabelCol</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.evaluation.MulticlassClassificationEvaluator.getLabelCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of labelCol or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.evaluation.MulticlassClassificationEvaluator.getMetricName">
<code class="descname">getMetricName</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/evaluation.html#MulticlassClassificationEvaluator.getMetricName"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.evaluation.MulticlassClassificationEvaluator.getMetricName" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of metricName or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.evaluation.MulticlassClassificationEvaluator.getOrDefault">
<code class="descname">getOrDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.evaluation.MulticlassClassificationEvaluator.getOrDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of a param in the user-supplied param map or its
default value. Raises an error if neither is set.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.evaluation.MulticlassClassificationEvaluator.getParam">
<code class="descname">getParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.evaluation.MulticlassClassificationEvaluator.getParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets a param by its name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.evaluation.MulticlassClassificationEvaluator.getPredictionCol">
<code class="descname">getPredictionCol</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.evaluation.MulticlassClassificationEvaluator.getPredictionCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Gets the value of predictionCol or its default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.evaluation.MulticlassClassificationEvaluator.hasDefault">
<code class="descname">hasDefault</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.evaluation.MulticlassClassificationEvaluator.hasDefault" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param has a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.evaluation.MulticlassClassificationEvaluator.hasParam">
<code class="descname">hasParam</code><span class="sig-paren">(</span><em>paramName</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.evaluation.MulticlassClassificationEvaluator.hasParam" title="Permalink to this definition">¶</a></dt>
<dd><p>Tests whether this instance contains a param with a given
(string) name.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.evaluation.MulticlassClassificationEvaluator.isDefined">
<code class="descname">isDefined</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.evaluation.MulticlassClassificationEvaluator.isDefined" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user or has
a default value.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.evaluation.MulticlassClassificationEvaluator.isLargerBetter">
<code class="descname">isLargerBetter</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.evaluation.MulticlassClassificationEvaluator.isLargerBetter" title="Permalink to this definition">¶</a></dt>
<dd><p>Indicates whether the metric returned by <a class="reference internal" href="#pyspark.ml.evaluation.MulticlassClassificationEvaluator.evaluate" title="pyspark.ml.evaluation.MulticlassClassificationEvaluator.evaluate"><code class="xref py py-meth docutils literal"><span class="pre">evaluate()</span></code></a> should be maximized
(True, default) or minimized (False).
A given evaluator may support multiple metrics which may be maximized or minimized.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.evaluation.MulticlassClassificationEvaluator.isSet">
<code class="descname">isSet</code><span class="sig-paren">(</span><em>param</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.evaluation.MulticlassClassificationEvaluator.isSet" title="Permalink to this definition">¶</a></dt>
<dd><p>Checks whether a param is explicitly set by user.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.evaluation.MulticlassClassificationEvaluator.labelCol">
<code class="descname">labelCol</code><em class="property"> = Param(parent='undefined', name='labelCol', doc='label column name')</em><a class="headerlink" href="#pyspark.ml.evaluation.MulticlassClassificationEvaluator.labelCol" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.evaluation.MulticlassClassificationEvaluator.metricName">
<code class="descname">metricName</code><em class="property"> = Param(parent='undefined', name='metricName', doc='metric name in evaluation (f1|precision|recall|weightedPrecision|weightedRecall)')</em><a class="headerlink" href="#pyspark.ml.evaluation.MulticlassClassificationEvaluator.metricName" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.evaluation.MulticlassClassificationEvaluator.params">
<code class="descname">params</code><a class="headerlink" href="#pyspark.ml.evaluation.MulticlassClassificationEvaluator.params" title="Permalink to this definition">¶</a></dt>
<dd><p>Returns all params ordered by name. The default implementation
uses <code class="xref py py-func docutils literal"><span class="pre">dir()</span></code> to get all attributes of type
<code class="xref py py-class docutils literal"><span class="pre">Param</span></code>.</p>
</dd></dl>
<dl class="attribute">
<dt id="pyspark.ml.evaluation.MulticlassClassificationEvaluator.predictionCol">
<code class="descname">predictionCol</code><em class="property"> = Param(parent='undefined', name='predictionCol', doc='prediction column name')</em><a class="headerlink" href="#pyspark.ml.evaluation.MulticlassClassificationEvaluator.predictionCol" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<dl class="method">
<dt id="pyspark.ml.evaluation.MulticlassClassificationEvaluator.setLabelCol">
<code class="descname">setLabelCol</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.evaluation.MulticlassClassificationEvaluator.setLabelCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.evaluation.MulticlassClassificationEvaluator.labelCol" title="pyspark.ml.evaluation.MulticlassClassificationEvaluator.labelCol"><code class="xref py py-attr docutils literal"><span class="pre">labelCol</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.evaluation.MulticlassClassificationEvaluator.setMetricName">
<code class="descname">setMetricName</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/evaluation.html#MulticlassClassificationEvaluator.setMetricName"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.evaluation.MulticlassClassificationEvaluator.setMetricName" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.evaluation.MulticlassClassificationEvaluator.metricName" title="pyspark.ml.evaluation.MulticlassClassificationEvaluator.metricName"><code class="xref py py-attr docutils literal"><span class="pre">metricName</span></code></a>.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.evaluation.MulticlassClassificationEvaluator.setParams">
<code class="descname">setParams</code><span class="sig-paren">(</span><em>self</em>, <em>predictionCol="prediction"</em>, <em>labelCol="label"</em>, <em>metricName="f1"</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/pyspark/ml/evaluation.html#MulticlassClassificationEvaluator.setParams"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#pyspark.ml.evaluation.MulticlassClassificationEvaluator.setParams" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets params for multiclass classification evaluator.</p>
</dd></dl>
<dl class="method">
<dt id="pyspark.ml.evaluation.MulticlassClassificationEvaluator.setPredictionCol">
<code class="descname">setPredictionCol</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="headerlink" href="#pyspark.ml.evaluation.MulticlassClassificationEvaluator.setPredictionCol" title="Permalink to this definition">¶</a></dt>
<dd><p>Sets the value of <a class="reference internal" href="#pyspark.ml.evaluation.MulticlassClassificationEvaluator.predictionCol" title="pyspark.ml.evaluation.MulticlassClassificationEvaluator.predictionCol"><code class="xref py py-attr docutils literal"><span class="pre">predictionCol</span></code></a>.</p>
</dd></dl>
</dd></dl>
</div>
</div>
</div>
</div>
</div>
<div class="sphinxsidebar" role="navigation" aria-label="main navigation">
<div class="sphinxsidebarwrapper">
<p class="logo"><a href="index.html">
<img class="logo" src="_static/spark-logo-hd.png" alt="Logo"/>
</a></p>
<h3><a href="index.html">Table Of Contents</a></h3>
<ul>
<li><a class="reference internal" href="#">pyspark.ml package</a><ul>
<li><a class="reference internal" href="#module-pyspark.ml">ML Pipeline APIs</a></li>
<li><a class="reference internal" href="#module-pyspark.ml.param">pyspark.ml.param module</a></li>
<li><a class="reference internal" href="#module-pyspark.ml.feature">pyspark.ml.feature module</a></li>
<li><a class="reference internal" href="#module-pyspark.ml.classification">pyspark.ml.classification module</a></li>
<li><a class="reference internal" href="#module-pyspark.ml.clustering">pyspark.ml.clustering module</a></li>
<li><a class="reference internal" href="#module-pyspark.ml.recommendation">pyspark.ml.recommendation module</a></li>
<li><a class="reference internal" href="#module-pyspark.ml.regression">pyspark.ml.regression module</a></li>
<li><a class="reference internal" href="#module-pyspark.ml.tuning">pyspark.ml.tuning module</a></li>
<li><a class="reference internal" href="#module-pyspark.ml.evaluation">pyspark.ml.evaluation module</a></li>
</ul>
</li>
</ul>
<h4>Previous topic</h4>
<p class="topless"><a href="pyspark.streaming.html"
title="previous chapter">pyspark.streaming module</a></p>
<h4>Next topic</h4>
<p class="topless"><a href="pyspark.mllib.html"
title="next chapter">pyspark.mllib package</a></p>
<div role="note" aria-label="source link">
<h3>This Page</h3>
<ul class="this-page-menu">
<li><a href="_sources/pyspark.ml.txt"
rel="nofollow">Show Source</a></li>
</ul>
</div>
<div id="searchbox" style="display: none" role="search">
<h3>Quick search</h3>
<form class="search" action="search.html" method="get">
<input type="text" name="q" />
<input type="submit" value="Go" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
<p class="searchtip" style="font-size: 90%">
Enter search terms or a module, class or function name.
</p>
</div>
<script type="text/javascript">$('#searchbox').show(0);</script>
</div>
</div>
<div class="clearer"></div>
</div>
<div class="related" role="navigation" aria-label="related navigation">
<h3>Navigation</h3>
<ul>
<li class="right" style="margin-right: 10px">
<a href="pyspark.mllib.html" title="pyspark.mllib package"
>next</a></li>
<li class="right" >
<a href="pyspark.streaming.html" title="pyspark.streaming module"
>previous</a> |</li>
<li class="nav-item nav-item-0"><a href="index.html">PySpark master documentation</a> »</li>
</ul>
</div>
<div class="footer" role="contentinfo">
© Copyright .
Created using <a href="http://sphinx-doc.org/">Sphinx</a> 1.3.1.
</div>
</body>
</html>