<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<!-- NewPage -->
<html lang="en">
<head>
<!-- Generated by javadoc (version 1.7.0_51) on Wed Sep 16 15:55:09 PDT 2015 -->
<title>DataFrame</title>
<meta name="date" content="2015-09-16">
<link rel="stylesheet" type="text/css" href="../../../../stylesheet.css" title="Style">
</head>
<body>
<script type="text/javascript"><!--
if (location.href.indexOf('is-external=true') == -1) {
parent.document.title="DataFrame";
}
//-->
</script>
<noscript>
<div>JavaScript is disabled on your browser.</div>
</noscript>
<!-- ========= START OF TOP NAVBAR ======= -->
<div class="topNav"><a name="navbar_top">
<!-- -->
</a><a href="#skip-navbar_top" title="Skip navigation links"></a><a name="navbar_top_firstrow">
<!-- -->
</a>
<ul class="navList" title="Navigation">
<li><a href="../../../../overview-summary.html">Overview</a></li>
<li><a href="package-summary.html">Package</a></li>
<li class="navBarCell1Rev">Class</li>
<li><a href="package-tree.html">Tree</a></li>
<li><a href="../../../../deprecated-list.html">Deprecated</a></li>
<li><a href="../../../../index-all.html">Index</a></li>
<li><a href="../../../../help-doc.html">Help</a></li>
</ul>
</div>
<div class="subNav">
<ul class="navList">
<li><a href="../../../../org/apache/spark/sql/ColumnName.html" title="class in org.apache.spark.sql"><span class="strong">Prev Class</span></a></li>
<li><a href="../../../../org/apache/spark/sql/DataFrameNaFunctions.html" title="class in org.apache.spark.sql"><span class="strong">Next Class</span></a></li>
</ul>
<ul class="navList">
<li><a href="../../../../index.html?org/apache/spark/sql/DataFrame.html" target="_top">Frames</a></li>
<li><a href="DataFrame.html" target="_top">No Frames</a></li>
</ul>
<ul class="navList" id="allclasses_navbar_top">
<li><a href="../../../../allclasses-noframe.html">All Classes</a></li>
</ul>
<div>
<script type="text/javascript"><!--
allClassesLink = document.getElementById("allclasses_navbar_top");
if(window==top) {
allClassesLink.style.display = "block";
}
else {
allClassesLink.style.display = "none";
}
//-->
</script>
</div>
<div>
<ul class="subNavList">
<li>Summary: </li>
<li>Nested | </li>
<li>Field | </li>
<li><a href="#constructor_summary">Constr</a> | </li>
<li><a href="#method_summary">Method</a></li>
</ul>
<ul class="subNavList">
<li>Detail: </li>
<li>Field | </li>
<li><a href="#constructor_detail">Constr</a> | </li>
<li><a href="#method_detail">Method</a></li>
</ul>
</div>
<a name="skip-navbar_top">
<!-- -->
</a></div>
<!-- ========= END OF TOP NAVBAR ========= -->
<!-- ======== START OF CLASS DATA ======== -->
<div class="header">
<div class="subTitle">org.apache.spark.sql</div>
<h2 title="Class DataFrame" class="title">Class DataFrame</h2>
</div>
<div class="contentContainer">
<ul class="inheritance">
<li>java.lang.Object</li>
<li>
<ul class="inheritance">
<li>org.apache.spark.sql.DataFrame</li>
</ul>
</li>
</ul>
<div class="description">
<ul class="blockList">
<li class="blockList">
<dl>
<dt>All Implemented Interfaces:</dt>
<dd>java.io.Serializable</dd>
</dl>
<hr>
<br>
<pre>public class <span class="strong">DataFrame</span>
extends java.lang.Object
implements scala.Serializable</pre>
<div class="block">:: Experimental ::
A distributed collection of data organized into named columns.
<p>
A <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> is equivalent to a relational table in Spark SQL. The following example creates
a <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> by pointing Spark SQL to a Parquet data set.
<pre><code>
val people = sqlContext.read.parquet("...") // in Scala
DataFrame people = sqlContext.read().parquet("...") // in Java
</code></pre>
<p>
Once created, it can be manipulated using the various domain-specific-language (DSL) functions
defined in: <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> (this class), <a href="../../../../org/apache/spark/sql/Column.html" title="class in org.apache.spark.sql"><code>Column</code></a>, and <a href="../../../../org/apache/spark/sql/functions.html" title="class in org.apache.spark.sql"><code>functions</code></a>.
<p>
To select a column from the data frame, use <code>apply</code> method in Scala and <code>col</code> in Java.
<pre><code>
val ageCol = people("age") // in Scala
Column ageCol = people.col("age") // in Java
</code></pre>
<p>
Note that the <a href="../../../../org/apache/spark/sql/Column.html" title="class in org.apache.spark.sql"><code>Column</code></a> type can also be manipulated through its various functions.
<pre><code>
// The following creates a new column that increases everybody's age by 10.
people("age") + 10 // in Scala
people.col("age").plus(10); // in Java
</code></pre>
<p>
A more concrete example in Scala:
<pre><code>
// To create DataFrame using SQLContext
val people = sqlContext.read.parquet("...")
val department = sqlContext.read.parquet("...")
people.filter("age > 30")
.join(department, people("deptId") === department("id"))
.groupBy(department("name"), "gender")
.agg(avg(people("salary")), max(people("age")))
</code></pre>
<p>
and in Java:
<pre><code>
// To create DataFrame using SQLContext
DataFrame people = sqlContext.read().parquet("...");
DataFrame department = sqlContext.read().parquet("...");
people.filter("age".gt(30))
.join(department, people.col("deptId").equalTo(department("id")))
.groupBy(department.col("name"), "gender")
.agg(avg(people.col("salary")), max(people.col("age")));
</code></pre>
<p></div>
<dl><dt><span class="strong">Since:</span></dt>
<dd>1.3.0</dd>
<dt><span class="strong">See Also:</span></dt><dd><a href="../../../../serialized-form.html#org.apache.spark.sql.DataFrame">Serialized Form</a></dd></dl>
</li>
</ul>
</div>
<div class="summary">
<ul class="blockList">
<li class="blockList">
<!-- ======== CONSTRUCTOR SUMMARY ======== -->
<ul class="blockList">
<li class="blockList"><a name="constructor_summary">
<!-- -->
</a>
<h3>Constructor Summary</h3>
<table class="overviewSummary" border="0" cellpadding="3" cellspacing="0" summary="Constructor Summary table, listing constructors, and an explanation">
<caption><span>Constructors</span><span class="tabEnd"> </span></caption>
<tr>
<th class="colOne" scope="col">Constructor and Description</th>
</tr>
<tr class="altColor">
<td class="colOne"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#DataFrame(org.apache.spark.sql.SQLContext, org.apache.spark.sql.catalyst.plans.logical.LogicalPlan)">DataFrame</a></strong>(<a href="../../../../org/apache/spark/sql/SQLContext.html" title="class in org.apache.spark.sql">SQLContext</a> sqlContext,
org.apache.spark.sql.catalyst.plans.logical.LogicalPlan logicalPlan)</code>
<div class="block">A constructor that automatically analyzes the logical plan.</div>
</td>
</tr>
</table>
</li>
</ul>
<!-- ========== METHOD SUMMARY =========== -->
<ul class="blockList">
<li class="blockList"><a name="method_summary">
<!-- -->
</a>
<h3>Method Summary</h3>
<table class="overviewSummary" border="0" cellpadding="3" cellspacing="0" summary="Method Summary table, listing methods, and an explanation">
<caption><span>Methods</span><span class="tabEnd"> </span></caption>
<tr>
<th class="colFirst" scope="col">Modifier and Type</th>
<th class="colLast" scope="col">Method and Description</th>
</tr>
<tr class="altColor">
<td class="colFirst"><code><a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#agg(org.apache.spark.sql.Column, org.apache.spark.sql.Column...)">agg</a></strong>(<a href="../../../../org/apache/spark/sql/Column.html" title="class in org.apache.spark.sql">Column</a> expr,
<a href="../../../../org/apache/spark/sql/Column.html" title="class in org.apache.spark.sql">Column</a>... exprs)</code>
<div class="block">Aggregates on the entire <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> without groups.</div>
</td>
</tr>
<tr class="rowColor">
<td class="colFirst"><code><a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#agg(org.apache.spark.sql.Column, scala.collection.Seq)">agg</a></strong>(<a href="../../../../org/apache/spark/sql/Column.html" title="class in org.apache.spark.sql">Column</a> expr,
scala.collection.Seq<<a href="../../../../org/apache/spark/sql/Column.html" title="class in org.apache.spark.sql">Column</a>> exprs)</code>
<div class="block">Aggregates on the entire <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> without groups.</div>
</td>
</tr>
<tr class="altColor">
<td class="colFirst"><code><a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#agg(scala.collection.immutable.Map)">agg</a></strong>(scala.collection.immutable.Map<java.lang.String,java.lang.String> exprs)</code>
<div class="block">(Scala-specific) Aggregates on the entire <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> without groups.</div>
</td>
</tr>
<tr class="rowColor">
<td class="colFirst"><code><a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#agg(java.util.Map)">agg</a></strong>(java.util.Map<java.lang.String,java.lang.String> exprs)</code>
<div class="block">(Java-specific) Aggregates on the entire <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> without groups.</div>
</td>
</tr>
<tr class="altColor">
<td class="colFirst"><code><a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#agg(scala.Tuple2, scala.collection.Seq)">agg</a></strong>(scala.Tuple2<java.lang.String,java.lang.String> aggExpr,
scala.collection.Seq<scala.Tuple2<java.lang.String,java.lang.String>> aggExprs)</code>
<div class="block">(Scala-specific) Aggregates on the entire <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> without groups.</div>
</td>
</tr>
<tr class="rowColor">
<td class="colFirst"><code><a href="../../../../org/apache/spark/sql/Column.html" title="class in org.apache.spark.sql">Column</a></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#apply(java.lang.String)">apply</a></strong>(java.lang.String colName)</code>
<div class="block">Selects column based on the column name and return it as a <a href="../../../../org/apache/spark/sql/Column.html" title="class in org.apache.spark.sql"><code>Column</code></a>.</div>
</td>
</tr>
<tr class="altColor">
<td class="colFirst"><code><a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#as(java.lang.String)">as</a></strong>(java.lang.String alias)</code>
<div class="block">Returns a new <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> with an alias set.</div>
</td>
</tr>
<tr class="rowColor">
<td class="colFirst"><code><a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#as(scala.Symbol)">as</a></strong>(scala.Symbol alias)</code>
<div class="block">(Scala-specific) Returns a new <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> with an alias set.</div>
</td>
</tr>
<tr class="altColor">
<td class="colFirst"><code><a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#cache()">cache</a></strong>()</code> </td>
</tr>
<tr class="rowColor">
<td class="colFirst"><code><a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#coalesce(int)">coalesce</a></strong>(int numPartitions)</code>
<div class="block">Returns a new <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> that has exactly <code>numPartitions</code> partitions.</div>
</td>
</tr>
<tr class="altColor">
<td class="colFirst"><code><a href="../../../../org/apache/spark/sql/Column.html" title="class in org.apache.spark.sql">Column</a></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#col(java.lang.String)">col</a></strong>(java.lang.String colName)</code>
<div class="block">Selects column based on the column name and return it as a <a href="../../../../org/apache/spark/sql/Column.html" title="class in org.apache.spark.sql"><code>Column</code></a>.</div>
</td>
</tr>
<tr class="rowColor">
<td class="colFirst"><code><a href="../../../../org/apache/spark/sql/Row.html" title="interface in org.apache.spark.sql">Row</a>[]</code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#collect()">collect</a></strong>()</code>
<div class="block">Returns an array that contains all of <a href="../../../../org/apache/spark/sql/Row.html" title="interface in org.apache.spark.sql"><code>Row</code></a>s in this <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a>.</div>
</td>
</tr>
<tr class="altColor">
<td class="colFirst"><code>java.util.List<<a href="../../../../org/apache/spark/sql/Row.html" title="interface in org.apache.spark.sql">Row</a>></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#collectAsList()">collectAsList</a></strong>()</code>
<div class="block">Returns a Java list that contains all of <a href="../../../../org/apache/spark/sql/Row.html" title="interface in org.apache.spark.sql"><code>Row</code></a>s in this <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a>.</div>
</td>
</tr>
<tr class="rowColor">
<td class="colFirst"><code>java.lang.String[]</code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#columns()">columns</a></strong>()</code>
<div class="block">Returns all column names as an array.</div>
</td>
</tr>
<tr class="altColor">
<td class="colFirst"><code>long</code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#count()">count</a></strong>()</code>
<div class="block">Returns the number of rows in the <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a>.</div>
</td>
</tr>
<tr class="rowColor">
<td class="colFirst"><code>void</code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#createJDBCTable(java.lang.String, java.lang.String, boolean)">createJDBCTable</a></strong>(java.lang.String url,
java.lang.String table,
boolean allowExisting)</code>
<div class="block"><strong>Deprecated.</strong>
<div class="block"><i>As of 1.340, replaced by <code>write().jdbc()</code>.</i></div>
</div>
</td>
</tr>
<tr class="altColor">
<td class="colFirst"><code><a href="../../../../org/apache/spark/sql/GroupedData.html" title="class in org.apache.spark.sql">GroupedData</a></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#cube(org.apache.spark.sql.Column...)">cube</a></strong>(<a href="../../../../org/apache/spark/sql/Column.html" title="class in org.apache.spark.sql">Column</a>... cols)</code>
<div class="block">Create a multi-dimensional cube for the current <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> using the specified columns,
so we can run aggregation on them.</div>
</td>
</tr>
<tr class="rowColor">
<td class="colFirst"><code><a href="../../../../org/apache/spark/sql/GroupedData.html" title="class in org.apache.spark.sql">GroupedData</a></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#cube(scala.collection.Seq)">cube</a></strong>(scala.collection.Seq<<a href="../../../../org/apache/spark/sql/Column.html" title="class in org.apache.spark.sql">Column</a>> cols)</code>
<div class="block">Create a multi-dimensional cube for the current <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> using the specified columns,
so we can run aggregation on them.</div>
</td>
</tr>
<tr class="altColor">
<td class="colFirst"><code><a href="../../../../org/apache/spark/sql/GroupedData.html" title="class in org.apache.spark.sql">GroupedData</a></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#cube(java.lang.String, scala.collection.Seq)">cube</a></strong>(java.lang.String col1,
scala.collection.Seq<java.lang.String> cols)</code>
<div class="block">Create a multi-dimensional cube for the current <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> using the specified columns,
so we can run aggregation on them.</div>
</td>
</tr>
<tr class="rowColor">
<td class="colFirst"><code><a href="../../../../org/apache/spark/sql/GroupedData.html" title="class in org.apache.spark.sql">GroupedData</a></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#cube(java.lang.String, java.lang.String...)">cube</a></strong>(java.lang.String col1,
java.lang.String... cols)</code>
<div class="block">Create a multi-dimensional cube for the current <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> using the specified columns,
so we can run aggregation on them.</div>
</td>
</tr>
<tr class="altColor">
<td class="colFirst"><code><a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#describe(scala.collection.Seq)">describe</a></strong>(scala.collection.Seq<java.lang.String> cols)</code>
<div class="block">Computes statistics for numeric columns, including count, mean, stddev, min, and max.</div>
</td>
</tr>
<tr class="rowColor">
<td class="colFirst"><code><a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#describe(java.lang.String...)">describe</a></strong>(java.lang.String... cols)</code>
<div class="block">Computes statistics for numeric columns, including count, mean, stddev, min, and max.</div>
</td>
</tr>
<tr class="altColor">
<td class="colFirst"><code><a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#distinct()">distinct</a></strong>()</code>
<div class="block">Returns a new <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> that contains only the unique rows from this <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a>.</div>
</td>
</tr>
<tr class="rowColor">
<td class="colFirst"><code><a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#drop(org.apache.spark.sql.Column)">drop</a></strong>(<a href="../../../../org/apache/spark/sql/Column.html" title="class in org.apache.spark.sql">Column</a> col)</code>
<div class="block">Returns a new <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> with a column dropped.</div>
</td>
</tr>
<tr class="altColor">
<td class="colFirst"><code><a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#drop(java.lang.String)">drop</a></strong>(java.lang.String colName)</code>
<div class="block">Returns a new <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> with a column dropped.</div>
</td>
</tr>
<tr class="rowColor">
<td class="colFirst"><code><a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#dropDuplicates()">dropDuplicates</a></strong>()</code>
<div class="block">Returns a new <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> that contains only the unique rows from this <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a>.</div>
</td>
</tr>
<tr class="altColor">
<td class="colFirst"><code><a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#dropDuplicates(scala.collection.Seq)">dropDuplicates</a></strong>(scala.collection.Seq<java.lang.String> colNames)</code>
<div class="block">(Scala-specific) Returns a new <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> with duplicate rows removed, considering only
the subset of columns.</div>
</td>
</tr>
<tr class="rowColor">
<td class="colFirst"><code><a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#dropDuplicates(java.lang.String[])">dropDuplicates</a></strong>(java.lang.String[] colNames)</code>
<div class="block">Returns a new <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> with duplicate rows removed, considering only
the subset of columns.</div>
</td>
</tr>
<tr class="altColor">
<td class="colFirst"><code>scala.Tuple2<java.lang.String,java.lang.String>[]</code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#dtypes()">dtypes</a></strong>()</code>
<div class="block">Returns all column names and their data types as an array.</div>
</td>
</tr>
<tr class="rowColor">
<td class="colFirst"><code><a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#except(org.apache.spark.sql.DataFrame)">except</a></strong>(<a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a> other)</code>
<div class="block">Returns a new <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> containing rows in this frame but not in another frame.</div>
</td>
</tr>
<tr class="altColor">
<td class="colFirst"><code>void</code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#explain()">explain</a></strong>()</code>
<div class="block">Only prints the physical plan to the console for debugging purposes.</div>
</td>
</tr>
<tr class="rowColor">
<td class="colFirst"><code>void</code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#explain(boolean)">explain</a></strong>(boolean extended)</code>
<div class="block">Prints the plans (logical and physical) to the console for debugging purposes.</div>
</td>
</tr>
<tr class="altColor">
<td class="colFirst"><code><A extends scala.Product> <br><a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#explode(scala.collection.Seq, scala.Function1, scala.reflect.api.TypeTags.TypeTag)">explode</a></strong>(scala.collection.Seq<<a href="../../../../org/apache/spark/sql/Column.html" title="class in org.apache.spark.sql">Column</a>> input,
scala.Function1<<a href="../../../../org/apache/spark/sql/Row.html" title="interface in org.apache.spark.sql">Row</a>,scala.collection.TraversableOnce<A>> f,
scala.reflect.api.TypeTags.TypeTag<A> evidence$1)</code>
<div class="block">(Scala-specific) Returns a new <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> where each row has been expanded to zero or more
rows by the provided function.</div>
</td>
</tr>
<tr class="rowColor">
<td class="colFirst"><code><A,B> <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#explode(java.lang.String, java.lang.String, scala.Function1, scala.reflect.api.TypeTags.TypeTag)">explode</a></strong>(java.lang.String inputColumn,
java.lang.String outputColumn,
scala.Function1<A,scala.collection.TraversableOnce<B>> f,
scala.reflect.api.TypeTags.TypeTag<B> evidence$2)</code>
<div class="block">(Scala-specific) Returns a new <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> where a single column has been expanded to zero
or more rows by the provided function.</div>
</td>
</tr>
<tr class="altColor">
<td class="colFirst"><code><a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#filter(org.apache.spark.sql.Column)">filter</a></strong>(<a href="../../../../org/apache/spark/sql/Column.html" title="class in org.apache.spark.sql">Column</a> condition)</code>
<div class="block">Filters rows using the given condition.</div>
</td>
</tr>
<tr class="rowColor">
<td class="colFirst"><code><a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#filter(java.lang.String)">filter</a></strong>(java.lang.String conditionExpr)</code>
<div class="block">Filters rows using the given SQL expression.</div>
</td>
</tr>
<tr class="altColor">
<td class="colFirst"><code><a href="../../../../org/apache/spark/sql/Row.html" title="interface in org.apache.spark.sql">Row</a></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#first()">first</a></strong>()</code>
<div class="block">Returns the first row.</div>
</td>
</tr>
<tr class="rowColor">
<td class="colFirst"><code><R> <a href="../../../../org/apache/spark/rdd/RDD.html" title="class in org.apache.spark.rdd">RDD</a><R></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#flatMap(scala.Function1, scala.reflect.ClassTag)">flatMap</a></strong>(scala.Function1<<a href="../../../../org/apache/spark/sql/Row.html" title="interface in org.apache.spark.sql">Row</a>,scala.collection.TraversableOnce<R>> f,
scala.reflect.ClassTag<R> evidence$4)</code>
<div class="block">Returns a new RDD by first applying a function to all rows of this <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a>,
and then flattening the results.</div>
</td>
</tr>
<tr class="altColor">
<td class="colFirst"><code>void</code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#foreach(scala.Function1)">foreach</a></strong>(scala.Function1<<a href="../../../../org/apache/spark/sql/Row.html" title="interface in org.apache.spark.sql">Row</a>,scala.runtime.BoxedUnit> f)</code>
<div class="block">Applies a function <code>f</code> to all rows.</div>
</td>
</tr>
<tr class="rowColor">
<td class="colFirst"><code>void</code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#foreachPartition(scala.Function1)">foreachPartition</a></strong>(scala.Function1<scala.collection.Iterator<<a href="../../../../org/apache/spark/sql/Row.html" title="interface in org.apache.spark.sql">Row</a>>,scala.runtime.BoxedUnit> f)</code>
<div class="block">Applies a function f to each partition of this <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a>.</div>
</td>
</tr>
<tr class="altColor">
<td class="colFirst"><code><a href="../../../../org/apache/spark/sql/GroupedData.html" title="class in org.apache.spark.sql">GroupedData</a></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#groupBy(org.apache.spark.sql.Column...)">groupBy</a></strong>(<a href="../../../../org/apache/spark/sql/Column.html" title="class in org.apache.spark.sql">Column</a>... cols)</code>
<div class="block">Groups the <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> using the specified columns, so we can run aggregation on them.</div>
</td>
</tr>
<tr class="rowColor">
<td class="colFirst"><code><a href="../../../../org/apache/spark/sql/GroupedData.html" title="class in org.apache.spark.sql">GroupedData</a></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#groupBy(scala.collection.Seq)">groupBy</a></strong>(scala.collection.Seq<<a href="../../../../org/apache/spark/sql/Column.html" title="class in org.apache.spark.sql">Column</a>> cols)</code>
<div class="block">Groups the <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> using the specified columns, so we can run aggregation on them.</div>
</td>
</tr>
<tr class="altColor">
<td class="colFirst"><code><a href="../../../../org/apache/spark/sql/GroupedData.html" title="class in org.apache.spark.sql">GroupedData</a></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#groupBy(java.lang.String, scala.collection.Seq)">groupBy</a></strong>(java.lang.String col1,
scala.collection.Seq<java.lang.String> cols)</code>
<div class="block">Groups the <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> using the specified columns, so we can run aggregation on them.</div>
</td>
</tr>
<tr class="rowColor">
<td class="colFirst"><code><a href="../../../../org/apache/spark/sql/GroupedData.html" title="class in org.apache.spark.sql">GroupedData</a></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#groupBy(java.lang.String, java.lang.String...)">groupBy</a></strong>(java.lang.String col1,
java.lang.String... cols)</code>
<div class="block">Groups the <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> using the specified columns, so we can run aggregation on them.</div>
</td>
</tr>
<tr class="altColor">
<td class="colFirst"><code><a href="../../../../org/apache/spark/sql/Row.html" title="interface in org.apache.spark.sql">Row</a></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#head()">head</a></strong>()</code>
<div class="block">Returns the first row.</div>
</td>
</tr>
<tr class="rowColor">
<td class="colFirst"><code><a href="../../../../org/apache/spark/sql/Row.html" title="interface in org.apache.spark.sql">Row</a>[]</code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#head(int)">head</a></strong>(int n)</code>
<div class="block">Returns the first <code>n</code> rows.</div>
</td>
</tr>
<tr class="altColor">
<td class="colFirst"><code>java.lang.String[]</code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#inputFiles()">inputFiles</a></strong>()</code>
<div class="block">Returns a best-effort snapshot of the files that compose this DataFrame.</div>
</td>
</tr>
<tr class="rowColor">
<td class="colFirst"><code>void</code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#insertInto(java.lang.String)">insertInto</a></strong>(java.lang.String tableName)</code>
<div class="block"><strong>Deprecated.</strong>
<div class="block"><i>As of 1.4.0, replaced by
<code>write().mode(SaveMode.Append).saveAsTable(tableName)</code>.</i></div>
</div>
</td>
</tr>
<tr class="altColor">
<td class="colFirst"><code>void</code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#insertInto(java.lang.String, boolean)">insertInto</a></strong>(java.lang.String tableName,
boolean overwrite)</code>
<div class="block"><strong>Deprecated.</strong>
<div class="block"><i>As of 1.4.0, replaced by
<code>write().mode(SaveMode.Append|SaveMode.Overwrite).saveAsTable(tableName)</code>.</i></div>
</div>
</td>
</tr>
<tr class="rowColor">
<td class="colFirst"><code>void</code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#insertIntoJDBC(java.lang.String, java.lang.String, boolean)">insertIntoJDBC</a></strong>(java.lang.String url,
java.lang.String table,
boolean overwrite)</code>
<div class="block"><strong>Deprecated.</strong>
<div class="block"><i>As of 1.4.0, replaced by <code>write().jdbc()</code>.</i></div>
</div>
</td>
</tr>
<tr class="altColor">
<td class="colFirst"><code><a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#intersect(org.apache.spark.sql.DataFrame)">intersect</a></strong>(<a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a> other)</code>
<div class="block">Returns a new <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> containing rows only in both this frame and another frame.</div>
</td>
</tr>
<tr class="rowColor">
<td class="colFirst"><code>boolean</code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#isLocal()">isLocal</a></strong>()</code>
<div class="block">Returns true if the <code>collect</code> and <code>take</code> methods can be run locally
(without any Spark executors).</div>
</td>
</tr>
<tr class="altColor">
<td class="colFirst"><code><a href="../../../../org/apache/spark/api/java/JavaRDD.html" title="class in org.apache.spark.api.java">JavaRDD</a><<a href="../../../../org/apache/spark/sql/Row.html" title="interface in org.apache.spark.sql">Row</a>></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#javaRDD()">javaRDD</a></strong>()</code>
<div class="block">Returns the content of the <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> as a <code>JavaRDD</code> of <a href="../../../../org/apache/spark/sql/Row.html" title="interface in org.apache.spark.sql"><code>Row</code></a>s.</div>
</td>
</tr>
<tr class="rowColor">
<td class="colFirst"><code>protected <a href="../../../../org/apache/spark/api/java/JavaRDD.html" title="class in org.apache.spark.api.java">JavaRDD</a><byte[]></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#javaToPython()">javaToPython</a></strong>()</code>
<div class="block">Converts a JavaRDD to a PythonRDD.</div>
</td>
</tr>
<tr class="altColor">
<td class="colFirst"><code><a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#join(org.apache.spark.sql.DataFrame)">join</a></strong>(<a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a> right)</code>
<div class="block">Cartesian join with another <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a>.</div>
</td>
</tr>
<tr class="rowColor">
<td class="colFirst"><code><a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#join(org.apache.spark.sql.DataFrame, org.apache.spark.sql.Column)">join</a></strong>(<a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a> right,
<a href="../../../../org/apache/spark/sql/Column.html" title="class in org.apache.spark.sql">Column</a> joinExprs)</code>
<div class="block">Inner join with another <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a>, using the given join expression.</div>
</td>
</tr>
<tr class="altColor">
<td class="colFirst"><code><a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#join(org.apache.spark.sql.DataFrame, org.apache.spark.sql.Column, java.lang.String)">join</a></strong>(<a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a> right,
<a href="../../../../org/apache/spark/sql/Column.html" title="class in org.apache.spark.sql">Column</a> joinExprs,
java.lang.String joinType)</code>
<div class="block">Join with another <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a>, using the given join expression.</div>
</td>
</tr>
<tr class="rowColor">
<td class="colFirst"><code><a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#join(org.apache.spark.sql.DataFrame, scala.collection.Seq)">join</a></strong>(<a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a> right,
scala.collection.Seq<java.lang.String> usingColumns)</code>
<div class="block">Inner equi-join with another <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> using the given columns.</div>
</td>
</tr>
<tr class="altColor">
<td class="colFirst"><code><a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#join(org.apache.spark.sql.DataFrame, java.lang.String)">join</a></strong>(<a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a> right,
java.lang.String usingColumn)</code>
<div class="block">Inner equi-join with another <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> using the given column.</div>
</td>
</tr>
<tr class="rowColor">
<td class="colFirst"><code><a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#limit(int)">limit</a></strong>(int n)</code>
<div class="block">Returns a new <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> by taking the first <code>n</code> rows.</div>
</td>
</tr>
<tr class="altColor">
<td class="colFirst"><code>protected org.apache.spark.sql.catalyst.plans.logical.LogicalPlan</code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#logicalPlan()">logicalPlan</a></strong>()</code> </td>
</tr>
<tr class="rowColor">
<td class="colFirst"><code><R> <a href="../../../../org/apache/spark/rdd/RDD.html" title="class in org.apache.spark.rdd">RDD</a><R></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#map(scala.Function1, scala.reflect.ClassTag)">map</a></strong>(scala.Function1<<a href="../../../../org/apache/spark/sql/Row.html" title="interface in org.apache.spark.sql">Row</a>,R> f,
scala.reflect.ClassTag<R> evidence$3)</code>
<div class="block">Returns a new RDD by applying a function to all rows of this DataFrame.</div>
</td>
</tr>
<tr class="altColor">
<td class="colFirst"><code><R> <a href="../../../../org/apache/spark/rdd/RDD.html" title="class in org.apache.spark.rdd">RDD</a><R></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#mapPartitions(scala.Function1, scala.reflect.ClassTag)">mapPartitions</a></strong>(scala.Function1<scala.collection.Iterator<<a href="../../../../org/apache/spark/sql/Row.html" title="interface in org.apache.spark.sql">Row</a>>,scala.collection.Iterator<R>> f,
scala.reflect.ClassTag<R> evidence$5)</code>
<div class="block">Returns a new RDD by applying a function to each partition of this DataFrame.</div>
</td>
</tr>
<tr class="rowColor">
<td class="colFirst"><code><a href="../../../../org/apache/spark/sql/DataFrameNaFunctions.html" title="class in org.apache.spark.sql">DataFrameNaFunctions</a></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#na()">na</a></strong>()</code>
<div class="block">Returns a <a href="../../../../org/apache/spark/sql/DataFrameNaFunctions.html" title="class in org.apache.spark.sql"><code>DataFrameNaFunctions</code></a> for working with missing data.</div>
</td>
</tr>
<tr class="altColor">
<td class="colFirst"><code>protected scala.collection.Seq<org.apache.spark.sql.catalyst.expressions.Expression></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#numericColumns()">numericColumns</a></strong>()</code> </td>
</tr>
<tr class="rowColor">
<td class="colFirst"><code><a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#orderBy(org.apache.spark.sql.Column...)">orderBy</a></strong>(<a href="../../../../org/apache/spark/sql/Column.html" title="class in org.apache.spark.sql">Column</a>... sortExprs)</code>
<div class="block">Returns a new <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> sorted by the given expressions.</div>
</td>
</tr>
<tr class="altColor">
<td class="colFirst"><code><a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#orderBy(scala.collection.Seq)">orderBy</a></strong>(scala.collection.Seq<<a href="../../../../org/apache/spark/sql/Column.html" title="class in org.apache.spark.sql">Column</a>> sortExprs)</code>
<div class="block">Returns a new <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> sorted by the given expressions.</div>
</td>
</tr>
<tr class="rowColor">
<td class="colFirst"><code><a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#orderBy(java.lang.String, scala.collection.Seq)">orderBy</a></strong>(java.lang.String sortCol,
scala.collection.Seq<java.lang.String> sortCols)</code>
<div class="block">Returns a new <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> sorted by the given expressions.</div>
</td>
</tr>
<tr class="altColor">
<td class="colFirst"><code><a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#orderBy(java.lang.String, java.lang.String...)">orderBy</a></strong>(java.lang.String sortCol,
java.lang.String... sortCols)</code>
<div class="block">Returns a new <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> sorted by the given expressions.</div>
</td>
</tr>
<tr class="rowColor">
<td class="colFirst"><code><a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#persist()">persist</a></strong>()</code> </td>
</tr>
<tr class="altColor">
<td class="colFirst"><code><a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#persist(org.apache.spark.storage.StorageLevel)">persist</a></strong>(<a href="../../../../org/apache/spark/storage/StorageLevel.html" title="class in org.apache.spark.storage">StorageLevel</a> newLevel)</code> </td>
</tr>
<tr class="rowColor">
<td class="colFirst"><code>void</code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#printSchema()">printSchema</a></strong>()</code>
<div class="block">Prints the schema to the console in a nice tree format.</div>
</td>
</tr>
<tr class="altColor">
<td class="colFirst"><code><a href="../../../../org/apache/spark/sql/SQLContext.QueryExecution.html" title="class in org.apache.spark.sql">SQLContext.QueryExecution</a></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#queryExecution()">queryExecution</a></strong>()</code> </td>
</tr>
<tr class="rowColor">
<td class="colFirst"><code><a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a>[]</code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#randomSplit(double[])">randomSplit</a></strong>(double[] weights)</code>
<div class="block">Randomly splits this <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> with the provided weights.</div>
</td>
</tr>
<tr class="altColor">
<td class="colFirst"><code><a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a>[]</code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#randomSplit(double[], long)">randomSplit</a></strong>(double[] weights,
long seed)</code>
<div class="block">Randomly splits this <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> with the provided weights.</div>
</td>
</tr>
<tr class="rowColor">
<td class="colFirst"><code><a href="../../../../org/apache/spark/rdd/RDD.html" title="class in org.apache.spark.rdd">RDD</a><<a href="../../../../org/apache/spark/sql/Row.html" title="interface in org.apache.spark.sql">Row</a>></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#rdd()">rdd</a></strong>()</code>
<div class="block">Represents the content of the <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> as an <code>RDD</code> of <a href="../../../../org/apache/spark/sql/Row.html" title="interface in org.apache.spark.sql"><code>Row</code></a>s.</div>
</td>
</tr>
<tr class="altColor">
<td class="colFirst"><code>void</code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#registerTempTable(java.lang.String)">registerTempTable</a></strong>(java.lang.String tableName)</code>
<div class="block">Registers this <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> as a temporary table using the given name.</div>
</td>
</tr>
<tr class="rowColor">
<td class="colFirst"><code><a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#repartition(int)">repartition</a></strong>(int numPartitions)</code>
<div class="block">Returns a new <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> that has exactly <code>numPartitions</code> partitions.</div>
</td>
</tr>
<tr class="altColor">
<td class="colFirst"><code>protected org.apache.spark.sql.catalyst.expressions.NamedExpression</code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#resolve(java.lang.String)">resolve</a></strong>(java.lang.String colName)</code> </td>
</tr>
<tr class="rowColor">
<td class="colFirst"><code><a href="../../../../org/apache/spark/sql/GroupedData.html" title="class in org.apache.spark.sql">GroupedData</a></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#rollup(org.apache.spark.sql.Column...)">rollup</a></strong>(<a href="../../../../org/apache/spark/sql/Column.html" title="class in org.apache.spark.sql">Column</a>... cols)</code>
<div class="block">Create a multi-dimensional rollup for the current <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> using the specified columns,
so we can run aggregation on them.</div>
</td>
</tr>
<tr class="altColor">
<td class="colFirst"><code><a href="../../../../org/apache/spark/sql/GroupedData.html" title="class in org.apache.spark.sql">GroupedData</a></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#rollup(scala.collection.Seq)">rollup</a></strong>(scala.collection.Seq<<a href="../../../../org/apache/spark/sql/Column.html" title="class in org.apache.spark.sql">Column</a>> cols)</code>
<div class="block">Create a multi-dimensional rollup for the current <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> using the specified columns,
so we can run aggregation on them.</div>
</td>
</tr>
<tr class="rowColor">
<td class="colFirst"><code><a href="../../../../org/apache/spark/sql/GroupedData.html" title="class in org.apache.spark.sql">GroupedData</a></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#rollup(java.lang.String, scala.collection.Seq)">rollup</a></strong>(java.lang.String col1,
scala.collection.Seq<java.lang.String> cols)</code>
<div class="block">Create a multi-dimensional rollup for the current <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> using the specified columns,
so we can run aggregation on them.</div>
</td>
</tr>
<tr class="altColor">
<td class="colFirst"><code><a href="../../../../org/apache/spark/sql/GroupedData.html" title="class in org.apache.spark.sql">GroupedData</a></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#rollup(java.lang.String, java.lang.String...)">rollup</a></strong>(java.lang.String col1,
java.lang.String... cols)</code>
<div class="block">Create a multi-dimensional rollup for the current <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> using the specified columns,
so we can run aggregation on them.</div>
</td>
</tr>
<tr class="rowColor">
<td class="colFirst"><code><a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#sample(boolean, double)">sample</a></strong>(boolean withReplacement,
double fraction)</code>
<div class="block">Returns a new <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> by sampling a fraction of rows, using a random seed.</div>
</td>
</tr>
<tr class="altColor">
<td class="colFirst"><code><a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#sample(boolean, double, long)">sample</a></strong>(boolean withReplacement,
double fraction,
long seed)</code>
<div class="block">Returns a new <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> by sampling a fraction of rows.</div>
</td>
</tr>
<tr class="rowColor">
<td class="colFirst"><code>void</code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#save(java.lang.String)">save</a></strong>(java.lang.String path)</code>
<div class="block"><strong>Deprecated.</strong>
<div class="block"><i>As of 1.4.0, replaced by <code>write().save(path)</code>.</i></div>
</div>
</td>
</tr>
<tr class="altColor">
<td class="colFirst"><code>void</code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#save(java.lang.String, org.apache.spark.sql.SaveMode)">save</a></strong>(java.lang.String path,
<a href="../../../../org/apache/spark/sql/SaveMode.html" title="enum in org.apache.spark.sql">SaveMode</a> mode)</code>
<div class="block"><strong>Deprecated.</strong>
<div class="block"><i>As of 1.4.0, replaced by <code>write().mode(mode).save(path)</code>.</i></div>
</div>
</td>
</tr>
<tr class="rowColor">
<td class="colFirst"><code>void</code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#save(java.lang.String, org.apache.spark.sql.SaveMode, java.util.Map)">save</a></strong>(java.lang.String source,
<a href="../../../../org/apache/spark/sql/SaveMode.html" title="enum in org.apache.spark.sql">SaveMode</a> mode,
java.util.Map<java.lang.String,java.lang.String> options)</code>
<div class="block"><strong>Deprecated.</strong>
<div class="block"><i>As of 1.4.0, replaced by
<code>write().format(source).mode(mode).options(options).save(path)</code>.</i></div>
</div>
</td>
</tr>
<tr class="altColor">
<td class="colFirst"><code>void</code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#save(java.lang.String, org.apache.spark.sql.SaveMode, scala.collection.immutable.Map)">save</a></strong>(java.lang.String source,
<a href="../../../../org/apache/spark/sql/SaveMode.html" title="enum in org.apache.spark.sql">SaveMode</a> mode,
scala.collection.immutable.Map<java.lang.String,java.lang.String> options)</code>
<div class="block"><strong>Deprecated.</strong>
<div class="block"><i>As of 1.4.0, replaced by
<code>write().format(source).mode(mode).options(options).save(path)</code>.</i></div>
</div>
</td>
</tr>
<tr class="rowColor">
<td class="colFirst"><code>void</code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#save(java.lang.String, java.lang.String)">save</a></strong>(java.lang.String path,
java.lang.String source)</code>
<div class="block"><strong>Deprecated.</strong>
<div class="block"><i>As of 1.4.0, replaced by <code>write().format(source).save(path)</code>.</i></div>
</div>
</td>
</tr>
<tr class="altColor">
<td class="colFirst"><code>void</code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#save(java.lang.String, java.lang.String, org.apache.spark.sql.SaveMode)">save</a></strong>(java.lang.String path,
java.lang.String source,
<a href="../../../../org/apache/spark/sql/SaveMode.html" title="enum in org.apache.spark.sql">SaveMode</a> mode)</code>
<div class="block"><strong>Deprecated.</strong>
<div class="block"><i>As of 1.4.0, replaced by <code>write().format(source).mode(mode).save(path)</code>.</i></div>
</div>
</td>
</tr>
<tr class="rowColor">
<td class="colFirst"><code>void</code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#saveAsParquetFile(java.lang.String)">saveAsParquetFile</a></strong>(java.lang.String path)</code>
<div class="block"><strong>Deprecated.</strong>
<div class="block"><i>As of 1.4.0, replaced by <code>write().parquet()</code>.</i></div>
</div>
</td>
</tr>
<tr class="altColor">
<td class="colFirst"><code>void</code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#saveAsTable(java.lang.String)">saveAsTable</a></strong>(java.lang.String tableName)</code>
<div class="block"><strong>Deprecated.</strong>
<div class="block"><i>As of 1.4.0, replaced by <code>write().saveAsTable(tableName)</code>.</i></div>
</div>
</td>
</tr>
<tr class="rowColor">
<td class="colFirst"><code>void</code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#saveAsTable(java.lang.String, org.apache.spark.sql.SaveMode)">saveAsTable</a></strong>(java.lang.String tableName,
<a href="../../../../org/apache/spark/sql/SaveMode.html" title="enum in org.apache.spark.sql">SaveMode</a> mode)</code>
<div class="block"><strong>Deprecated.</strong>
<div class="block"><i>As of 1.4.0, replaced by <code>write().mode(mode).saveAsTable(tableName)</code>.</i></div>
</div>
</td>
</tr>
<tr class="altColor">
<td class="colFirst"><code>void</code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#saveAsTable(java.lang.String, java.lang.String)">saveAsTable</a></strong>(java.lang.String tableName,
java.lang.String source)</code>
<div class="block"><strong>Deprecated.</strong>
<div class="block"><i>As of 1.4.0, replaced by <code>write().format(source).saveAsTable(tableName)</code>.</i></div>
</div>
</td>
</tr>
<tr class="rowColor">
<td class="colFirst"><code>void</code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#saveAsTable(java.lang.String, java.lang.String, org.apache.spark.sql.SaveMode)">saveAsTable</a></strong>(java.lang.String tableName,
java.lang.String source,
<a href="../../../../org/apache/spark/sql/SaveMode.html" title="enum in org.apache.spark.sql">SaveMode</a> mode)</code>
<div class="block"><strong>Deprecated.</strong>
<div class="block"><i>As of 1.4.0, replaced by <code>write().mode(mode).saveAsTable(tableName)</code>.</i></div>
</div>
</td>
</tr>
<tr class="altColor">
<td class="colFirst"><code>void</code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#saveAsTable(java.lang.String, java.lang.String, org.apache.spark.sql.SaveMode, java.util.Map)">saveAsTable</a></strong>(java.lang.String tableName,
java.lang.String source,
<a href="../../../../org/apache/spark/sql/SaveMode.html" title="enum in org.apache.spark.sql">SaveMode</a> mode,
java.util.Map<java.lang.String,java.lang.String> options)</code>
<div class="block"><strong>Deprecated.</strong>
<div class="block"><i>As of 1.4.0, replaced by
<code>write().format(source).mode(mode).options(options).saveAsTable(tableName)</code>.</i></div>
</div>
</td>
</tr>
<tr class="rowColor">
<td class="colFirst"><code>void</code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#saveAsTable(java.lang.String, java.lang.String, org.apache.spark.sql.SaveMode, scala.collection.immutable.Map)">saveAsTable</a></strong>(java.lang.String tableName,
java.lang.String source,
<a href="../../../../org/apache/spark/sql/SaveMode.html" title="enum in org.apache.spark.sql">SaveMode</a> mode,
scala.collection.immutable.Map<java.lang.String,java.lang.String> options)</code>
<div class="block"><strong>Deprecated.</strong>
<div class="block"><i>As of 1.4.0, replaced by
<code>write().format(source).mode(mode).options(options).saveAsTable(tableName)</code>.</i></div>
</div>
</td>
</tr>
<tr class="altColor">
<td class="colFirst"><code><a href="../../../../org/apache/spark/sql/types/StructType.html" title="class in org.apache.spark.sql.types">StructType</a></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#schema()">schema</a></strong>()</code>
<div class="block">Returns the schema of this <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a>.</div>
</td>
</tr>
<tr class="rowColor">
<td class="colFirst"><code><a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#select(org.apache.spark.sql.Column...)">select</a></strong>(<a href="../../../../org/apache/spark/sql/Column.html" title="class in org.apache.spark.sql">Column</a>... cols)</code>
<div class="block">Selects a set of column based expressions.</div>
</td>
</tr>
<tr class="altColor">
<td class="colFirst"><code><a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#select(scala.collection.Seq)">select</a></strong>(scala.collection.Seq<<a href="../../../../org/apache/spark/sql/Column.html" title="class in org.apache.spark.sql">Column</a>> cols)</code>
<div class="block">Selects a set of column based expressions.</div>
</td>
</tr>
<tr class="rowColor">
<td class="colFirst"><code><a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#select(java.lang.String, scala.collection.Seq)">select</a></strong>(java.lang.String col,
scala.collection.Seq<java.lang.String> cols)</code>
<div class="block">Selects a set of columns.</div>
</td>
</tr>
<tr class="altColor">
<td class="colFirst"><code><a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#select(java.lang.String, java.lang.String...)">select</a></strong>(java.lang.String col,
java.lang.String... cols)</code>
<div class="block">Selects a set of columns.</div>
</td>
</tr>
<tr class="rowColor">
<td class="colFirst"><code><a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#selectExpr(scala.collection.Seq)">selectExpr</a></strong>(scala.collection.Seq<java.lang.String> exprs)</code>
<div class="block">Selects a set of SQL expressions.</div>
</td>
</tr>
<tr class="altColor">
<td class="colFirst"><code><a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#selectExpr(java.lang.String...)">selectExpr</a></strong>(java.lang.String... exprs)</code>
<div class="block">Selects a set of SQL expressions.</div>
</td>
</tr>
<tr class="rowColor">
<td class="colFirst"><code>void</code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#show()">show</a></strong>()</code>
<div class="block">Displays the top 20 rows of <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> in a tabular form.</div>
</td>
</tr>
<tr class="altColor">
<td class="colFirst"><code>void</code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#show(boolean)">show</a></strong>(boolean truncate)</code>
<div class="block">Displays the top 20 rows of <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> in a tabular form.</div>
</td>
</tr>
<tr class="rowColor">
<td class="colFirst"><code>void</code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#show(int)">show</a></strong>(int numRows)</code>
<div class="block">Displays the <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> in a tabular form.</div>
</td>
</tr>
<tr class="altColor">
<td class="colFirst"><code>void</code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#show(int, boolean)">show</a></strong>(int numRows,
boolean truncate)</code>
<div class="block">Displays the <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> in a tabular form.</div>
</td>
</tr>
<tr class="rowColor">
<td class="colFirst"><code><a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#sort(org.apache.spark.sql.Column...)">sort</a></strong>(<a href="../../../../org/apache/spark/sql/Column.html" title="class in org.apache.spark.sql">Column</a>... sortExprs)</code>
<div class="block">Returns a new <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> sorted by the given expressions.</div>
</td>
</tr>
<tr class="altColor">
<td class="colFirst"><code><a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#sort(scala.collection.Seq)">sort</a></strong>(scala.collection.Seq<<a href="../../../../org/apache/spark/sql/Column.html" title="class in org.apache.spark.sql">Column</a>> sortExprs)</code>
<div class="block">Returns a new <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> sorted by the given expressions.</div>
</td>
</tr>
<tr class="rowColor">
<td class="colFirst"><code><a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#sort(java.lang.String, scala.collection.Seq)">sort</a></strong>(java.lang.String sortCol,
scala.collection.Seq<java.lang.String> sortCols)</code>
<div class="block">Returns a new <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> sorted by the specified column, all in ascending order.</div>
</td>
</tr>
<tr class="altColor">
<td class="colFirst"><code><a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#sort(java.lang.String, java.lang.String...)">sort</a></strong>(java.lang.String sortCol,
java.lang.String... sortCols)</code>
<div class="block">Returns a new <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> sorted by the specified column, all in ascending order.</div>
</td>
</tr>
<tr class="rowColor">
<td class="colFirst"><code><a href="../../../../org/apache/spark/sql/SQLContext.html" title="class in org.apache.spark.sql">SQLContext</a></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#sqlContext()">sqlContext</a></strong>()</code> </td>
</tr>
<tr class="altColor">
<td class="colFirst"><code><a href="../../../../org/apache/spark/sql/DataFrameStatFunctions.html" title="class in org.apache.spark.sql">DataFrameStatFunctions</a></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#stat()">stat</a></strong>()</code>
<div class="block">Returns a <a href="../../../../org/apache/spark/sql/DataFrameStatFunctions.html" title="class in org.apache.spark.sql"><code>DataFrameStatFunctions</code></a> for working statistic functions support.</div>
</td>
</tr>
<tr class="rowColor">
<td class="colFirst"><code><a href="../../../../org/apache/spark/sql/Row.html" title="interface in org.apache.spark.sql">Row</a>[]</code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#take(int)">take</a></strong>(int n)</code>
<div class="block">Returns the first <code>n</code> rows in the <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a>.</div>
</td>
</tr>
<tr class="altColor">
<td class="colFirst"><code><a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#toDF()">toDF</a></strong>()</code>
<div class="block">Returns the object itself.</div>
</td>
</tr>
<tr class="rowColor">
<td class="colFirst"><code><a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#toDF(scala.collection.Seq)">toDF</a></strong>(scala.collection.Seq<java.lang.String> colNames)</code>
<div class="block">Returns a new <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> with columns renamed.</div>
</td>
</tr>
<tr class="altColor">
<td class="colFirst"><code><a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#toDF(java.lang.String...)">toDF</a></strong>(java.lang.String... colNames)</code>
<div class="block">Returns a new <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> with columns renamed.</div>
</td>
</tr>
<tr class="rowColor">
<td class="colFirst"><code><a href="../../../../org/apache/spark/api/java/JavaRDD.html" title="class in org.apache.spark.api.java">JavaRDD</a><<a href="../../../../org/apache/spark/sql/Row.html" title="interface in org.apache.spark.sql">Row</a>></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#toJavaRDD()">toJavaRDD</a></strong>()</code>
<div class="block">Returns the content of the <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> as a <code>JavaRDD</code> of <a href="../../../../org/apache/spark/sql/Row.html" title="interface in org.apache.spark.sql"><code>Row</code></a>s.</div>
</td>
</tr>
<tr class="altColor">
<td class="colFirst"><code><a href="../../../../org/apache/spark/rdd/RDD.html" title="class in org.apache.spark.rdd">RDD</a><java.lang.String></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#toJSON()">toJSON</a></strong>()</code>
<div class="block">Returns the content of the <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> as a RDD of JSON strings.</div>
</td>
</tr>
<tr class="rowColor">
<td class="colFirst"><code><a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#toSchemaRDD()">toSchemaRDD</a></strong>()</code>
<div class="block"><strong>Deprecated.</strong>
<div class="block"><i>As of 1.3.0, replaced by <code>toDF()</code>.</i></div>
</div>
</td>
</tr>
<tr class="altColor">
<td class="colFirst"><code>java.lang.String</code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#toString()">toString</a></strong>()</code> </td>
</tr>
<tr class="rowColor">
<td class="colFirst"><code><a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#unionAll(org.apache.spark.sql.DataFrame)">unionAll</a></strong>(<a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a> other)</code>
<div class="block">Returns a new <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> containing union of rows in this frame and another frame.</div>
</td>
</tr>
<tr class="altColor">
<td class="colFirst"><code><a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#unpersist()">unpersist</a></strong>()</code> </td>
</tr>
<tr class="rowColor">
<td class="colFirst"><code><a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#unpersist(boolean)">unpersist</a></strong>(boolean blocking)</code> </td>
</tr>
<tr class="altColor">
<td class="colFirst"><code><a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#where(org.apache.spark.sql.Column)">where</a></strong>(<a href="../../../../org/apache/spark/sql/Column.html" title="class in org.apache.spark.sql">Column</a> condition)</code>
<div class="block">Filters rows using the given condition.</div>
</td>
</tr>
<tr class="rowColor">
<td class="colFirst"><code><a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#where(java.lang.String)">where</a></strong>(java.lang.String conditionExpr)</code>
<div class="block">Filters rows using the given SQL expression.</div>
</td>
</tr>
<tr class="altColor">
<td class="colFirst"><code><a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#withColumn(java.lang.String, org.apache.spark.sql.Column)">withColumn</a></strong>(java.lang.String colName,
<a href="../../../../org/apache/spark/sql/Column.html" title="class in org.apache.spark.sql">Column</a> col)</code>
<div class="block">Returns a new <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> by adding a column or replacing the existing column that has
the same name.</div>
</td>
</tr>
<tr class="rowColor">
<td class="colFirst"><code><a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#withColumnRenamed(java.lang.String, java.lang.String)">withColumnRenamed</a></strong>(java.lang.String existingName,
java.lang.String newName)</code>
<div class="block">Returns a new <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> with a column renamed.</div>
</td>
</tr>
<tr class="altColor">
<td class="colFirst"><code><a href="../../../../org/apache/spark/sql/DataFrameWriter.html" title="class in org.apache.spark.sql">DataFrameWriter</a></code></td>
<td class="colLast"><code><strong><a href="../../../../org/apache/spark/sql/DataFrame.html#write()">write</a></strong>()</code>
<div class="block">:: Experimental ::
Interface for saving the content of the <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> out into external storage.</div>
</td>
</tr>
</table>
<ul class="blockList">
<li class="blockList"><a name="methods_inherited_from_class_java.lang.Object">
<!-- -->
</a>
<h3>Methods inherited from class java.lang.Object</h3>
<code>clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait</code></li>
</ul>
</li>
</ul>
</li>
</ul>
</div>
<div class="details">
<ul class="blockList">
<li class="blockList">
<!-- ========= CONSTRUCTOR DETAIL ======== -->
<ul class="blockList">
<li class="blockList"><a name="constructor_detail">
<!-- -->
</a>
<h3>Constructor Detail</h3>
<a name="DataFrame(org.apache.spark.sql.SQLContext, org.apache.spark.sql.catalyst.plans.logical.LogicalPlan)">
<!-- -->
</a>
<ul class="blockListLast">
<li class="blockList">
<h4>DataFrame</h4>
<pre>public DataFrame(<a href="../../../../org/apache/spark/sql/SQLContext.html" title="class in org.apache.spark.sql">SQLContext</a> sqlContext,
org.apache.spark.sql.catalyst.plans.logical.LogicalPlan logicalPlan)</pre>
<div class="block">A constructor that automatically analyzes the logical plan.
<p>
This reports error eagerly as the <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> is constructed, unless
<code>SQLConf.dataFrameEagerAnalysis</code> is turned off.</div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>sqlContext</code> - (undocumented)</dd><dd><code>logicalPlan</code> - (undocumented)</dd></dl>
</li>
</ul>
</li>
</ul>
<!-- ============ METHOD DETAIL ========== -->
<ul class="blockList">
<li class="blockList"><a name="method_detail">
<!-- -->
</a>
<h3>Method Detail</h3>
<a name="toDF(java.lang.String...)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>toDF</h4>
<pre>public <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a> toDF(java.lang.String... colNames)</pre>
<div class="block">Returns a new <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> with columns renamed. This can be quite convenient in conversion
from a RDD of tuples into a <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> with meaningful names. For example:
<pre><code>
val rdd: RDD[(Int, String)] = ...
rdd.toDF() // this implicit conversion creates a DataFrame with column name _1 and _2
rdd.toDF("id", "name") // this creates a DataFrame with column name "id" and "name"
</code></pre></div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>colNames</code> - (undocumented)</dd>
<dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.3.0</dd></dl>
</li>
</ul>
<a name="sort(java.lang.String, java.lang.String...)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>sort</h4>
<pre>public <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a> sort(java.lang.String sortCol,
java.lang.String... sortCols)</pre>
<div class="block">Returns a new <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> sorted by the specified column, all in ascending order.
<pre><code>
// The following 3 are equivalent
df.sort("sortcol")
df.sort($"sortcol")
df.sort($"sortcol".asc)
</code></pre></div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>sortCol</code> - (undocumented)</dd><dd><code>sortCols</code> - (undocumented)</dd>
<dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.3.0</dd></dl>
</li>
</ul>
<a name="sort(org.apache.spark.sql.Column...)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>sort</h4>
<pre>public <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a> sort(<a href="../../../../org/apache/spark/sql/Column.html" title="class in org.apache.spark.sql">Column</a>... sortExprs)</pre>
<div class="block">Returns a new <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> sorted by the given expressions. For example:
<pre><code>
df.sort($"col1", $"col2".desc)
</code></pre></div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>sortExprs</code> - (undocumented)</dd>
<dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.3.0</dd></dl>
</li>
</ul>
<a name="orderBy(java.lang.String, java.lang.String...)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>orderBy</h4>
<pre>public <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a> orderBy(java.lang.String sortCol,
java.lang.String... sortCols)</pre>
<div class="block">Returns a new <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> sorted by the given expressions.
This is an alias of the <code>sort</code> function.</div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>sortCol</code> - (undocumented)</dd><dd><code>sortCols</code> - (undocumented)</dd>
<dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.3.0</dd></dl>
</li>
</ul>
<a name="orderBy(org.apache.spark.sql.Column...)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>orderBy</h4>
<pre>public <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a> orderBy(<a href="../../../../org/apache/spark/sql/Column.html" title="class in org.apache.spark.sql">Column</a>... sortExprs)</pre>
<div class="block">Returns a new <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> sorted by the given expressions.
This is an alias of the <code>sort</code> function.</div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>sortExprs</code> - (undocumented)</dd>
<dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.3.0</dd></dl>
</li>
</ul>
<a name="select(org.apache.spark.sql.Column...)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>select</h4>
<pre>public <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a> select(<a href="../../../../org/apache/spark/sql/Column.html" title="class in org.apache.spark.sql">Column</a>... cols)</pre>
<div class="block">Selects a set of column based expressions.
<pre><code>
df.select($"colA", $"colB" + 1)
</code></pre></div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>cols</code> - (undocumented)</dd>
<dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.3.0</dd></dl>
</li>
</ul>
<a name="select(java.lang.String, java.lang.String...)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>select</h4>
<pre>public <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a> select(java.lang.String col,
java.lang.String... cols)</pre>
<div class="block">Selects a set of columns. This is a variant of <code>select</code> that can only select
existing columns using column names (i.e. cannot construct expressions).
<p>
<pre><code>
// The following two are equivalent:
df.select("colA", "colB")
df.select($"colA", $"colB")
</code></pre></div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>col</code> - (undocumented)</dd><dd><code>cols</code> - (undocumented)</dd>
<dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.3.0</dd></dl>
</li>
</ul>
<a name="selectExpr(java.lang.String...)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>selectExpr</h4>
<pre>public <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a> selectExpr(java.lang.String... exprs)</pre>
<div class="block">Selects a set of SQL expressions. This is a variant of <code>select</code> that accepts
SQL expressions.
<p>
<pre><code>
df.selectExpr("colA", "colB as newName", "abs(colC)")
</code></pre></div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>exprs</code> - (undocumented)</dd>
<dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.3.0</dd></dl>
</li>
</ul>
<a name="groupBy(org.apache.spark.sql.Column...)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>groupBy</h4>
<pre>public <a href="../../../../org/apache/spark/sql/GroupedData.html" title="class in org.apache.spark.sql">GroupedData</a> groupBy(<a href="../../../../org/apache/spark/sql/Column.html" title="class in org.apache.spark.sql">Column</a>... cols)</pre>
<div class="block">Groups the <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> using the specified columns, so we can run aggregation on them.
See <a href="../../../../org/apache/spark/sql/GroupedData.html" title="class in org.apache.spark.sql"><code>GroupedData</code></a> for all the available aggregate functions.
<p>
<pre><code>
// Compute the average for all numeric columns grouped by department.
df.groupBy($"department").avg()
// Compute the max age and average salary, grouped by department and gender.
df.groupBy($"department", $"gender").agg(Map(
"salary" -> "avg",
"age" -> "max"
))
</code></pre></div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>cols</code> - (undocumented)</dd>
<dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.3.0</dd></dl>
</li>
</ul>
<a name="rollup(org.apache.spark.sql.Column...)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>rollup</h4>
<pre>public <a href="../../../../org/apache/spark/sql/GroupedData.html" title="class in org.apache.spark.sql">GroupedData</a> rollup(<a href="../../../../org/apache/spark/sql/Column.html" title="class in org.apache.spark.sql">Column</a>... cols)</pre>
<div class="block">Create a multi-dimensional rollup for the current <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> using the specified columns,
so we can run aggregation on them.
See <a href="../../../../org/apache/spark/sql/GroupedData.html" title="class in org.apache.spark.sql"><code>GroupedData</code></a> for all the available aggregate functions.
<p>
<pre><code>
// Compute the average for all numeric columns rolluped by department and group.
df.rollup($"department", $"group").avg()
// Compute the max age and average salary, rolluped by department and gender.
df.rollup($"department", $"gender").agg(Map(
"salary" -> "avg",
"age" -> "max"
))
</code></pre></div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>cols</code> - (undocumented)</dd>
<dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.4.0</dd></dl>
</li>
</ul>
<a name="cube(org.apache.spark.sql.Column...)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>cube</h4>
<pre>public <a href="../../../../org/apache/spark/sql/GroupedData.html" title="class in org.apache.spark.sql">GroupedData</a> cube(<a href="../../../../org/apache/spark/sql/Column.html" title="class in org.apache.spark.sql">Column</a>... cols)</pre>
<div class="block">Create a multi-dimensional cube for the current <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> using the specified columns,
so we can run aggregation on them.
See <a href="../../../../org/apache/spark/sql/GroupedData.html" title="class in org.apache.spark.sql"><code>GroupedData</code></a> for all the available aggregate functions.
<p>
<pre><code>
// Compute the average for all numeric columns cubed by department and group.
df.cube($"department", $"group").avg()
// Compute the max age and average salary, cubed by department and gender.
df.cube($"department", $"gender").agg(Map(
"salary" -> "avg",
"age" -> "max"
))
</code></pre></div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>cols</code> - (undocumented)</dd>
<dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.4.0</dd></dl>
</li>
</ul>
<a name="groupBy(java.lang.String, java.lang.String...)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>groupBy</h4>
<pre>public <a href="../../../../org/apache/spark/sql/GroupedData.html" title="class in org.apache.spark.sql">GroupedData</a> groupBy(java.lang.String col1,
java.lang.String... cols)</pre>
<div class="block">Groups the <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> using the specified columns, so we can run aggregation on them.
See <a href="../../../../org/apache/spark/sql/GroupedData.html" title="class in org.apache.spark.sql"><code>GroupedData</code></a> for all the available aggregate functions.
<p>
This is a variant of groupBy that can only group by existing columns using column names
(i.e. cannot construct expressions).
<p>
<pre><code>
// Compute the average for all numeric columns grouped by department.
df.groupBy("department").avg()
// Compute the max age and average salary, grouped by department and gender.
df.groupBy($"department", $"gender").agg(Map(
"salary" -> "avg",
"age" -> "max"
))
</code></pre></div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>col1</code> - (undocumented)</dd><dd><code>cols</code> - (undocumented)</dd>
<dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.3.0</dd></dl>
</li>
</ul>
<a name="rollup(java.lang.String, java.lang.String...)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>rollup</h4>
<pre>public <a href="../../../../org/apache/spark/sql/GroupedData.html" title="class in org.apache.spark.sql">GroupedData</a> rollup(java.lang.String col1,
java.lang.String... cols)</pre>
<div class="block">Create a multi-dimensional rollup for the current <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> using the specified columns,
so we can run aggregation on them.
See <a href="../../../../org/apache/spark/sql/GroupedData.html" title="class in org.apache.spark.sql"><code>GroupedData</code></a> for all the available aggregate functions.
<p>
This is a variant of rollup that can only group by existing columns using column names
(i.e. cannot construct expressions).
<p>
<pre><code>
// Compute the average for all numeric columns rolluped by department and group.
df.rollup("department", "group").avg()
// Compute the max age and average salary, rolluped by department and gender.
df.rollup($"department", $"gender").agg(Map(
"salary" -> "avg",
"age" -> "max"
))
</code></pre></div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>col1</code> - (undocumented)</dd><dd><code>cols</code> - (undocumented)</dd>
<dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.4.0</dd></dl>
</li>
</ul>
<a name="cube(java.lang.String, java.lang.String...)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>cube</h4>
<pre>public <a href="../../../../org/apache/spark/sql/GroupedData.html" title="class in org.apache.spark.sql">GroupedData</a> cube(java.lang.String col1,
java.lang.String... cols)</pre>
<div class="block">Create a multi-dimensional cube for the current <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> using the specified columns,
so we can run aggregation on them.
See <a href="../../../../org/apache/spark/sql/GroupedData.html" title="class in org.apache.spark.sql"><code>GroupedData</code></a> for all the available aggregate functions.
<p>
This is a variant of cube that can only group by existing columns using column names
(i.e. cannot construct expressions).
<p>
<pre><code>
// Compute the average for all numeric columns cubed by department and group.
df.cube("department", "group").avg()
// Compute the max age and average salary, cubed by department and gender.
df.cube($"department", $"gender").agg(Map(
"salary" -> "avg",
"age" -> "max"
))
</code></pre></div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>col1</code> - (undocumented)</dd><dd><code>cols</code> - (undocumented)</dd>
<dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.4.0</dd></dl>
</li>
</ul>
<a name="agg(org.apache.spark.sql.Column, org.apache.spark.sql.Column...)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>agg</h4>
<pre>public <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a> agg(<a href="../../../../org/apache/spark/sql/Column.html" title="class in org.apache.spark.sql">Column</a> expr,
<a href="../../../../org/apache/spark/sql/Column.html" title="class in org.apache.spark.sql">Column</a>... exprs)</pre>
<div class="block">Aggregates on the entire <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> without groups.
<pre><code>
// df.agg(...) is a shorthand for df.groupBy().agg(...)
df.agg(max($"age"), avg($"salary"))
df.groupBy().agg(max($"age"), avg($"salary"))
</code></pre></div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>expr</code> - (undocumented)</dd><dd><code>exprs</code> - (undocumented)</dd>
<dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.3.0</dd></dl>
</li>
</ul>
<a name="describe(java.lang.String...)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>describe</h4>
<pre>public <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a> describe(java.lang.String... cols)</pre>
<div class="block">Computes statistics for numeric columns, including count, mean, stddev, min, and max.
If no columns are given, this function computes statistics for all numerical columns.
<p>
This function is meant for exploratory data analysis, as we make no guarantee about the
backward compatibility of the schema of the resulting <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a>. If you want to
programmatically compute summary statistics, use the <code>agg</code> function instead.
<p>
<pre><code>
df.describe("age", "height").show()
// output:
// summary age height
// count 10.0 10.0
// mean 53.3 178.05
// stddev 11.6 15.7
// min 18.0 163.0
// max 92.0 192.0
</code></pre>
<p></div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>cols</code> - (undocumented)</dd>
<dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.3.1</dd></dl>
</li>
</ul>
<a name="sqlContext()">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>sqlContext</h4>
<pre>public <a href="../../../../org/apache/spark/sql/SQLContext.html" title="class in org.apache.spark.sql">SQLContext</a> sqlContext()</pre>
</li>
</ul>
<a name="queryExecution()">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>queryExecution</h4>
<pre>public <a href="../../../../org/apache/spark/sql/SQLContext.QueryExecution.html" title="class in org.apache.spark.sql">SQLContext.QueryExecution</a> queryExecution()</pre>
</li>
</ul>
<a name="logicalPlan()">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>logicalPlan</h4>
<pre>protected org.apache.spark.sql.catalyst.plans.logical.LogicalPlan logicalPlan()</pre>
</li>
</ul>
<a name="resolve(java.lang.String)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>resolve</h4>
<pre>protected org.apache.spark.sql.catalyst.expressions.NamedExpression resolve(java.lang.String colName)</pre>
</li>
</ul>
<a name="numericColumns()">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>numericColumns</h4>
<pre>protected scala.collection.Seq<org.apache.spark.sql.catalyst.expressions.Expression> numericColumns()</pre>
</li>
</ul>
<a name="toString()">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>toString</h4>
<pre>public java.lang.String toString()</pre>
<dl>
<dt><strong>Overrides:</strong></dt>
<dd><code>toString</code> in class <code>java.lang.Object</code></dd>
</dl>
</li>
</ul>
<a name="toDF()">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>toDF</h4>
<pre>public <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a> toDF()</pre>
<div class="block">Returns the object itself.</div>
<dl><dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.3.0</dd></dl>
</li>
</ul>
<a name="toDF(scala.collection.Seq)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>toDF</h4>
<pre>public <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a> toDF(scala.collection.Seq<java.lang.String> colNames)</pre>
<div class="block">Returns a new <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> with columns renamed. This can be quite convenient in conversion
from a RDD of tuples into a <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> with meaningful names. For example:
<pre><code>
val rdd: RDD[(Int, String)] = ...
rdd.toDF() // this implicit conversion creates a DataFrame with column name _1 and _2
rdd.toDF("id", "name") // this creates a DataFrame with column name "id" and "name"
</code></pre></div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>colNames</code> - (undocumented)</dd>
<dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.3.0</dd></dl>
</li>
</ul>
<a name="schema()">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>schema</h4>
<pre>public <a href="../../../../org/apache/spark/sql/types/StructType.html" title="class in org.apache.spark.sql.types">StructType</a> schema()</pre>
<div class="block">Returns the schema of this <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a>.</div>
<dl><dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.3.0</dd></dl>
</li>
</ul>
<a name="dtypes()">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>dtypes</h4>
<pre>public scala.Tuple2<java.lang.String,java.lang.String>[] dtypes()</pre>
<div class="block">Returns all column names and their data types as an array.</div>
<dl><dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.3.0</dd></dl>
</li>
</ul>
<a name="columns()">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>columns</h4>
<pre>public java.lang.String[] columns()</pre>
<div class="block">Returns all column names as an array.</div>
<dl><dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.3.0</dd></dl>
</li>
</ul>
<a name="printSchema()">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>printSchema</h4>
<pre>public void printSchema()</pre>
<div class="block">Prints the schema to the console in a nice tree format.</div>
<dl><dt><span class="strong">Since:</span></dt>
<dd>1.3.0</dd></dl>
</li>
</ul>
<a name="explain(boolean)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>explain</h4>
<pre>public void explain(boolean extended)</pre>
<div class="block">Prints the plans (logical and physical) to the console for debugging purposes.</div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>extended</code> - (undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.3.0</dd></dl>
</li>
</ul>
<a name="explain()">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>explain</h4>
<pre>public void explain()</pre>
<div class="block">Only prints the physical plan to the console for debugging purposes.</div>
<dl><dt><span class="strong">Since:</span></dt>
<dd>1.3.0</dd></dl>
</li>
</ul>
<a name="isLocal()">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>isLocal</h4>
<pre>public boolean isLocal()</pre>
<div class="block">Returns true if the <code>collect</code> and <code>take</code> methods can be run locally
(without any Spark executors).</div>
<dl><dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.3.0</dd></dl>
</li>
</ul>
<a name="show(int)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>show</h4>
<pre>public void show(int numRows)</pre>
<div class="block">Displays the <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> in a tabular form. Strings more than 20 characters will be
truncated, and all cells will be aligned right. For example:
<pre><code>
year month AVG('Adj Close) MAX('Adj Close)
1980 12 0.503218 0.595103
1981 01 0.523289 0.570307
1982 02 0.436504 0.475256
1983 03 0.410516 0.442194
1984 04 0.450090 0.483521
</code></pre></div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>numRows</code> - Number of rows to show
<p></dd><dt><span class="strong">Since:</span></dt>
<dd>1.3.0</dd></dl>
</li>
</ul>
<a name="show()">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>show</h4>
<pre>public void show()</pre>
<div class="block">Displays the top 20 rows of <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> in a tabular form. Strings more than 20 characters
will be truncated, and all cells will be aligned right.</div>
<dl><dt><span class="strong">Since:</span></dt>
<dd>1.3.0</dd></dl>
</li>
</ul>
<a name="show(boolean)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>show</h4>
<pre>public void show(boolean truncate)</pre>
<div class="block">Displays the top 20 rows of <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> in a tabular form.
<p></div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>truncate</code> - Whether truncate long strings. If true, strings more than 20 characters will
be truncated and all cells will be aligned right
<p></dd><dt><span class="strong">Since:</span></dt>
<dd>1.5.0</dd></dl>
</li>
</ul>
<a name="show(int, boolean)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>show</h4>
<pre>public void show(int numRows,
boolean truncate)</pre>
<div class="block">Displays the <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> in a tabular form. For example:
<pre><code>
year month AVG('Adj Close) MAX('Adj Close)
1980 12 0.503218 0.595103
1981 01 0.523289 0.570307
1982 02 0.436504 0.475256
1983 03 0.410516 0.442194
1984 04 0.450090 0.483521
</code></pre></div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>numRows</code> - Number of rows to show</dd><dd><code>truncate</code> - Whether truncate long strings. If true, strings more than 20 characters will
be truncated and all cells will be aligned right
<p></dd><dt><span class="strong">Since:</span></dt>
<dd>1.5.0</dd></dl>
</li>
</ul>
<a name="na()">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>na</h4>
<pre>public <a href="../../../../org/apache/spark/sql/DataFrameNaFunctions.html" title="class in org.apache.spark.sql">DataFrameNaFunctions</a> na()</pre>
<div class="block">Returns a <a href="../../../../org/apache/spark/sql/DataFrameNaFunctions.html" title="class in org.apache.spark.sql"><code>DataFrameNaFunctions</code></a> for working with missing data.
<pre><code>
// Dropping rows containing any null values.
df.na.drop()
</code></pre>
<p></div>
<dl><dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.3.1</dd></dl>
</li>
</ul>
<a name="stat()">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>stat</h4>
<pre>public <a href="../../../../org/apache/spark/sql/DataFrameStatFunctions.html" title="class in org.apache.spark.sql">DataFrameStatFunctions</a> stat()</pre>
<div class="block">Returns a <a href="../../../../org/apache/spark/sql/DataFrameStatFunctions.html" title="class in org.apache.spark.sql"><code>DataFrameStatFunctions</code></a> for working statistic functions support.
<pre><code>
// Finding frequent items in column with name 'a'.
df.stat.freqItems(Seq("a"))
</code></pre>
<p></div>
<dl><dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.4.0</dd></dl>
</li>
</ul>
<a name="join(org.apache.spark.sql.DataFrame)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>join</h4>
<pre>public <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a> join(<a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a> right)</pre>
<div class="block">Cartesian join with another <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a>.
<p>
Note that cartesian joins are very expensive without an extra filter that can be pushed down.
<p></div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>right</code> - Right side of the join operation.</dd>
<dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.3.0</dd></dl>
</li>
</ul>
<a name="join(org.apache.spark.sql.DataFrame, java.lang.String)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>join</h4>
<pre>public <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a> join(<a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a> right,
java.lang.String usingColumn)</pre>
<div class="block">Inner equi-join with another <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> using the given column.
<p>
Different from other join functions, the join column will only appear once in the output,
i.e. similar to SQL's <code>JOIN USING</code> syntax.
<p>
<pre><code>
// Joining df1 and df2 using the column "user_id"
df1.join(df2, "user_id")
</code></pre>
<p>
Note that if you perform a self-join using this function without aliasing the input
<a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a>s, you will NOT be able to reference any columns after the join, since
there is no way to disambiguate which side of the join you would like to reference.
<p></div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>right</code> - Right side of the join operation.</dd><dd><code>usingColumn</code> - Name of the column to join on. This column must exist on both sides.</dd>
<dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.4.0</dd></dl>
</li>
</ul>
<a name="join(org.apache.spark.sql.DataFrame, scala.collection.Seq)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>join</h4>
<pre>public <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a> join(<a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a> right,
scala.collection.Seq<java.lang.String> usingColumns)</pre>
<div class="block">Inner equi-join with another <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> using the given columns.
<p>
Different from other join functions, the join columns will only appear once in the output,
i.e. similar to SQL's <code>JOIN USING</code> syntax.
<p>
<pre><code>
// Joining df1 and df2 using the columns "user_id" and "user_name"
df1.join(df2, Seq("user_id", "user_name"))
</code></pre>
<p>
Note that if you perform a self-join using this function without aliasing the input
<a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a>s, you will NOT be able to reference any columns after the join, since
there is no way to disambiguate which side of the join you would like to reference.
<p></div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>right</code> - Right side of the join operation.</dd><dd><code>usingColumns</code> - Names of the columns to join on. This columns must exist on both sides.</dd>
<dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.4.0</dd></dl>
</li>
</ul>
<a name="join(org.apache.spark.sql.DataFrame, org.apache.spark.sql.Column)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>join</h4>
<pre>public <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a> join(<a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a> right,
<a href="../../../../org/apache/spark/sql/Column.html" title="class in org.apache.spark.sql">Column</a> joinExprs)</pre>
<div class="block">Inner join with another <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a>, using the given join expression.
<p>
<pre><code>
// The following two are equivalent:
df1.join(df2, $"df1Key" === $"df2Key")
df1.join(df2).where($"df1Key" === $"df2Key")
</code></pre></div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>right</code> - (undocumented)</dd><dd><code>joinExprs</code> - (undocumented)</dd>
<dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.3.0</dd></dl>
</li>
</ul>
<a name="join(org.apache.spark.sql.DataFrame, org.apache.spark.sql.Column, java.lang.String)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>join</h4>
<pre>public <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a> join(<a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a> right,
<a href="../../../../org/apache/spark/sql/Column.html" title="class in org.apache.spark.sql">Column</a> joinExprs,
java.lang.String joinType)</pre>
<div class="block">Join with another <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a>, using the given join expression. The following performs
a full outer join between <code>df1</code> and <code>df2</code>.
<p>
<pre><code>
// Scala:
import org.apache.spark.sql.functions._
df1.join(df2, $"df1Key" === $"df2Key", "outer")
// Java:
import static org.apache.spark.sql.functions.*;
df1.join(df2, col("df1Key").equalTo(col("df2Key")), "outer");
</code></pre>
<p></div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>right</code> - Right side of the join.</dd><dd><code>joinExprs</code> - Join expression.</dd><dd><code>joinType</code> - One of: <code>inner</code>, <code>outer</code>, <code>left_outer</code>, <code>right_outer</code>, <code>leftsemi</code>.</dd>
<dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.3.0</dd></dl>
</li>
</ul>
<a name="sort(java.lang.String, scala.collection.Seq)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>sort</h4>
<pre>public <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a> sort(java.lang.String sortCol,
scala.collection.Seq<java.lang.String> sortCols)</pre>
<div class="block">Returns a new <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> sorted by the specified column, all in ascending order.
<pre><code>
// The following 3 are equivalent
df.sort("sortcol")
df.sort($"sortcol")
df.sort($"sortcol".asc)
</code></pre></div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>sortCol</code> - (undocumented)</dd><dd><code>sortCols</code> - (undocumented)</dd>
<dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.3.0</dd></dl>
</li>
</ul>
<a name="sort(scala.collection.Seq)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>sort</h4>
<pre>public <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a> sort(scala.collection.Seq<<a href="../../../../org/apache/spark/sql/Column.html" title="class in org.apache.spark.sql">Column</a>> sortExprs)</pre>
<div class="block">Returns a new <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> sorted by the given expressions. For example:
<pre><code>
df.sort($"col1", $"col2".desc)
</code></pre></div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>sortExprs</code> - (undocumented)</dd>
<dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.3.0</dd></dl>
</li>
</ul>
<a name="orderBy(java.lang.String, scala.collection.Seq)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>orderBy</h4>
<pre>public <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a> orderBy(java.lang.String sortCol,
scala.collection.Seq<java.lang.String> sortCols)</pre>
<div class="block">Returns a new <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> sorted by the given expressions.
This is an alias of the <code>sort</code> function.</div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>sortCol</code> - (undocumented)</dd><dd><code>sortCols</code> - (undocumented)</dd>
<dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.3.0</dd></dl>
</li>
</ul>
<a name="orderBy(scala.collection.Seq)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>orderBy</h4>
<pre>public <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a> orderBy(scala.collection.Seq<<a href="../../../../org/apache/spark/sql/Column.html" title="class in org.apache.spark.sql">Column</a>> sortExprs)</pre>
<div class="block">Returns a new <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> sorted by the given expressions.
This is an alias of the <code>sort</code> function.</div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>sortExprs</code> - (undocumented)</dd>
<dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.3.0</dd></dl>
</li>
</ul>
<a name="apply(java.lang.String)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>apply</h4>
<pre>public <a href="../../../../org/apache/spark/sql/Column.html" title="class in org.apache.spark.sql">Column</a> apply(java.lang.String colName)</pre>
<div class="block">Selects column based on the column name and return it as a <a href="../../../../org/apache/spark/sql/Column.html" title="class in org.apache.spark.sql"><code>Column</code></a>.
Note that the column name can also reference to a nested column like <code>a.b</code>.</div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>colName</code> - (undocumented)</dd>
<dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.3.0</dd></dl>
</li>
</ul>
<a name="col(java.lang.String)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>col</h4>
<pre>public <a href="../../../../org/apache/spark/sql/Column.html" title="class in org.apache.spark.sql">Column</a> col(java.lang.String colName)</pre>
<div class="block">Selects column based on the column name and return it as a <a href="../../../../org/apache/spark/sql/Column.html" title="class in org.apache.spark.sql"><code>Column</code></a>.
Note that the column name can also reference to a nested column like <code>a.b</code>.</div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>colName</code> - (undocumented)</dd>
<dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.3.0</dd></dl>
</li>
</ul>
<a name="as(java.lang.String)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>as</h4>
<pre>public <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a> as(java.lang.String alias)</pre>
<div class="block">Returns a new <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> with an alias set.</div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>alias</code> - (undocumented)</dd>
<dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.3.0</dd></dl>
</li>
</ul>
<a name="as(scala.Symbol)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>as</h4>
<pre>public <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a> as(scala.Symbol alias)</pre>
<div class="block">(Scala-specific) Returns a new <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> with an alias set.</div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>alias</code> - (undocumented)</dd>
<dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.3.0</dd></dl>
</li>
</ul>
<a name="select(scala.collection.Seq)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>select</h4>
<pre>public <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a> select(scala.collection.Seq<<a href="../../../../org/apache/spark/sql/Column.html" title="class in org.apache.spark.sql">Column</a>> cols)</pre>
<div class="block">Selects a set of column based expressions.
<pre><code>
df.select($"colA", $"colB" + 1)
</code></pre></div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>cols</code> - (undocumented)</dd>
<dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.3.0</dd></dl>
</li>
</ul>
<a name="select(java.lang.String, scala.collection.Seq)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>select</h4>
<pre>public <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a> select(java.lang.String col,
scala.collection.Seq<java.lang.String> cols)</pre>
<div class="block">Selects a set of columns. This is a variant of <code>select</code> that can only select
existing columns using column names (i.e. cannot construct expressions).
<p>
<pre><code>
// The following two are equivalent:
df.select("colA", "colB")
df.select($"colA", $"colB")
</code></pre></div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>col</code> - (undocumented)</dd><dd><code>cols</code> - (undocumented)</dd>
<dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.3.0</dd></dl>
</li>
</ul>
<a name="selectExpr(scala.collection.Seq)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>selectExpr</h4>
<pre>public <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a> selectExpr(scala.collection.Seq<java.lang.String> exprs)</pre>
<div class="block">Selects a set of SQL expressions. This is a variant of <code>select</code> that accepts
SQL expressions.
<p>
<pre><code>
df.selectExpr("colA", "colB as newName", "abs(colC)")
</code></pre></div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>exprs</code> - (undocumented)</dd>
<dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.3.0</dd></dl>
</li>
</ul>
<a name="filter(org.apache.spark.sql.Column)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>filter</h4>
<pre>public <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a> filter(<a href="../../../../org/apache/spark/sql/Column.html" title="class in org.apache.spark.sql">Column</a> condition)</pre>
<div class="block">Filters rows using the given condition.
<pre><code>
// The following are equivalent:
peopleDf.filter($"age" > 15)
peopleDf.where($"age" > 15)
</code></pre></div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>condition</code> - (undocumented)</dd>
<dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.3.0</dd></dl>
</li>
</ul>
<a name="filter(java.lang.String)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>filter</h4>
<pre>public <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a> filter(java.lang.String conditionExpr)</pre>
<div class="block">Filters rows using the given SQL expression.
<pre><code>
peopleDf.filter("age > 15")
</code></pre></div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>conditionExpr</code> - (undocumented)</dd>
<dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.3.0</dd></dl>
</li>
</ul>
<a name="where(org.apache.spark.sql.Column)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>where</h4>
<pre>public <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a> where(<a href="../../../../org/apache/spark/sql/Column.html" title="class in org.apache.spark.sql">Column</a> condition)</pre>
<div class="block">Filters rows using the given condition. This is an alias for <code>filter</code>.
<pre><code>
// The following are equivalent:
peopleDf.filter($"age" > 15)
peopleDf.where($"age" > 15)
</code></pre></div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>condition</code> - (undocumented)</dd>
<dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.3.0</dd></dl>
</li>
</ul>
<a name="where(java.lang.String)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>where</h4>
<pre>public <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a> where(java.lang.String conditionExpr)</pre>
<div class="block">Filters rows using the given SQL expression.
<pre><code>
peopleDf.where("age > 15")
</code></pre></div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>conditionExpr</code> - (undocumented)</dd>
<dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.5.0</dd></dl>
</li>
</ul>
<a name="groupBy(scala.collection.Seq)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>groupBy</h4>
<pre>public <a href="../../../../org/apache/spark/sql/GroupedData.html" title="class in org.apache.spark.sql">GroupedData</a> groupBy(scala.collection.Seq<<a href="../../../../org/apache/spark/sql/Column.html" title="class in org.apache.spark.sql">Column</a>> cols)</pre>
<div class="block">Groups the <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> using the specified columns, so we can run aggregation on them.
See <a href="../../../../org/apache/spark/sql/GroupedData.html" title="class in org.apache.spark.sql"><code>GroupedData</code></a> for all the available aggregate functions.
<p>
<pre><code>
// Compute the average for all numeric columns grouped by department.
df.groupBy($"department").avg()
// Compute the max age and average salary, grouped by department and gender.
df.groupBy($"department", $"gender").agg(Map(
"salary" -> "avg",
"age" -> "max"
))
</code></pre></div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>cols</code> - (undocumented)</dd>
<dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.3.0</dd></dl>
</li>
</ul>
<a name="rollup(scala.collection.Seq)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>rollup</h4>
<pre>public <a href="../../../../org/apache/spark/sql/GroupedData.html" title="class in org.apache.spark.sql">GroupedData</a> rollup(scala.collection.Seq<<a href="../../../../org/apache/spark/sql/Column.html" title="class in org.apache.spark.sql">Column</a>> cols)</pre>
<div class="block">Create a multi-dimensional rollup for the current <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> using the specified columns,
so we can run aggregation on them.
See <a href="../../../../org/apache/spark/sql/GroupedData.html" title="class in org.apache.spark.sql"><code>GroupedData</code></a> for all the available aggregate functions.
<p>
<pre><code>
// Compute the average for all numeric columns rolluped by department and group.
df.rollup($"department", $"group").avg()
// Compute the max age and average salary, rolluped by department and gender.
df.rollup($"department", $"gender").agg(Map(
"salary" -> "avg",
"age" -> "max"
))
</code></pre></div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>cols</code> - (undocumented)</dd>
<dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.4.0</dd></dl>
</li>
</ul>
<a name="cube(scala.collection.Seq)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>cube</h4>
<pre>public <a href="../../../../org/apache/spark/sql/GroupedData.html" title="class in org.apache.spark.sql">GroupedData</a> cube(scala.collection.Seq<<a href="../../../../org/apache/spark/sql/Column.html" title="class in org.apache.spark.sql">Column</a>> cols)</pre>
<div class="block">Create a multi-dimensional cube for the current <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> using the specified columns,
so we can run aggregation on them.
See <a href="../../../../org/apache/spark/sql/GroupedData.html" title="class in org.apache.spark.sql"><code>GroupedData</code></a> for all the available aggregate functions.
<p>
<pre><code>
// Compute the average for all numeric columns cubed by department and group.
df.cube($"department", $"group").avg()
// Compute the max age and average salary, cubed by department and gender.
df.cube($"department", $"gender").agg(Map(
"salary" -> "avg",
"age" -> "max"
))
</code></pre></div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>cols</code> - (undocumented)</dd>
<dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.4.0</dd></dl>
</li>
</ul>
<a name="groupBy(java.lang.String, scala.collection.Seq)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>groupBy</h4>
<pre>public <a href="../../../../org/apache/spark/sql/GroupedData.html" title="class in org.apache.spark.sql">GroupedData</a> groupBy(java.lang.String col1,
scala.collection.Seq<java.lang.String> cols)</pre>
<div class="block">Groups the <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> using the specified columns, so we can run aggregation on them.
See <a href="../../../../org/apache/spark/sql/GroupedData.html" title="class in org.apache.spark.sql"><code>GroupedData</code></a> for all the available aggregate functions.
<p>
This is a variant of groupBy that can only group by existing columns using column names
(i.e. cannot construct expressions).
<p>
<pre><code>
// Compute the average for all numeric columns grouped by department.
df.groupBy("department").avg()
// Compute the max age and average salary, grouped by department and gender.
df.groupBy($"department", $"gender").agg(Map(
"salary" -> "avg",
"age" -> "max"
))
</code></pre></div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>col1</code> - (undocumented)</dd><dd><code>cols</code> - (undocumented)</dd>
<dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.3.0</dd></dl>
</li>
</ul>
<a name="rollup(java.lang.String, scala.collection.Seq)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>rollup</h4>
<pre>public <a href="../../../../org/apache/spark/sql/GroupedData.html" title="class in org.apache.spark.sql">GroupedData</a> rollup(java.lang.String col1,
scala.collection.Seq<java.lang.String> cols)</pre>
<div class="block">Create a multi-dimensional rollup for the current <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> using the specified columns,
so we can run aggregation on them.
See <a href="../../../../org/apache/spark/sql/GroupedData.html" title="class in org.apache.spark.sql"><code>GroupedData</code></a> for all the available aggregate functions.
<p>
This is a variant of rollup that can only group by existing columns using column names
(i.e. cannot construct expressions).
<p>
<pre><code>
// Compute the average for all numeric columns rolluped by department and group.
df.rollup("department", "group").avg()
// Compute the max age and average salary, rolluped by department and gender.
df.rollup($"department", $"gender").agg(Map(
"salary" -> "avg",
"age" -> "max"
))
</code></pre></div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>col1</code> - (undocumented)</dd><dd><code>cols</code> - (undocumented)</dd>
<dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.4.0</dd></dl>
</li>
</ul>
<a name="cube(java.lang.String, scala.collection.Seq)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>cube</h4>
<pre>public <a href="../../../../org/apache/spark/sql/GroupedData.html" title="class in org.apache.spark.sql">GroupedData</a> cube(java.lang.String col1,
scala.collection.Seq<java.lang.String> cols)</pre>
<div class="block">Create a multi-dimensional cube for the current <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> using the specified columns,
so we can run aggregation on them.
See <a href="../../../../org/apache/spark/sql/GroupedData.html" title="class in org.apache.spark.sql"><code>GroupedData</code></a> for all the available aggregate functions.
<p>
This is a variant of cube that can only group by existing columns using column names
(i.e. cannot construct expressions).
<p>
<pre><code>
// Compute the average for all numeric columns cubed by department and group.
df.cube("department", "group").avg()
// Compute the max age and average salary, cubed by department and gender.
df.cube($"department", $"gender").agg(Map(
"salary" -> "avg",
"age" -> "max"
))
</code></pre></div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>col1</code> - (undocumented)</dd><dd><code>cols</code> - (undocumented)</dd>
<dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.4.0</dd></dl>
</li>
</ul>
<a name="agg(scala.Tuple2, scala.collection.Seq)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>agg</h4>
<pre>public <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a> agg(scala.Tuple2<java.lang.String,java.lang.String> aggExpr,
scala.collection.Seq<scala.Tuple2<java.lang.String,java.lang.String>> aggExprs)</pre>
<div class="block">(Scala-specific) Aggregates on the entire <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> without groups.
<pre><code>
// df.agg(...) is a shorthand for df.groupBy().agg(...)
df.agg("age" -> "max", "salary" -> "avg")
df.groupBy().agg("age" -> "max", "salary" -> "avg")
</code></pre></div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>aggExpr</code> - (undocumented)</dd><dd><code>aggExprs</code> - (undocumented)</dd>
<dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.3.0</dd></dl>
</li>
</ul>
<a name="agg(scala.collection.immutable.Map)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>agg</h4>
<pre>public <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a> agg(scala.collection.immutable.Map<java.lang.String,java.lang.String> exprs)</pre>
<div class="block">(Scala-specific) Aggregates on the entire <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> without groups.
<pre><code>
// df.agg(...) is a shorthand for df.groupBy().agg(...)
df.agg(Map("age" -> "max", "salary" -> "avg"))
df.groupBy().agg(Map("age" -> "max", "salary" -> "avg"))
</code></pre></div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>exprs</code> - (undocumented)</dd>
<dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.3.0</dd></dl>
</li>
</ul>
<a name="agg(java.util.Map)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>agg</h4>
<pre>public <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a> agg(java.util.Map<java.lang.String,java.lang.String> exprs)</pre>
<div class="block">(Java-specific) Aggregates on the entire <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> without groups.
<pre><code>
// df.agg(...) is a shorthand for df.groupBy().agg(...)
df.agg(Map("age" -> "max", "salary" -> "avg"))
df.groupBy().agg(Map("age" -> "max", "salary" -> "avg"))
</code></pre></div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>exprs</code> - (undocumented)</dd>
<dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.3.0</dd></dl>
</li>
</ul>
<a name="agg(org.apache.spark.sql.Column, scala.collection.Seq)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>agg</h4>
<pre>public <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a> agg(<a href="../../../../org/apache/spark/sql/Column.html" title="class in org.apache.spark.sql">Column</a> expr,
scala.collection.Seq<<a href="../../../../org/apache/spark/sql/Column.html" title="class in org.apache.spark.sql">Column</a>> exprs)</pre>
<div class="block">Aggregates on the entire <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> without groups.
<pre><code>
// df.agg(...) is a shorthand for df.groupBy().agg(...)
df.agg(max($"age"), avg($"salary"))
df.groupBy().agg(max($"age"), avg($"salary"))
</code></pre></div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>expr</code> - (undocumented)</dd><dd><code>exprs</code> - (undocumented)</dd>
<dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.3.0</dd></dl>
</li>
</ul>
<a name="limit(int)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>limit</h4>
<pre>public <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a> limit(int n)</pre>
<div class="block">Returns a new <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> by taking the first <code>n</code> rows. The difference between this function
and <code>head</code> is that <code>head</code> returns an array while <code>limit</code> returns a new <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a>.</div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>n</code> - (undocumented)</dd>
<dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.3.0</dd></dl>
</li>
</ul>
<a name="unionAll(org.apache.spark.sql.DataFrame)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>unionAll</h4>
<pre>public <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a> unionAll(<a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a> other)</pre>
<div class="block">Returns a new <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> containing union of rows in this frame and another frame.
This is equivalent to <code>UNION ALL</code> in SQL.</div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>other</code> - (undocumented)</dd>
<dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.3.0</dd></dl>
</li>
</ul>
<a name="intersect(org.apache.spark.sql.DataFrame)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>intersect</h4>
<pre>public <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a> intersect(<a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a> other)</pre>
<div class="block">Returns a new <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> containing rows only in both this frame and another frame.
This is equivalent to <code>INTERSECT</code> in SQL.</div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>other</code> - (undocumented)</dd>
<dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.3.0</dd></dl>
</li>
</ul>
<a name="except(org.apache.spark.sql.DataFrame)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>except</h4>
<pre>public <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a> except(<a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a> other)</pre>
<div class="block">Returns a new <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> containing rows in this frame but not in another frame.
This is equivalent to <code>EXCEPT</code> in SQL.</div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>other</code> - (undocumented)</dd>
<dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.3.0</dd></dl>
</li>
</ul>
<a name="sample(boolean, double, long)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>sample</h4>
<pre>public <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a> sample(boolean withReplacement,
double fraction,
long seed)</pre>
<div class="block">Returns a new <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> by sampling a fraction of rows.
<p></div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>withReplacement</code> - Sample with replacement or not.</dd><dd><code>fraction</code> - Fraction of rows to generate.</dd><dd><code>seed</code> - Seed for sampling.</dd>
<dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.3.0</dd></dl>
</li>
</ul>
<a name="sample(boolean, double)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>sample</h4>
<pre>public <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a> sample(boolean withReplacement,
double fraction)</pre>
<div class="block">Returns a new <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> by sampling a fraction of rows, using a random seed.
<p></div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>withReplacement</code> - Sample with replacement or not.</dd><dd><code>fraction</code> - Fraction of rows to generate.</dd>
<dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.3.0</dd></dl>
</li>
</ul>
<a name="randomSplit(double[], long)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>randomSplit</h4>
<pre>public <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a>[] randomSplit(double[] weights,
long seed)</pre>
<div class="block">Randomly splits this <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> with the provided weights.
<p></div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>weights</code> - weights for splits, will be normalized if they don't sum to 1.</dd><dd><code>seed</code> - Seed for sampling.</dd>
<dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.4.0</dd></dl>
</li>
</ul>
<a name="randomSplit(double[])">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>randomSplit</h4>
<pre>public <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a>[] randomSplit(double[] weights)</pre>
<div class="block">Randomly splits this <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> with the provided weights.
<p></div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>weights</code> - weights for splits, will be normalized if they don't sum to 1.</dd>
<dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.4.0</dd></dl>
</li>
</ul>
<a name="explode(scala.collection.Seq, scala.Function1, scala.reflect.api.TypeTags.TypeTag)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>explode</h4>
<pre>public <A extends scala.Product> <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a> explode(scala.collection.Seq<<a href="../../../../org/apache/spark/sql/Column.html" title="class in org.apache.spark.sql">Column</a>> input,
scala.Function1<<a href="../../../../org/apache/spark/sql/Row.html" title="interface in org.apache.spark.sql">Row</a>,scala.collection.TraversableOnce<A>> f,
scala.reflect.api.TypeTags.TypeTag<A> evidence$1)</pre>
<div class="block">(Scala-specific) Returns a new <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> where each row has been expanded to zero or more
rows by the provided function. This is similar to a <code>LATERAL VIEW</code> in HiveQL. The columns of
the input row are implicitly joined with each row that is output by the function.
<p>
The following example uses this function to count the number of books which contain
a given word:
<p>
<pre><code>
case class Book(title: String, words: String)
val df: RDD[Book]
case class Word(word: String)
val allWords = df.explode('words) {
case Row(words: String) => words.split(" ").map(Word(_))
}
val bookCountPerWord = allWords.groupBy("word").agg(countDistinct("title"))
</code></pre></div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>input</code> - (undocumented)</dd><dd><code>f</code> - (undocumented)</dd><dd><code>evidence$1</code> - (undocumented)</dd>
<dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.3.0</dd></dl>
</li>
</ul>
<a name="explode(java.lang.String, java.lang.String, scala.Function1, scala.reflect.api.TypeTags.TypeTag)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>explode</h4>
<pre>public <A,B> <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a> explode(java.lang.String inputColumn,
java.lang.String outputColumn,
scala.Function1<A,scala.collection.TraversableOnce<B>> f,
scala.reflect.api.TypeTags.TypeTag<B> evidence$2)</pre>
<div class="block">(Scala-specific) Returns a new <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> where a single column has been expanded to zero
or more rows by the provided function. This is similar to a <code>LATERAL VIEW</code> in HiveQL. All
columns of the input row are implicitly joined with each value that is output by the function.
<p>
<pre><code>
df.explode("words", "word"){words: String => words.split(" ")}
</code></pre></div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>inputColumn</code> - (undocumented)</dd><dd><code>outputColumn</code> - (undocumented)</dd><dd><code>f</code> - (undocumented)</dd><dd><code>evidence$2</code> - (undocumented)</dd>
<dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.3.0</dd></dl>
</li>
</ul>
<a name="withColumn(java.lang.String, org.apache.spark.sql.Column)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>withColumn</h4>
<pre>public <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a> withColumn(java.lang.String colName,
<a href="../../../../org/apache/spark/sql/Column.html" title="class in org.apache.spark.sql">Column</a> col)</pre>
<div class="block">Returns a new <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> by adding a column or replacing the existing column that has
the same name.</div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>colName</code> - (undocumented)</dd><dd><code>col</code> - (undocumented)</dd>
<dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.3.0</dd></dl>
</li>
</ul>
<a name="withColumnRenamed(java.lang.String, java.lang.String)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>withColumnRenamed</h4>
<pre>public <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a> withColumnRenamed(java.lang.String existingName,
java.lang.String newName)</pre>
<div class="block">Returns a new <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> with a column renamed.
This is a no-op if schema doesn't contain existingName.</div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>existingName</code> - (undocumented)</dd><dd><code>newName</code> - (undocumented)</dd>
<dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.3.0</dd></dl>
</li>
</ul>
<a name="drop(java.lang.String)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>drop</h4>
<pre>public <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a> drop(java.lang.String colName)</pre>
<div class="block">Returns a new <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> with a column dropped.
This is a no-op if schema doesn't contain column name.</div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>colName</code> - (undocumented)</dd>
<dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.4.0</dd></dl>
</li>
</ul>
<a name="drop(org.apache.spark.sql.Column)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>drop</h4>
<pre>public <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a> drop(<a href="../../../../org/apache/spark/sql/Column.html" title="class in org.apache.spark.sql">Column</a> col)</pre>
<div class="block">Returns a new <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> with a column dropped.
This version of drop accepts a Column rather than a name.
This is a no-op if the DataFrame doesn't have a column
with an equivalent expression.</div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>col</code> - (undocumented)</dd>
<dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.4.1</dd></dl>
</li>
</ul>
<a name="dropDuplicates()">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>dropDuplicates</h4>
<pre>public <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a> dropDuplicates()</pre>
<div class="block">Returns a new <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> that contains only the unique rows from this <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a>.
This is an alias for <code>distinct</code>.</div>
<dl><dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.4.0</dd></dl>
</li>
</ul>
<a name="dropDuplicates(scala.collection.Seq)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>dropDuplicates</h4>
<pre>public <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a> dropDuplicates(scala.collection.Seq<java.lang.String> colNames)</pre>
<div class="block">(Scala-specific) Returns a new <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> with duplicate rows removed, considering only
the subset of columns.
<p></div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>colNames</code> - (undocumented)</dd>
<dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.4.0</dd></dl>
</li>
</ul>
<a name="dropDuplicates(java.lang.String[])">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>dropDuplicates</h4>
<pre>public <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a> dropDuplicates(java.lang.String[] colNames)</pre>
<div class="block">Returns a new <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> with duplicate rows removed, considering only
the subset of columns.
<p></div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>colNames</code> - (undocumented)</dd>
<dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.4.0</dd></dl>
</li>
</ul>
<a name="describe(scala.collection.Seq)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>describe</h4>
<pre>public <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a> describe(scala.collection.Seq<java.lang.String> cols)</pre>
<div class="block">Computes statistics for numeric columns, including count, mean, stddev, min, and max.
If no columns are given, this function computes statistics for all numerical columns.
<p>
This function is meant for exploratory data analysis, as we make no guarantee about the
backward compatibility of the schema of the resulting <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a>. If you want to
programmatically compute summary statistics, use the <code>agg</code> function instead.
<p>
<pre><code>
df.describe("age", "height").show()
// output:
// summary age height
// count 10.0 10.0
// mean 53.3 178.05
// stddev 11.6 15.7
// min 18.0 163.0
// max 92.0 192.0
</code></pre>
<p></div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>cols</code> - (undocumented)</dd>
<dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.3.1</dd></dl>
</li>
</ul>
<a name="head(int)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>head</h4>
<pre>public <a href="../../../../org/apache/spark/sql/Row.html" title="interface in org.apache.spark.sql">Row</a>[] head(int n)</pre>
<div class="block">Returns the first <code>n</code> rows.</div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>n</code> - (undocumented)</dd>
<dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.3.0</dd></dl>
</li>
</ul>
<a name="head()">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>head</h4>
<pre>public <a href="../../../../org/apache/spark/sql/Row.html" title="interface in org.apache.spark.sql">Row</a> head()</pre>
<div class="block">Returns the first row.</div>
<dl><dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.3.0</dd></dl>
</li>
</ul>
<a name="first()">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>first</h4>
<pre>public <a href="../../../../org/apache/spark/sql/Row.html" title="interface in org.apache.spark.sql">Row</a> first()</pre>
<div class="block">Returns the first row. Alias for head().</div>
<dl><dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.3.0</dd></dl>
</li>
</ul>
<a name="map(scala.Function1, scala.reflect.ClassTag)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>map</h4>
<pre>public <R> <a href="../../../../org/apache/spark/rdd/RDD.html" title="class in org.apache.spark.rdd">RDD</a><R> map(scala.Function1<<a href="../../../../org/apache/spark/sql/Row.html" title="interface in org.apache.spark.sql">Row</a>,R> f,
scala.reflect.ClassTag<R> evidence$3)</pre>
<div class="block">Returns a new RDD by applying a function to all rows of this DataFrame.</div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>f</code> - (undocumented)</dd><dd><code>evidence$3</code> - (undocumented)</dd>
<dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.3.0</dd></dl>
</li>
</ul>
<a name="flatMap(scala.Function1, scala.reflect.ClassTag)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>flatMap</h4>
<pre>public <R> <a href="../../../../org/apache/spark/rdd/RDD.html" title="class in org.apache.spark.rdd">RDD</a><R> flatMap(scala.Function1<<a href="../../../../org/apache/spark/sql/Row.html" title="interface in org.apache.spark.sql">Row</a>,scala.collection.TraversableOnce<R>> f,
scala.reflect.ClassTag<R> evidence$4)</pre>
<div class="block">Returns a new RDD by first applying a function to all rows of this <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a>,
and then flattening the results.</div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>f</code> - (undocumented)</dd><dd><code>evidence$4</code> - (undocumented)</dd>
<dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.3.0</dd></dl>
</li>
</ul>
<a name="mapPartitions(scala.Function1, scala.reflect.ClassTag)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>mapPartitions</h4>
<pre>public <R> <a href="../../../../org/apache/spark/rdd/RDD.html" title="class in org.apache.spark.rdd">RDD</a><R> mapPartitions(scala.Function1<scala.collection.Iterator<<a href="../../../../org/apache/spark/sql/Row.html" title="interface in org.apache.spark.sql">Row</a>>,scala.collection.Iterator<R>> f,
scala.reflect.ClassTag<R> evidence$5)</pre>
<div class="block">Returns a new RDD by applying a function to each partition of this DataFrame.</div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>f</code> - (undocumented)</dd><dd><code>evidence$5</code> - (undocumented)</dd>
<dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.3.0</dd></dl>
</li>
</ul>
<a name="foreach(scala.Function1)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>foreach</h4>
<pre>public void foreach(scala.Function1<<a href="../../../../org/apache/spark/sql/Row.html" title="interface in org.apache.spark.sql">Row</a>,scala.runtime.BoxedUnit> f)</pre>
<div class="block">Applies a function <code>f</code> to all rows.</div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>f</code> - (undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.3.0</dd></dl>
</li>
</ul>
<a name="foreachPartition(scala.Function1)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>foreachPartition</h4>
<pre>public void foreachPartition(scala.Function1<scala.collection.Iterator<<a href="../../../../org/apache/spark/sql/Row.html" title="interface in org.apache.spark.sql">Row</a>>,scala.runtime.BoxedUnit> f)</pre>
<div class="block">Applies a function f to each partition of this <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a>.</div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>f</code> - (undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.3.0</dd></dl>
</li>
</ul>
<a name="take(int)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>take</h4>
<pre>public <a href="../../../../org/apache/spark/sql/Row.html" title="interface in org.apache.spark.sql">Row</a>[] take(int n)</pre>
<div class="block">Returns the first <code>n</code> rows in the <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a>.</div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>n</code> - (undocumented)</dd>
<dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.3.0</dd></dl>
</li>
</ul>
<a name="collect()">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>collect</h4>
<pre>public <a href="../../../../org/apache/spark/sql/Row.html" title="interface in org.apache.spark.sql">Row</a>[] collect()</pre>
<div class="block">Returns an array that contains all of <a href="../../../../org/apache/spark/sql/Row.html" title="interface in org.apache.spark.sql"><code>Row</code></a>s in this <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a>.</div>
<dl><dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.3.0</dd></dl>
</li>
</ul>
<a name="collectAsList()">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>collectAsList</h4>
<pre>public java.util.List<<a href="../../../../org/apache/spark/sql/Row.html" title="interface in org.apache.spark.sql">Row</a>> collectAsList()</pre>
<div class="block">Returns a Java list that contains all of <a href="../../../../org/apache/spark/sql/Row.html" title="interface in org.apache.spark.sql"><code>Row</code></a>s in this <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a>.</div>
<dl><dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.3.0</dd></dl>
</li>
</ul>
<a name="count()">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>count</h4>
<pre>public long count()</pre>
<div class="block">Returns the number of rows in the <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a>.</div>
<dl><dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.3.0</dd></dl>
</li>
</ul>
<a name="repartition(int)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>repartition</h4>
<pre>public <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a> repartition(int numPartitions)</pre>
<div class="block">Returns a new <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> that has exactly <code>numPartitions</code> partitions.</div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>numPartitions</code> - (undocumented)</dd>
<dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.3.0</dd></dl>
</li>
</ul>
<a name="coalesce(int)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>coalesce</h4>
<pre>public <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a> coalesce(int numPartitions)</pre>
<div class="block">Returns a new <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> that has exactly <code>numPartitions</code> partitions.
Similar to coalesce defined on an <code>RDD</code>, this operation results in a narrow dependency, e.g.
if you go from 1000 partitions to 100 partitions, there will not be a shuffle, instead each of
the 100 new partitions will claim 10 of the current partitions.</div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>numPartitions</code> - (undocumented)</dd>
<dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.4.0</dd></dl>
</li>
</ul>
<a name="distinct()">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>distinct</h4>
<pre>public <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a> distinct()</pre>
<div class="block">Returns a new <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> that contains only the unique rows from this <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a>.
This is an alias for <code>dropDuplicates</code>.</div>
<dl><dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.3.0</dd></dl>
</li>
</ul>
<a name="persist()">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>persist</h4>
<pre>public <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a> persist()</pre>
<dl><dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.3.0</dd></dl>
</li>
</ul>
<a name="cache()">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>cache</h4>
<pre>public <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a> cache()</pre>
<dl><dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.3.0</dd></dl>
</li>
</ul>
<a name="persist(org.apache.spark.storage.StorageLevel)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>persist</h4>
<pre>public <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a> persist(<a href="../../../../org/apache/spark/storage/StorageLevel.html" title="class in org.apache.spark.storage">StorageLevel</a> newLevel)</pre>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>newLevel</code> - (undocumented)</dd>
<dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.3.0</dd></dl>
</li>
</ul>
<a name="unpersist(boolean)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>unpersist</h4>
<pre>public <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a> unpersist(boolean blocking)</pre>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>blocking</code> - (undocumented)</dd>
<dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.3.0</dd></dl>
</li>
</ul>
<a name="unpersist()">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>unpersist</h4>
<pre>public <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a> unpersist()</pre>
<dl><dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.3.0</dd></dl>
</li>
</ul>
<a name="rdd()">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>rdd</h4>
<pre>public <a href="../../../../org/apache/spark/rdd/RDD.html" title="class in org.apache.spark.rdd">RDD</a><<a href="../../../../org/apache/spark/sql/Row.html" title="interface in org.apache.spark.sql">Row</a>> rdd()</pre>
<div class="block">Represents the content of the <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> as an <code>RDD</code> of <a href="../../../../org/apache/spark/sql/Row.html" title="interface in org.apache.spark.sql"><code>Row</code></a>s. Note that the RDD is
memoized. Once called, it won't change even if you change any query planning related Spark SQL
configurations (e.g. <code>spark.sql.shuffle.partitions</code>).</div>
<dl><dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.3.0</dd></dl>
</li>
</ul>
<a name="toJavaRDD()">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>toJavaRDD</h4>
<pre>public <a href="../../../../org/apache/spark/api/java/JavaRDD.html" title="class in org.apache.spark.api.java">JavaRDD</a><<a href="../../../../org/apache/spark/sql/Row.html" title="interface in org.apache.spark.sql">Row</a>> toJavaRDD()</pre>
<div class="block">Returns the content of the <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> as a <code>JavaRDD</code> of <a href="../../../../org/apache/spark/sql/Row.html" title="interface in org.apache.spark.sql"><code>Row</code></a>s.</div>
<dl><dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.3.0</dd></dl>
</li>
</ul>
<a name="javaRDD()">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>javaRDD</h4>
<pre>public <a href="../../../../org/apache/spark/api/java/JavaRDD.html" title="class in org.apache.spark.api.java">JavaRDD</a><<a href="../../../../org/apache/spark/sql/Row.html" title="interface in org.apache.spark.sql">Row</a>> javaRDD()</pre>
<div class="block">Returns the content of the <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> as a <code>JavaRDD</code> of <a href="../../../../org/apache/spark/sql/Row.html" title="interface in org.apache.spark.sql"><code>Row</code></a>s.</div>
<dl><dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.3.0</dd></dl>
</li>
</ul>
<a name="registerTempTable(java.lang.String)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>registerTempTable</h4>
<pre>public void registerTempTable(java.lang.String tableName)</pre>
<div class="block">Registers this <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> as a temporary table using the given name. The lifetime of this
temporary table is tied to the <a href="../../../../org/apache/spark/sql/SQLContext.html" title="class in org.apache.spark.sql"><code>SQLContext</code></a> that was used to create this DataFrame.
<p></div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>tableName</code> - (undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.3.0</dd></dl>
</li>
</ul>
<a name="write()">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>write</h4>
<pre>public <a href="../../../../org/apache/spark/sql/DataFrameWriter.html" title="class in org.apache.spark.sql">DataFrameWriter</a> write()</pre>
<div class="block">:: Experimental ::
Interface for saving the content of the <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> out into external storage.
<p></div>
<dl><dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.4.0</dd></dl>
</li>
</ul>
<a name="toJSON()">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>toJSON</h4>
<pre>public <a href="../../../../org/apache/spark/rdd/RDD.html" title="class in org.apache.spark.rdd">RDD</a><java.lang.String> toJSON()</pre>
<div class="block">Returns the content of the <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> as a RDD of JSON strings.</div>
<dl><dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd><dt><span class="strong">Since:</span></dt>
<dd>1.3.0</dd></dl>
</li>
</ul>
<a name="inputFiles()">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>inputFiles</h4>
<pre>public java.lang.String[] inputFiles()</pre>
<div class="block">Returns a best-effort snapshot of the files that compose this DataFrame. This method simply
asks each constituent BaseRelation for its respective files and takes the union of all results.
Depending on the source relations, this may not find all input files. Duplicates are removed.</div>
<dl><dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd></dl>
</li>
</ul>
<a name="javaToPython()">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>javaToPython</h4>
<pre>protected <a href="../../../../org/apache/spark/api/java/JavaRDD.html" title="class in org.apache.spark.api.java">JavaRDD</a><byte[]> javaToPython()</pre>
<div class="block">Converts a JavaRDD to a PythonRDD.</div>
<dl><dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd></dl>
</li>
</ul>
<a name="toSchemaRDD()">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>toSchemaRDD</h4>
<pre>public <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql">DataFrame</a> toSchemaRDD()</pre>
<div class="block"><span class="strong">Deprecated.</span> <i>As of 1.3.0, replaced by <code>toDF()</code>.</i></div>
<dl><dt><span class="strong">Returns:</span></dt><dd>(undocumented)</dd></dl>
</li>
</ul>
<a name="createJDBCTable(java.lang.String, java.lang.String, boolean)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>createJDBCTable</h4>
<pre>public void createJDBCTable(java.lang.String url,
java.lang.String table,
boolean allowExisting)</pre>
<div class="block"><span class="strong">Deprecated.</span> <i>As of 1.340, replaced by <code>write().jdbc()</code>.</i></div>
<div class="block">Save this <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> to a JDBC database at <code>url</code> under the table name <code>table</code>.
This will run a <code>CREATE TABLE</code> and a bunch of <code>INSERT INTO</code> statements.
If you pass <code>true</code> for <code>allowExisting</code>, it will drop any table with the
given name; if you pass <code>false</code>, it will throw if the table already
exists.</div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>url</code> - (undocumented)</dd><dd><code>table</code> - (undocumented)</dd><dd><code>allowExisting</code> - (undocumented)</dd></dl>
</li>
</ul>
<a name="insertIntoJDBC(java.lang.String, java.lang.String, boolean)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>insertIntoJDBC</h4>
<pre>public void insertIntoJDBC(java.lang.String url,
java.lang.String table,
boolean overwrite)</pre>
<div class="block"><span class="strong">Deprecated.</span> <i>As of 1.4.0, replaced by <code>write().jdbc()</code>.</i></div>
<div class="block">Save this <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> to a JDBC database at <code>url</code> under the table name <code>table</code>.
Assumes the table already exists and has a compatible schema. If you
pass <code>true</code> for <code>overwrite</code>, it will <code>TRUNCATE</code> the table before
performing the <code>INSERT</code>s.
<p>
The table must already exist on the database. It must have a schema
that is compatible with the schema of this RDD; inserting the rows of
the RDD in order via the simple statement
<code>INSERT INTO table VALUES (?, ?, ..., ?)</code> should not fail.</div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>url</code> - (undocumented)</dd><dd><code>table</code> - (undocumented)</dd><dd><code>overwrite</code> - (undocumented)</dd></dl>
</li>
</ul>
<a name="saveAsParquetFile(java.lang.String)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>saveAsParquetFile</h4>
<pre>public void saveAsParquetFile(java.lang.String path)</pre>
<div class="block"><span class="strong">Deprecated.</span> <i>As of 1.4.0, replaced by <code>write().parquet()</code>.</i></div>
<div class="block">Saves the contents of this <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a> as a parquet file, preserving the schema.
Files that are written out using this method can be read back in as a <a href="../../../../org/apache/spark/sql/DataFrame.html" title="class in org.apache.spark.sql"><code>DataFrame</code></a>
using the <code>parquetFile</code> function in <a href="../../../../org/apache/spark/sql/SQLContext.html" title="class in org.apache.spark.sql"><code>SQLContext</code></a>.</div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>path</code> - (undocumented)</dd></dl>
</li>
</ul>
<a name="saveAsTable(java.lang.String)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>saveAsTable</h4>
<pre>public void saveAsTable(java.lang.String tableName)</pre>
<div class="block"><span class="strong">Deprecated.</span> <i>As of 1.4.0, replaced by <code>write().saveAsTable(tableName)</code>.</i></div>
<div class="block">Creates a table from the the contents of this DataFrame.
It will use the default data source configured by spark.sql.sources.default.
This will fail if the table already exists.
<p>
Note that this currently only works with DataFrames that are created from a HiveContext as
there is no notion of a persisted catalog in a standard SQL context. Instead you can write
an RDD out to a parquet file, and then register that file as a table. This "table" can then
be the target of an <code>insertInto</code>.
<p>
When the DataFrame is created from a non-partitioned <code>HadoopFsRelation</code> with a single input
path, and the data source provider can be mapped to an existing Hive builtin SerDe (i.e. ORC
and Parquet), the table is persisted in a Hive compatible format, which means other systems
like Hive will be able to read this table. Otherwise, the table is persisted in a Spark SQL
specific format.
<p></div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>tableName</code> - (undocumented)</dd></dl>
</li>
</ul>
<a name="saveAsTable(java.lang.String, org.apache.spark.sql.SaveMode)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>saveAsTable</h4>
<pre>public void saveAsTable(java.lang.String tableName,
<a href="../../../../org/apache/spark/sql/SaveMode.html" title="enum in org.apache.spark.sql">SaveMode</a> mode)</pre>
<div class="block"><span class="strong">Deprecated.</span> <i>As of 1.4.0, replaced by <code>write().mode(mode).saveAsTable(tableName)</code>.</i></div>
<div class="block">Creates a table from the the contents of this DataFrame, using the default data source
configured by spark.sql.sources.default and <code>SaveMode.ErrorIfExists</code> as the save mode.
<p>
Note that this currently only works with DataFrames that are created from a HiveContext as
there is no notion of a persisted catalog in a standard SQL context. Instead you can write
an RDD out to a parquet file, and then register that file as a table. This "table" can then
be the target of an <code>insertInto</code>.
<p>
When the DataFrame is created from a non-partitioned <code>HadoopFsRelation</code> with a single input
path, and the data source provider can be mapped to an existing Hive builtin SerDe (i.e. ORC
and Parquet), the table is persisted in a Hive compatible format, which means other systems
like Hive will be able to read this table. Otherwise, the table is persisted in a Spark SQL
specific format.
<p></div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>tableName</code> - (undocumented)</dd><dd><code>mode</code> - (undocumented)</dd></dl>
</li>
</ul>
<a name="saveAsTable(java.lang.String, java.lang.String)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>saveAsTable</h4>
<pre>public void saveAsTable(java.lang.String tableName,
java.lang.String source)</pre>
<div class="block"><span class="strong">Deprecated.</span> <i>As of 1.4.0, replaced by <code>write().format(source).saveAsTable(tableName)</code>.</i></div>
<div class="block">Creates a table at the given path from the the contents of this DataFrame
based on a given data source and a set of options,
using <code>SaveMode.ErrorIfExists</code> as the save mode.
<p>
Note that this currently only works with DataFrames that are created from a HiveContext as
there is no notion of a persisted catalog in a standard SQL context. Instead you can write
an RDD out to a parquet file, and then register that file as a table. This "table" can then
be the target of an <code>insertInto</code>.
<p>
When the DataFrame is created from a non-partitioned <code>HadoopFsRelation</code> with a single input
path, and the data source provider can be mapped to an existing Hive builtin SerDe (i.e. ORC
and Parquet), the table is persisted in a Hive compatible format, which means other systems
like Hive will be able to read this table. Otherwise, the table is persisted in a Spark SQL
specific format.
<p></div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>tableName</code> - (undocumented)</dd><dd><code>source</code> - (undocumented)</dd></dl>
</li>
</ul>
<a name="saveAsTable(java.lang.String, java.lang.String, org.apache.spark.sql.SaveMode)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>saveAsTable</h4>
<pre>public void saveAsTable(java.lang.String tableName,
java.lang.String source,
<a href="../../../../org/apache/spark/sql/SaveMode.html" title="enum in org.apache.spark.sql">SaveMode</a> mode)</pre>
<div class="block"><span class="strong">Deprecated.</span> <i>As of 1.4.0, replaced by <code>write().mode(mode).saveAsTable(tableName)</code>.</i></div>
<div class="block">:: Experimental ::
Creates a table at the given path from the the contents of this DataFrame
based on a given data source, <a href="../../../../org/apache/spark/sql/SaveMode.html" title="enum in org.apache.spark.sql"><code>SaveMode</code></a> specified by mode, and a set of options.
<p>
Note that this currently only works with DataFrames that are created from a HiveContext as
there is no notion of a persisted catalog in a standard SQL context. Instead you can write
an RDD out to a parquet file, and then register that file as a table. This "table" can then
be the target of an <code>insertInto</code>.
<p>
When the DataFrame is created from a non-partitioned <code>HadoopFsRelation</code> with a single input
path, and the data source provider can be mapped to an existing Hive builtin SerDe (i.e. ORC
and Parquet), the table is persisted in a Hive compatible format, which means other systems
like Hive will be able to read this table. Otherwise, the table is persisted in a Spark SQL
specific format.
<p></div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>tableName</code> - (undocumented)</dd><dd><code>source</code> - (undocumented)</dd><dd><code>mode</code> - (undocumented)</dd></dl>
</li>
</ul>
<a name="saveAsTable(java.lang.String, java.lang.String, org.apache.spark.sql.SaveMode, java.util.Map)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>saveAsTable</h4>
<pre>public void saveAsTable(java.lang.String tableName,
java.lang.String source,
<a href="../../../../org/apache/spark/sql/SaveMode.html" title="enum in org.apache.spark.sql">SaveMode</a> mode,
java.util.Map<java.lang.String,java.lang.String> options)</pre>
<div class="block"><span class="strong">Deprecated.</span> <i>As of 1.4.0, replaced by
<code>write().format(source).mode(mode).options(options).saveAsTable(tableName)</code>.</i></div>
<div class="block">Creates a table at the given path from the the contents of this DataFrame
based on a given data source, <a href="../../../../org/apache/spark/sql/SaveMode.html" title="enum in org.apache.spark.sql"><code>SaveMode</code></a> specified by mode, and a set of options.
<p>
Note that this currently only works with DataFrames that are created from a HiveContext as
there is no notion of a persisted catalog in a standard SQL context. Instead you can write
an RDD out to a parquet file, and then register that file as a table. This "table" can then
be the target of an <code>insertInto</code>.
<p>
When the DataFrame is created from a non-partitioned <code>HadoopFsRelation</code> with a single input
path, and the data source provider can be mapped to an existing Hive builtin SerDe (i.e. ORC
and Parquet), the table is persisted in a Hive compatible format, which means other systems
like Hive will be able to read this table. Otherwise, the table is persisted in a Spark SQL
specific format.
<p></div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>tableName</code> - (undocumented)</dd><dd><code>source</code> - (undocumented)</dd><dd><code>mode</code> - (undocumented)</dd><dd><code>options</code> - (undocumented)</dd></dl>
</li>
</ul>
<a name="saveAsTable(java.lang.String, java.lang.String, org.apache.spark.sql.SaveMode, scala.collection.immutable.Map)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>saveAsTable</h4>
<pre>public void saveAsTable(java.lang.String tableName,
java.lang.String source,
<a href="../../../../org/apache/spark/sql/SaveMode.html" title="enum in org.apache.spark.sql">SaveMode</a> mode,
scala.collection.immutable.Map<java.lang.String,java.lang.String> options)</pre>
<div class="block"><span class="strong">Deprecated.</span> <i>As of 1.4.0, replaced by
<code>write().format(source).mode(mode).options(options).saveAsTable(tableName)</code>.</i></div>
<div class="block">(Scala-specific)
Creates a table from the the contents of this DataFrame based on a given data source,
<a href="../../../../org/apache/spark/sql/SaveMode.html" title="enum in org.apache.spark.sql"><code>SaveMode</code></a> specified by mode, and a set of options.
<p>
Note that this currently only works with DataFrames that are created from a HiveContext as
there is no notion of a persisted catalog in a standard SQL context. Instead you can write
an RDD out to a parquet file, and then register that file as a table. This "table" can then
be the target of an <code>insertInto</code>.
<p>
When the DataFrame is created from a non-partitioned <code>HadoopFsRelation</code> with a single input
path, and the data source provider can be mapped to an existing Hive builtin SerDe (i.e. ORC
and Parquet), the table is persisted in a Hive compatible format, which means other systems
like Hive will be able to read this table. Otherwise, the table is persisted in a Spark SQL
specific format.
<p></div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>tableName</code> - (undocumented)</dd><dd><code>source</code> - (undocumented)</dd><dd><code>mode</code> - (undocumented)</dd><dd><code>options</code> - (undocumented)</dd></dl>
</li>
</ul>
<a name="save(java.lang.String)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>save</h4>
<pre>public void save(java.lang.String path)</pre>
<div class="block"><span class="strong">Deprecated.</span> <i>As of 1.4.0, replaced by <code>write().save(path)</code>.</i></div>
<div class="block">Saves the contents of this DataFrame to the given path,
using the default data source configured by spark.sql.sources.default and
<code>SaveMode.ErrorIfExists</code> as the save mode.</div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>path</code> - (undocumented)</dd></dl>
</li>
</ul>
<a name="save(java.lang.String, org.apache.spark.sql.SaveMode)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>save</h4>
<pre>public void save(java.lang.String path,
<a href="../../../../org/apache/spark/sql/SaveMode.html" title="enum in org.apache.spark.sql">SaveMode</a> mode)</pre>
<div class="block"><span class="strong">Deprecated.</span> <i>As of 1.4.0, replaced by <code>write().mode(mode).save(path)</code>.</i></div>
<div class="block">Saves the contents of this DataFrame to the given path and <a href="../../../../org/apache/spark/sql/SaveMode.html" title="enum in org.apache.spark.sql"><code>SaveMode</code></a> specified by mode,
using the default data source configured by spark.sql.sources.default.</div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>path</code> - (undocumented)</dd><dd><code>mode</code> - (undocumented)</dd></dl>
</li>
</ul>
<a name="save(java.lang.String, java.lang.String)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>save</h4>
<pre>public void save(java.lang.String path,
java.lang.String source)</pre>
<div class="block"><span class="strong">Deprecated.</span> <i>As of 1.4.0, replaced by <code>write().format(source).save(path)</code>.</i></div>
<div class="block">Saves the contents of this DataFrame to the given path based on the given data source,
using <code>SaveMode.ErrorIfExists</code> as the save mode.</div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>path</code> - (undocumented)</dd><dd><code>source</code> - (undocumented)</dd></dl>
</li>
</ul>
<a name="save(java.lang.String, java.lang.String, org.apache.spark.sql.SaveMode)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>save</h4>
<pre>public void save(java.lang.String path,
java.lang.String source,
<a href="../../../../org/apache/spark/sql/SaveMode.html" title="enum in org.apache.spark.sql">SaveMode</a> mode)</pre>
<div class="block"><span class="strong">Deprecated.</span> <i>As of 1.4.0, replaced by <code>write().format(source).mode(mode).save(path)</code>.</i></div>
<div class="block">Saves the contents of this DataFrame to the given path based on the given data source and
<a href="../../../../org/apache/spark/sql/SaveMode.html" title="enum in org.apache.spark.sql"><code>SaveMode</code></a> specified by mode.</div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>path</code> - (undocumented)</dd><dd><code>source</code> - (undocumented)</dd><dd><code>mode</code> - (undocumented)</dd></dl>
</li>
</ul>
<a name="save(java.lang.String, org.apache.spark.sql.SaveMode, java.util.Map)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>save</h4>
<pre>public void save(java.lang.String source,
<a href="../../../../org/apache/spark/sql/SaveMode.html" title="enum in org.apache.spark.sql">SaveMode</a> mode,
java.util.Map<java.lang.String,java.lang.String> options)</pre>
<div class="block"><span class="strong">Deprecated.</span> <i>As of 1.4.0, replaced by
<code>write().format(source).mode(mode).options(options).save(path)</code>.</i></div>
<div class="block">Saves the contents of this DataFrame based on the given data source,
<a href="../../../../org/apache/spark/sql/SaveMode.html" title="enum in org.apache.spark.sql"><code>SaveMode</code></a> specified by mode, and a set of options.</div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>source</code> - (undocumented)</dd><dd><code>mode</code> - (undocumented)</dd><dd><code>options</code> - (undocumented)</dd></dl>
</li>
</ul>
<a name="save(java.lang.String, org.apache.spark.sql.SaveMode, scala.collection.immutable.Map)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>save</h4>
<pre>public void save(java.lang.String source,
<a href="../../../../org/apache/spark/sql/SaveMode.html" title="enum in org.apache.spark.sql">SaveMode</a> mode,
scala.collection.immutable.Map<java.lang.String,java.lang.String> options)</pre>
<div class="block"><span class="strong">Deprecated.</span> <i>As of 1.4.0, replaced by
<code>write().format(source).mode(mode).options(options).save(path)</code>.</i></div>
<div class="block">(Scala-specific)
Saves the contents of this DataFrame based on the given data source,
<a href="../../../../org/apache/spark/sql/SaveMode.html" title="enum in org.apache.spark.sql"><code>SaveMode</code></a> specified by mode, and a set of options</div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>source</code> - (undocumented)</dd><dd><code>mode</code> - (undocumented)</dd><dd><code>options</code> - (undocumented)</dd></dl>
</li>
</ul>
<a name="insertInto(java.lang.String, boolean)">
<!-- -->
</a>
<ul class="blockList">
<li class="blockList">
<h4>insertInto</h4>
<pre>public void insertInto(java.lang.String tableName,
boolean overwrite)</pre>
<div class="block"><span class="strong">Deprecated.</span> <i>As of 1.4.0, replaced by
<code>write().mode(SaveMode.Append|SaveMode.Overwrite).saveAsTable(tableName)</code>.</i></div>
<div class="block">Adds the rows from this RDD to the specified table, optionally overwriting the existing data.</div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>tableName</code> - (undocumented)</dd><dd><code>overwrite</code> - (undocumented)</dd></dl>
</li>
</ul>
<a name="insertInto(java.lang.String)">
<!-- -->
</a>
<ul class="blockListLast">
<li class="blockList">
<h4>insertInto</h4>
<pre>public void insertInto(java.lang.String tableName)</pre>
<div class="block"><span class="strong">Deprecated.</span> <i>As of 1.4.0, replaced by
<code>write().mode(SaveMode.Append).saveAsTable(tableName)</code>.</i></div>
<div class="block">Adds the rows from this RDD to the specified table.
Throws an exception if the table already exists.</div>
<dl><dt><span class="strong">Parameters:</span></dt><dd><code>tableName</code> - (undocumented)</dd></dl>
</li>
</ul>
</li>
</ul>
</li>
</ul>
</div>
</div>
<!-- ========= END OF CLASS DATA ========= -->
<!-- ======= START OF BOTTOM NAVBAR ====== -->
<div class="bottomNav"><a name="navbar_bottom">
<!-- -->
</a><a href="#skip-navbar_bottom" title="Skip navigation links"></a><a name="navbar_bottom_firstrow">
<!-- -->
</a>
<ul class="navList" title="Navigation">
<li><a href="../../../../overview-summary.html">Overview</a></li>
<li><a href="package-summary.html">Package</a></li>
<li class="navBarCell1Rev">Class</li>
<li><a href="package-tree.html">Tree</a></li>
<li><a href="../../../../deprecated-list.html">Deprecated</a></li>
<li><a href="../../../../index-all.html">Index</a></li>
<li><a href="../../../../help-doc.html">Help</a></li>
</ul>
</div>
<div class="subNav">
<ul class="navList">
<li><a href="../../../../org/apache/spark/sql/ColumnName.html" title="class in org.apache.spark.sql"><span class="strong">Prev Class</span></a></li>
<li><a href="../../../../org/apache/spark/sql/DataFrameNaFunctions.html" title="class in org.apache.spark.sql"><span class="strong">Next Class</span></a></li>
</ul>
<ul class="navList">
<li><a href="../../../../index.html?org/apache/spark/sql/DataFrame.html" target="_top">Frames</a></li>
<li><a href="DataFrame.html" target="_top">No Frames</a></li>
</ul>
<ul class="navList" id="allclasses_navbar_bottom">
<li><a href="../../../../allclasses-noframe.html">All Classes</a></li>
</ul>
<div>
<script type="text/javascript"><!--
allClassesLink = document.getElementById("allclasses_navbar_bottom");
if(window==top) {
allClassesLink.style.display = "block";
}
else {
allClassesLink.style.display = "none";
}
//-->
</script>
</div>
<div>
<ul class="subNavList">
<li>Summary: </li>
<li>Nested | </li>
<li>Field | </li>
<li><a href="#constructor_summary">Constr</a> | </li>
<li><a href="#method_summary">Method</a></li>
</ul>
<ul class="subNavList">
<li>Detail: </li>
<li>Field | </li>
<li><a href="#constructor_detail">Constr</a> | </li>
<li><a href="#method_detail">Method</a></li>
</ul>
</div>
<a name="skip-navbar_bottom">
<!-- -->
</a></div>
<!-- ======== END OF BOTTOM NAVBAR ======= -->
<script defer="defer" type="text/javascript" src="../../../../lib/jquery.js"></script><script defer="defer" type="text/javascript" src="../../../../lib/api-javadocs.js"></script></body>
</html>