diff options
Diffstat (limited to 'site/docs/0.6.1/api/core/spark/SparkContext.html')
| -rw-r--r-- | site/docs/0.6.1/api/core/spark/SparkContext.html | 1000 |
1 files changed, 1000 insertions, 0 deletions
diff --git a/site/docs/0.6.1/api/core/spark/SparkContext.html b/site/docs/0.6.1/api/core/spark/SparkContext.html new file mode 100644 index 000000000..0120240ef --- /dev/null +++ b/site/docs/0.6.1/api/core/spark/SparkContext.html @@ -0,0 +1,1000 @@ +<?xml version='1.0' encoding='UTF-8'?> +<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN" "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd"> +<html> + <head> + <title>spark.SparkContext</title> + <meta http-equiv="content-type" content="text/html; charset=UTF-8" /> + + <link type="text/css" media="screen" rel="stylesheet" href="../lib/template.css" /> + <script type="text/javascript" src="../lib/jquery.js"></script> + <script type="text/javascript" src="../lib/jquery-ui.js"></script> + <script type="text/javascript" src="../lib/template.js"></script> + <script type="text/javascript" src="../lib/tools.tooltip.js"></script> + + </head> + <body onload="sh_highlightDocument('../lib/', '.min.js');" class="type"> + <div id="definition"> + <a title="Go to companion" href="SparkContext$.html"><img src="../lib/class_to_object_big.png" /></a> + <p id="owner"><a name="spark" class="extype" href="package.html">spark</a></p> + <h1><a title="Go to companion" href="SparkContext$.html">SparkContext</a></h1> + </div> + + <h4 class="signature" id="signature"> + <span class="modifier_kind"> + <span class="modifier"></span> + <span class="kind">class</span> + </span> + <span class="symbol"> + <span class="name">SparkContext</span><span class="result"> extends <a name="spark.Logging" class="extype" href="Logging.html">Logging</a></span> + </span> + </h4> + + <div class="fullcommenttop" id="comment"><div class="comment cmt"><p>Main entry point for Spark functionality. A SparkContext represents the connection to a Spark +cluster, and can be used to create RDDs, accumulators and broadcast variables on that cluster. +</p></div><div class="toggleContainer block"> + <span class="toggle">Linear Supertypes</span> + <div class="superTypes hiddenContent"><a name="spark.Logging" class="extype" href="Logging.html">Logging</a>, AnyRef, <span name="scala.Any" class="extype">Any</span></div> + </div></div> + + + <div id="mbrsel"> + <div id="textfilter"><span class="pre"></span><span class="input"><input accesskey="/" type="text" /></span><span class="post"></span></div> + <div id="order"> + <span class="filtertype">Ordering</span> + <ol><li class="alpha in"><span>Alphabetic</span></li><li class="inherit out"><span>By inheritance</span></li></ol> + </div> + <div id="ancestors"> + <span class="filtertype">Inherited</span> + <ol><li class="hideall out"><span>Hide All</span></li> + <li class="showall in"><span>Show all</span></li></ol> + <ol id="linearization"><li name="spark.SparkContext" class="in"><span>SparkContext</span></li><li name="spark.Logging" class="in"><span>Logging</span></li><li name="scala.AnyRef" class="in"><span>AnyRef</span></li><li name="scala.Any" class="in"><span>Any</span></li></ol> + </div> + <div id="visbl"> + <span class="filtertype">Visibility</span> + <ol><li class="public in"><span>Public</span></li><li class="all out"><span>All</span></li></ol> + </div> + </div> + + <div id="template"> + <div id="allMembers"> + <div class="members" id="constructors"> + <h3>Instance Constructors</h3> + <ol><li visbl="pub" name="spark.SparkContext#this" data-isabs="false"> + <a id="this:SparkContext"></a> + <h4 class="signature"> + <span class="modifier_kind"> + <span class="modifier"></span> + <span class="kind">new</span> + </span> + <span class="symbol"> + <span class="name">SparkContext</span><span class="params">(<span name="master">master: String</span>, <span name="jobName">jobName: String</span>)</span> + </span> + </h4> + <p class="shortcomment cmt"></p><div class="fullcomment"><div class="comment cmt"></div><dl class="paramcmts block"><dt class="param">master</dt><dd class="cmt"><p>Cluster URL to connect to (e.g. mesos://host:port, spark://host:port, local[4]).</p></dd><dt class="param">jobName</dt><dd class="cmt"><p>A name for your job, to display on the cluster web UI +</p></dd></dl></div> + </li><li visbl="pub" name="spark.SparkContext#this" data-isabs="false"> + <a id="this:SparkContext"></a> + <h4 class="signature"> + <span class="modifier_kind"> + <span class="modifier"></span> + <span class="kind">new</span> + </span> + <span class="symbol"> + <span class="name">SparkContext</span><span class="params">(<span name="master">master: String</span>, <span name="jobName">jobName: String</span>, <span name="sparkHome">sparkHome: String</span>, <span name="jars">jars: Seq[String]</span>)</span> + </span> + </h4> + <p class="shortcomment cmt"></p><div class="fullcomment"><div class="comment cmt"></div><dl class="paramcmts block"><dt class="param">master</dt><dd class="cmt"><p>Cluster URL to connect to (e.g. mesos://host:port, spark://host:port, local[4]).</p></dd><dt class="param">jobName</dt><dd class="cmt"><p>A name for your job, to display on the cluster web UI</p></dd><dt class="param">sparkHome</dt><dd class="cmt"><p>Location where Spark is installed on cluster nodes.</p></dd><dt class="param">jars</dt><dd class="cmt"><p>Collection of JARs to send to the cluster. These can be paths on the local file + system or HDFS, HTTP, HTTPS, or FTP URLs. +</p></dd></dl></div> + </li><li visbl="pub" name="spark.SparkContext#this" data-isabs="false"> + <a id="this:SparkContext"></a> + <h4 class="signature"> + <span class="modifier_kind"> + <span class="modifier"></span> + <span class="kind">new</span> + </span> + <span class="symbol"> + <span class="name">SparkContext</span><span class="params">(<span name="master">master: String</span>, <span name="jobName">jobName: String</span>, <span name="sparkHome">sparkHome: String</span>, <span name="jars">jars: Seq[String]</span>, <span name="environment">environment: <span name="scala.collection.Map" class="extype">Map</span>[String, String]</span>)</span> + </span> + </h4> + <p class="shortcomment cmt"></p><div class="fullcomment"><div class="comment cmt"></div><dl class="paramcmts block"><dt class="param">master</dt><dd class="cmt"><p>Cluster URL to connect to (e.g. mesos://host:port, spark://host:port, local[4]).</p></dd><dt class="param">jobName</dt><dd class="cmt"><p>A name for your job, to display on the cluster web UI.</p></dd><dt class="param">sparkHome</dt><dd class="cmt"><p>Location where Spark is installed on cluster nodes.</p></dd><dt class="param">jars</dt><dd class="cmt"><p>Collection of JARs to send to the cluster. These can be paths on the local file + system or HDFS, HTTP, HTTPS, or FTP URLs.</p></dd><dt class="param">environment</dt><dd class="cmt"><p>Environment variables to set on worker nodes. +</p></dd></dl></div> + </li></ol> + </div> + + + + + + <div class="values members" id="values"> + <h3>Value Members</h3> + <ol><li visbl="pub" name="scala.AnyRef#!=" data-isabs="false"> + <a id="!=(AnyRef):Boolean"></a> + <h4 class="signature"> + <span class="modifier_kind"> + <span class="modifier">final </span> + <span class="kind">def</span> + </span> + <span class="symbol"> + <span class="name" title="gt4s: $bang$eq">!=</span><span class="params">(<span name="arg0">arg0: AnyRef</span>)</span><span class="result">: <span name="scala.Boolean" class="extype">Boolean</span></span> + </span> + </h4> + <div class="fullcomment"><dl class="attributes block"> <dt>Definition Classes</dt><dd>AnyRef</dd></dl></div> + </li><li visbl="pub" name="scala.Any#!=" data-isabs="false"> + <a id="!=(Any):Boolean"></a> + <h4 class="signature"> + <span class="modifier_kind"> + <span class="modifier">final </span> + <span class="kind">def</span> + </span> + <span class="symbol"> + <span class="name" title="gt4s: $bang$eq">!=</span><span class="params">(<span name="arg0">arg0: <span name="scala.Any" class="extype">Any</span></span>)</span><span class="result">: <span name="scala.Boolean" class="extype">Boolean</span></span> + </span> + </h4> + <div class="fullcomment"><dl class="attributes block"> <dt>Definition Classes</dt><dd>Any</dd></dl></div> + </li><li visbl="pub" name="scala.AnyRef###" data-isabs="false"> + <a id="##():Int"></a> + <h4 class="signature"> + <span class="modifier_kind"> + <span class="modifier">final </span> + <span class="kind">def</span> + </span> + <span class="symbol"> + <span class="name" title="gt4s: $hash$hash">##</span><span class="params">()</span><span class="result">: <span name="scala.Int" class="extype">Int</span></span> + </span> + </h4> + <div class="fullcomment"><dl class="attributes block"> <dt>Definition Classes</dt><dd>AnyRef → Any</dd></dl></div> + </li><li visbl="pub" name="scala.AnyRef#==" data-isabs="false"> + <a id="==(AnyRef):Boolean"></a> + <h4 class="signature"> + <span class="modifier_kind"> + <span class="modifier">final </span> + <span class="kind">def</span> + </span> + <span class="symbol"> + <span class="name" title="gt4s: $eq$eq">==</span><span class="params">(<span name="arg0">arg0: AnyRef</span>)</span><span class="result">: <span name="scala.Boolean" class="extype">Boolean</span></span> + </span> + </h4> + <div class="fullcomment"><dl class="attributes block"> <dt>Definition Classes</dt><dd>AnyRef</dd></dl></div> + </li><li visbl="pub" name="scala.Any#==" data-isabs="false"> + <a id="==(Any):Boolean"></a> + <h4 class="signature"> + <span class="modifier_kind"> + <span class="modifier">final </span> + <span class="kind">def</span> + </span> + <span class="symbol"> + <span class="name" title="gt4s: $eq$eq">==</span><span class="params">(<span name="arg0">arg0: <span name="scala.Any" class="extype">Any</span></span>)</span><span class="result">: <span name="scala.Boolean" class="extype">Boolean</span></span> + </span> + </h4> + <div class="fullcomment"><dl class="attributes block"> <dt>Definition Classes</dt><dd>Any</dd></dl></div> + </li><li visbl="pub" name="spark.SparkContext#accumulable" data-isabs="false"> + <a id="accumulable[T, R](T)(AccumulableParam[T, R]):Accumulable[T, R]"></a> + <h4 class="signature"> + <span class="modifier_kind"> + <span class="modifier"></span> + <span class="kind">def</span> + </span> + <span class="symbol"> + <span class="name">accumulable</span><span class="tparams">[<span name="T">T</span>, <span name="R">R</span>]</span><span class="params">(<span name="initialValue">initialValue: T</span>)</span><span class="params">(<span class="implicit">implicit </span><span name="param">param: <a name="spark.AccumulableParam" class="extype" href="AccumulableParam.html">AccumulableParam</a>[T, R]</span>)</span><span class="result">: <a name="spark.Accumulable" class="extype" href="Accumulable.html">Accumulable</a>[T, R]</span> + </span> + </h4> + <p class="shortcomment cmt">Create an <a name="spark.Accumulable" class="extype" href="Accumulable.html">Accumulable</a> shared variable, with a <code>+=</code> method</p><div class="fullcomment"><div class="comment cmt"><p>Create an <a name="spark.Accumulable" class="extype" href="Accumulable.html">Accumulable</a> shared variable, with a <code>+=</code> method</p></div><dl class="paramcmts block"><dt class="tparam">T</dt><dd class="cmt"><p>accumulator type</p></dd><dt class="tparam">R</dt><dd class="cmt"><p>type that can be added to the accumulator +</p></dd></dl></div> + </li><li visbl="pub" name="spark.SparkContext#accumulableCollection" data-isabs="false"> + <a id="accumulableCollection[R, T](R)((R) ⇒ Growable[T] with TraversableOnce[T] with Serializable):Accumulable[R, T]"></a> + <h4 class="signature"> + <span class="modifier_kind"> + <span class="modifier"></span> + <span class="kind">def</span> + </span> + <span class="symbol"> + <span class="name">accumulableCollection</span><span class="tparams">[<span name="R">R</span>, <span name="T">T</span>]</span><span class="params">(<span name="initialValue">initialValue: R</span>)</span><span class="params">(<span class="implicit">implicit </span><span name="arg0">arg0: (R) ⇒ <span name="scala.collection.generic.Growable" class="extype">Growable</span>[T] with TraversableOnce[T] with <span name="java.io.Serializable" class="extype">Serializable</span></span>)</span><span class="result">: <a name="spark.Accumulable" class="extype" href="Accumulable.html">Accumulable</a>[R, T]</span> + </span> + </h4> + <p class="shortcomment cmt">Create an accumulator from a "mutable collection" type.</p><div class="fullcomment"><div class="comment cmt"><p>Create an accumulator from a "mutable collection" type.</p><p>Growable and TraversableOnce are the standard APIs that guarantee += and ++=, implemented by +standard mutable collections. So you can use this with mutable Map, Set, etc. +</p></div></div> + </li><li visbl="pub" name="spark.SparkContext#accumulator" data-isabs="false"> + <a id="accumulator[T](T)(AccumulatorParam[T]):Accumulator[T]"></a> + <h4 class="signature"> + <span class="modifier_kind"> + <span class="modifier"></span> + <span class="kind">def</span> + </span> + <span class="symbol"> + <span class="name">accumulator</span><span class="tparams">[<span name="T">T</span>]</span><span class="params">(<span name="initialValue">initialValue: T</span>)</span><span class="params">(<span class="implicit">implicit </span><span name="param">param: <a name="spark.AccumulatorParam" class="extype" href="AccumulatorParam.html">AccumulatorParam</a>[T]</span>)</span><span class="result">: <a name="spark.Accumulator" class="extype" href="Accumulator.html">Accumulator</a>[T]</span> + </span> + </h4> + <p class="shortcomment cmt">Create an <a name="spark.Accumulator" class="extype" href="Accumulator.html">Accumulator</a> variable of a given type, which tasks can "add" values +to using the <code>+=</code> method.</p><div class="fullcomment"><div class="comment cmt"><p>Create an <a name="spark.Accumulator" class="extype" href="Accumulator.html">Accumulator</a> variable of a given type, which tasks can "add" values +to using the <code>+=</code> method. Only the master can access the accumulator's <code>value</code>. +</p></div></div> + </li><li visbl="pub" name="spark.SparkContext#addFile" data-isabs="false"> + <a id="addFile(String):Unit"></a> + <h4 class="signature"> + <span class="modifier_kind"> + <span class="modifier"></span> + <span class="kind">def</span> + </span> + <span class="symbol"> + <span class="name">addFile</span><span class="params">(<span name="path">path: String</span>)</span><span class="result">: <span name="scala.Unit" class="extype">Unit</span></span> + </span> + </h4> + <p class="shortcomment cmt">Add a file to be downloaded into the working directory of this Spark job on every node.</p><div class="fullcomment"><div class="comment cmt"><p>Add a file to be downloaded into the working directory of this Spark job on every node. +The <code>path</code> passed can be either a local file, a file in HDFS (or other Hadoop-supported +filesystems), or an HTTP, HTTPS or FTP URI. +</p></div></div> + </li><li visbl="pub" name="spark.SparkContext#addJar" data-isabs="false"> + <a id="addJar(String):Unit"></a> + <h4 class="signature"> + <span class="modifier_kind"> + <span class="modifier"></span> + <span class="kind">def</span> + </span> + <span class="symbol"> + <span class="name">addJar</span><span class="params">(<span name="path">path: String</span>)</span><span class="result">: <span name="scala.Unit" class="extype">Unit</span></span> + </span> + </h4> + <p class="shortcomment cmt">Adds a JAR dependency for all tasks to be executed on this SparkContext in the future.</p><div class="fullcomment"><div class="comment cmt"><p>Adds a JAR dependency for all tasks to be executed on this SparkContext in the future. +The <code>path</code> passed can be either a local file, a file in HDFS (or other Hadoop-supported +filesystems), or an HTTP, HTTPS or FTP URI. +</p></div></div> + </li><li visbl="pub" name="scala.Any#asInstanceOf" data-isabs="false"> + <a id="asInstanceOf[T0]:T0"></a> + <h4 class="signature"> + <span class="modifier_kind"> + <span class="modifier">final </span> + <span class="kind">def</span> + </span> + <span class="symbol"> + <span class="name">asInstanceOf</span><span class="tparams">[<span name="T0">T0</span>]</span><span class="result">: T0</span> + </span> + </h4> + <div class="fullcomment"><dl class="attributes block"> <dt>Definition Classes</dt><dd>Any</dd></dl></div> + </li><li visbl="pub" name="spark.SparkContext#broadcast" data-isabs="false"> + <a id="broadcast[T](T):Broadcast[T]"></a> + <h4 class="signature"> + <span class="modifier_kind"> + <span class="modifier"></span> + <span class="kind">def</span> + </span> + <span class="symbol"> + <span class="name">broadcast</span><span class="tparams">[<span name="T">T</span>]</span><span class="params">(<span name="value">value: T</span>)</span><span class="result">: <a name="spark.broadcast.Broadcast" class="extype" href="broadcast/Broadcast.html">Broadcast</a>[T]</span> + </span> + </h4> + <p class="shortcomment cmt">Broadcast a read-only variable to the cluster, returning a spark.Broadcast object for +reading it in distributed functions.</p><div class="fullcomment"><div class="comment cmt"><p>Broadcast a read-only variable to the cluster, returning a spark.Broadcast object for +reading it in distributed functions. The variable will be sent to each cluster only once. +</p></div></div> + </li><li visbl="pub" name="spark.SparkContext#clearFiles" data-isabs="false"> + <a id="clearFiles():Unit"></a> + <h4 class="signature"> + <span class="modifier_kind"> + <span class="modifier"></span> + <span class="kind">def</span> + </span> + <span class="symbol"> + <span class="name">clearFiles</span><span class="params">()</span><span class="result">: <span name="scala.Unit" class="extype">Unit</span></span> + </span> + </h4> + <p class="shortcomment cmt">Clear the job's list of files added by <code>addFile</code> so that they do not get donwloaded to +any new nodes.</p> + </li><li visbl="pub" name="spark.SparkContext#clearJars" data-isabs="false"> + <a id="clearJars():Unit"></a> + <h4 class="signature"> + <span class="modifier_kind"> + <span class="modifier"></span> + <span class="kind">def</span> + </span> + <span class="symbol"> + <span class="name">clearJars</span><span class="params">()</span><span class="result">: <span name="scala.Unit" class="extype">Unit</span></span> + </span> + </h4> + <p class="shortcomment cmt">Clear the job's list of JARs added by <code>addJar</code> so that they do not get downloaded to +any new nodes.</p> + </li><li visbl="prt" name="scala.AnyRef#clone" data-isabs="false"> + <a id="clone():AnyRef"></a> + <h4 class="signature"> + <span class="modifier_kind"> + <span class="modifier"></span> + <span class="kind">def</span> + </span> + <span class="symbol"> + <span class="name">clone</span><span class="params">()</span><span class="result">: AnyRef</span> + </span> + </h4> + <div class="fullcomment"><dl class="attributes block"> <dt>Attributes</dt><dd>protected[<a name="java.lang" class="extype" href="../java/lang/package.html">lang</a>] </dd><dt>Definition Classes</dt><dd>AnyRef</dd><dt>Annotations</dt><dd> + <span class="name">@throws</span><span class="args">()</span> + + </dd></dl></div> + </li><li visbl="pub" name="spark.SparkContext#defaultMinSplits" data-isabs="false"> + <a id="defaultMinSplits:Int"></a> + <h4 class="signature"> + <span class="modifier_kind"> + <span class="modifier"></span> + <span class="kind">def</span> + </span> + <span class="symbol"> + <span class="name">defaultMinSplits</span><span class="result">: <span name="scala.Int" class="extype">Int</span></span> + </span> + </h4> + <p class="shortcomment cmt">Default min number of splits for Hadoop RDDs when not given by user</p> + </li><li visbl="pub" name="spark.SparkContext#defaultParallelism" data-isabs="false"> + <a id="defaultParallelism:Int"></a> + <h4 class="signature"> + <span class="modifier_kind"> + <span class="modifier"></span> + <span class="kind">def</span> + </span> + <span class="symbol"> + <span class="name">defaultParallelism</span><span class="result">: <span name="scala.Int" class="extype">Int</span></span> + </span> + </h4> + <p class="shortcomment cmt">Default level of parallelism to use when not given by user (e.</p><div class="fullcomment"><div class="comment cmt"><p>Default level of parallelism to use when not given by user (e.g. for reduce tasks)</p></div></div> + </li><li visbl="pub" name="scala.AnyRef#eq" data-isabs="false"> + <a id="eq(AnyRef):Boolean"></a> + <h4 class="signature"> + <span class="modifier_kind"> + <span class="modifier">final </span> + <span class="kind">def</span> + </span> + <span class="symbol"> + <span class="name">eq</span><span class="params">(<span name="arg0">arg0: AnyRef</span>)</span><span class="result">: <span name="scala.Boolean" class="extype">Boolean</span></span> + </span> + </h4> + <div class="fullcomment"><dl class="attributes block"> <dt>Definition Classes</dt><dd>AnyRef</dd></dl></div> + </li><li visbl="pub" name="scala.AnyRef#equals" data-isabs="false"> + <a id="equals(Any):Boolean"></a> + <h4 class="signature"> + <span class="modifier_kind"> + <span class="modifier"></span> + <span class="kind">def</span> + </span> + <span class="symbol"> + <span class="name">equals</span><span class="params">(<span name="arg0">arg0: <span name="scala.Any" class="extype">Any</span></span>)</span><span class="result">: <span name="scala.Boolean" class="extype">Boolean</span></span> + </span> + </h4> + <div class="fullcomment"><dl class="attributes block"> <dt>Definition Classes</dt><dd>AnyRef → Any</dd></dl></div> + </li><li visbl="prt" name="scala.AnyRef#finalize" data-isabs="false"> + <a id="finalize():Unit"></a> + <h4 class="signature"> + <span class="modifier_kind"> + <span class="modifier"></span> + <span class="kind">def</span> + </span> + <span class="symbol"> + <span class="name">finalize</span><span class="params">()</span><span class="result">: <span name="scala.Unit" class="extype">Unit</span></span> + </span> + </h4> + <div class="fullcomment"><dl class="attributes block"> <dt>Attributes</dt><dd>protected[<a name="java.lang" class="extype" href="../java/lang/package.html">lang</a>] </dd><dt>Definition Classes</dt><dd>AnyRef</dd><dt>Annotations</dt><dd> + <span class="name">@throws</span><span class="args">()</span> + + </dd></dl></div> + </li><li visbl="pub" name="scala.AnyRef#getClass" data-isabs="false"> + <a id="getClass():java.lang.Class[_]"></a> + <h4 class="signature"> + <span class="modifier_kind"> + <span class="modifier">final </span> + <span class="kind">def</span> + </span> + <span class="symbol"> + <span class="name">getClass</span><span class="params">()</span><span class="result">: java.lang.Class[_]</span> + </span> + </h4> + <div class="fullcomment"><dl class="attributes block"> <dt>Definition Classes</dt><dd>AnyRef → Any</dd></dl></div> + </li><li visbl="pub" name="spark.SparkContext#hadoopFile" data-isabs="false"> + <a id="hadoopFile[K, V, F<:InputFormat[K, V]](String)(ClassManifest[K],ClassManifest[V],ClassManifest[F]):RDD[(K, V)]"></a> + <h4 class="signature"> + <span class="modifier_kind"> + <span class="modifier"></span> + <span class="kind">def</span> + </span> + <span class="symbol"> + <span class="name">hadoopFile</span><span class="tparams">[<span name="K">K</span>, <span name="V">V</span>, <span name="F">F <: <span name="org.apache.hadoop.mapred.InputFormat" class="extype">InputFormat</span>[K, V]</span>]</span><span class="params">(<span name="path">path: String</span>)</span><span class="params">(<span class="implicit">implicit </span><span name="km">km: ClassManifest[K]</span>, <span name="vm">vm: ClassManifest[V]</span>, <span name="fm">fm: ClassManifest[F]</span>)</span><span class="result">: <a name="spark.RDD" class="extype" href="RDD.html">RDD</a>[(K, V)]</span> + </span> + </h4> + <p class="shortcomment cmt">Smarter version of hadoopFile() that uses class manifests to figure out the classes of keys, +values and the InputFormat so that users don't need to pass them directly.</p><div class="fullcomment"><div class="comment cmt"><p>Smarter version of hadoopFile() that uses class manifests to figure out the classes of keys, +values and the InputFormat so that users don't need to pass them directly. Instead, callers +can just write, for example,</p><pre> +<span class="kw">val</span> file = sparkContext.hadoopFile[LongWritable, Text, TextInputFormat](path) +</pre></div></div> + </li><li visbl="pub" name="spark.SparkContext#hadoopFile" data-isabs="false"> + <a id="hadoopFile[K, V, F<:InputFormat[K, V]](String,Int)(ClassManifest[K],ClassManifest[V],ClassManifest[F]):RDD[(K, V)]"></a> + <h4 class="signature"> + <span class="modifier_kind"> + <span class="modifier"></span> + <span class="kind">def</span> + </span> + <span class="symbol"> + <span class="name">hadoopFile</span><span class="tparams">[<span name="K">K</span>, <span name="V">V</span>, <span name="F">F <: <span name="org.apache.hadoop.mapred.InputFormat" class="extype">InputFormat</span>[K, V]</span>]</span><span class="params">(<span name="path">path: String</span>, <span name="minSplits">minSplits: <span name="scala.Int" class="extype">Int</span></span>)</span><span class="params">(<span class="implicit">implicit </span><span name="km">km: ClassManifest[K]</span>, <span name="vm">vm: ClassManifest[V]</span>, <span name="fm">fm: ClassManifest[F]</span>)</span><span class="result">: <a name="spark.RDD" class="extype" href="RDD.html">RDD</a>[(K, V)]</span> + </span> + </h4> + <p class="shortcomment cmt">Smarter version of hadoopFile() that uses class manifests to figure out the classes of keys, +values and the InputFormat so that users don't need to pass them directly.</p><div class="fullcomment"><div class="comment cmt"><p>Smarter version of hadoopFile() that uses class manifests to figure out the classes of keys, +values and the InputFormat so that users don't need to pass them directly. Instead, callers +can just write, for example,</p><pre> +<span class="kw">val</span> file = sparkContext.hadoopFile[LongWritable, Text, TextInputFormat](path, minSplits) +</pre></div></div> + </li><li visbl="pub" name="spark.SparkContext#hadoopFile" data-isabs="false"> + <a id="hadoopFile[K, V](String,Class[_ <: org.apache.hadoop.mapred.InputFormat[K,V]],Class[K],Class[V],Int):RDD[(K, V)]"></a> + <h4 class="signature"> + <span class="modifier_kind"> + <span class="modifier"></span> + <span class="kind">def</span> + </span> + <span class="symbol"> + <span class="name">hadoopFile</span><span class="tparams">[<span name="K">K</span>, <span name="V">V</span>]</span><span class="params">(<span name="path">path: String</span>, <span name="inputFormatClass">inputFormatClass: Class[_ <: org.apache.hadoop.mapred.InputFormat[K,V]]</span>, <span name="keyClass">keyClass: Class[K]</span>, <span name="valueClass">valueClass: Class[V]</span>, <span name="minSplits">minSplits: <span name="scala.Int" class="extype">Int</span></span>)</span><span class="result">: <a name="spark.RDD" class="extype" href="RDD.html">RDD</a>[(K, V)]</span> + </span> + </h4> + <p class="shortcomment cmt">Get an RDD for a Hadoop file with an arbitrary InputFormat</p> + </li><li visbl="pub" name="spark.SparkContext#hadoopRDD" data-isabs="false"> + <a id="hadoopRDD[K, V](JobConf,Class[_ <: org.apache.hadoop.mapred.InputFormat[K,V]],Class[K],Class[V],Int):RDD[(K, V)]"></a> + <h4 class="signature"> + <span class="modifier_kind"> + <span class="modifier"></span> + <span class="kind">def</span> + </span> + <span class="symbol"> + <span class="name">hadoopRDD</span><span class="tparams">[<span name="K">K</span>, <span name="V">V</span>]</span><span class="params">(<span name="conf">conf: <span name="org.apache.hadoop.mapred.JobConf" class="extype">JobConf</span></span>, <span name="inputFormatClass">inputFormatClass: Class[_ <: org.apache.hadoop.mapred.InputFormat[K,V]]</span>, <span name="keyClass">keyClass: Class[K]</span>, <span name="valueClass">valueClass: Class[V]</span>, <span name="minSplits">minSplits: <span name="scala.Int" class="extype">Int</span></span>)</span><span class="result">: <a name="spark.RDD" class="extype" href="RDD.html">RDD</a>[(K, V)]</span> + </span> + </h4> + <p class="shortcomment cmt">Get an RDD for a Hadoop-readable dataset from a Hadoop JobConf giving its InputFormat and any +other necessary info (e.</p><div class="fullcomment"><div class="comment cmt"><p>Get an RDD for a Hadoop-readable dataset from a Hadoop JobConf giving its InputFormat and any +other necessary info (e.g. file name for a filesystem-based dataset, table name for HyperTable, +etc). +</p></div></div> + </li><li visbl="pub" name="scala.AnyRef#hashCode" data-isabs="false"> + <a id="hashCode():Int"></a> + <h4 class="signature"> + <span class="modifier_kind"> + <span class="modifier"></span> + <span class="kind">def</span> + </span> + <span class="symbol"> + <span class="name">hashCode</span><span class="params">()</span><span class="result">: <span name="scala.Int" class="extype">Int</span></span> + </span> + </h4> + <div class="fullcomment"><dl class="attributes block"> <dt>Definition Classes</dt><dd>AnyRef → Any</dd></dl></div> + </li><li visbl="prt" name="spark.Logging#initLogging" data-isabs="false"> + <a id="initLogging():Unit"></a> + <h4 class="signature"> + <span class="modifier_kind"> + <span class="modifier"></span> + <span class="kind">def</span> + </span> + <span class="symbol"> + <span class="name">initLogging</span><span class="params">()</span><span class="result">: <span name="scala.Unit" class="extype">Unit</span></span> + </span> + </h4> + <div class="fullcomment"><dl class="attributes block"> <dt>Attributes</dt><dd>protected </dd><dt>Definition Classes</dt><dd><a name="spark.Logging" class="extype" href="Logging.html">Logging</a></dd></dl></div> + </li><li visbl="pub" name="scala.Any#isInstanceOf" data-isabs="false"> + <a id="isInstanceOf[T0]:Boolean"></a> + <h4 class="signature"> + <span class="modifier_kind"> + <span class="modifier">final </span> + <span class="kind">def</span> + </span> + <span class="symbol"> + <span class="name">isInstanceOf</span><span class="tparams">[<span name="T0">T0</span>]</span><span class="result">: <span name="scala.Boolean" class="extype">Boolean</span></span> + </span> + </h4> + <div class="fullcomment"><dl class="attributes block"> <dt>Definition Classes</dt><dd>Any</dd></dl></div> + </li><li visbl="prt" name="spark.Logging#log" data-isabs="false"> + <a id="log:Logger"></a> + <h4 class="signature"> + <span class="modifier_kind"> + <span class="modifier"></span> + <span class="kind">def</span> + </span> + <span class="symbol"> + <span class="name">log</span><span class="result">: <span name="org.slf4j.Logger" class="extype">Logger</span></span> + </span> + </h4> + <div class="fullcomment"><dl class="attributes block"> <dt>Attributes</dt><dd>protected </dd><dt>Definition Classes</dt><dd><a name="spark.Logging" class="extype" href="Logging.html">Logging</a></dd></dl></div> + </li><li visbl="prt" name="spark.Logging#logDebug" data-isabs="false"> + <a id="logDebug(⇒ String,Throwable):Unit"></a> + <h4 class="signature"> + <span class="modifier_kind"> + <span class="modifier"></span> + <span class="kind">def</span> + </span> + <span class="symbol"> + <span class="name">logDebug</span><span class="params">(<span name="msg">msg: ⇒ String</span>, <span name="throwable">throwable: Throwable</span>)</span><span class="result">: <span name="scala.Unit" class="extype">Unit</span></span> + </span> + </h4> + <div class="fullcomment"><dl class="attributes block"> <dt>Attributes</dt><dd>protected </dd><dt>Definition Classes</dt><dd><a name="spark.Logging" class="extype" href="Logging.html">Logging</a></dd></dl></div> + </li><li visbl="prt" name="spark.Logging#logDebug" data-isabs="false"> + <a id="logDebug(⇒ String):Unit"></a> + <h4 class="signature"> + <span class="modifier_kind"> + <span class="modifier"></span> + <span class="kind">def</span> + </span> + <span class="symbol"> + <span class="name">logDebug</span><span class="params">(<span name="msg">msg: ⇒ String</span>)</span><span class="result">: <span name="scala.Unit" class="extype">Unit</span></span> + </span> + </h4> + <div class="fullcomment"><dl class="attributes block"> <dt>Attributes</dt><dd>protected </dd><dt>Definition Classes</dt><dd><a name="spark.Logging" class="extype" href="Logging.html">Logging</a></dd></dl></div> + </li><li visbl="prt" name="spark.Logging#logError" data-isabs="false"> + <a id="logError(⇒ String,Throwable):Unit"></a> + <h4 class="signature"> + <span class="modifier_kind"> + <span class="modifier"></span> + <span class="kind">def</span> + </span> + <span class="symbol"> + <span class="name">logError</span><span class="params">(<span name="msg">msg: ⇒ String</span>, <span name="throwable">throwable: Throwable</span>)</span><span class="result">: <span name="scala.Unit" class="extype">Unit</span></span> + </span> + </h4> + <div class="fullcomment"><dl class="attributes block"> <dt>Attributes</dt><dd>protected </dd><dt>Definition Classes</dt><dd><a name="spark.Logging" class="extype" href="Logging.html">Logging</a></dd></dl></div> + </li><li visbl="prt" name="spark.Logging#logError" data-isabs="false"> + <a id="logError(⇒ String):Unit"></a> + <h4 class="signature"> + <span class="modifier_kind"> + <span class="modifier"></span> + <span class="kind">def</span> + </span> + <span class="symbol"> + <span class="name">logError</span><span class="params">(<span name="msg">msg: ⇒ String</span>)</span><span class="result">: <span name="scala.Unit" class="extype">Unit</span></span> + </span> + </h4> + <div class="fullcomment"><dl class="attributes block"> <dt>Attributes</dt><dd>protected </dd><dt>Definition Classes</dt><dd><a name="spark.Logging" class="extype" href="Logging.html">Logging</a></dd></dl></div> + </li><li visbl="prt" name="spark.Logging#logInfo" data-isabs="false"> + <a id="logInfo(⇒ String,Throwable):Unit"></a> + <h4 class="signature"> + <span class="modifier_kind"> + <span class="modifier"></span> + <span class="kind">def</span> + </span> + <span class="symbol"> + <span class="name">logInfo</span><span class="params">(<span name="msg">msg: ⇒ String</span>, <span name="throwable">throwable: Throwable</span>)</span><span class="result">: <span name="scala.Unit" class="extype">Unit</span></span> + </span> + </h4> + <div class="fullcomment"><dl class="attributes block"> <dt>Attributes</dt><dd>protected </dd><dt>Definition Classes</dt><dd><a name="spark.Logging" class="extype" href="Logging.html">Logging</a></dd></dl></div> + </li><li visbl="prt" name="spark.Logging#logInfo" data-isabs="false"> + <a id="logInfo(⇒ String):Unit"></a> + <h4 class="signature"> + <span class="modifier_kind"> + <span class="modifier"></span> + <span class="kind">def</span> + </span> + <span class="symbol"> + <span class="name">logInfo</span><span class="params">(<span name="msg">msg: ⇒ String</span>)</span><span class="result">: <span name="scala.Unit" class="extype">Unit</span></span> + </span> + </h4> + <div class="fullcomment"><dl class="attributes block"> <dt>Attributes</dt><dd>protected </dd><dt>Definition Classes</dt><dd><a name="spark.Logging" class="extype" href="Logging.html">Logging</a></dd></dl></div> + </li><li visbl="prt" name="spark.Logging#logTrace" data-isabs="false"> + <a id="logTrace(⇒ String,Throwable):Unit"></a> + <h4 class="signature"> + <span class="modifier_kind"> + <span class="modifier"></span> + <span class="kind">def</span> + </span> + <span class="symbol"> + <span class="name">logTrace</span><span class="params">(<span name="msg">msg: ⇒ String</span>, <span name="throwable">throwable: Throwable</span>)</span><span class="result">: <span name="scala.Unit" class="extype">Unit</span></span> + </span> + </h4> + <div class="fullcomment"><dl class="attributes block"> <dt>Attributes</dt><dd>protected </dd><dt>Definition Classes</dt><dd><a name="spark.Logging" class="extype" href="Logging.html">Logging</a></dd></dl></div> + </li><li visbl="prt" name="spark.Logging#logTrace" data-isabs="false"> + <a id="logTrace(⇒ String):Unit"></a> + <h4 class="signature"> + <span class="modifier_kind"> + <span class="modifier"></span> + <span class="kind">def</span> + </span> + <span class="symbol"> + <span class="name">logTrace</span><span class="params">(<span name="msg">msg: ⇒ String</span>)</span><span class="result">: <span name="scala.Unit" class="extype">Unit</span></span> + </span> + </h4> + <div class="fullcomment"><dl class="attributes block"> <dt>Attributes</dt><dd>protected </dd><dt>Definition Classes</dt><dd><a name="spark.Logging" class="extype" href="Logging.html">Logging</a></dd></dl></div> + </li><li visbl="prt" name="spark.Logging#logWarning" data-isabs="false"> + <a id="logWarning(⇒ String,Throwable):Unit"></a> + <h4 class="signature"> + <span class="modifier_kind"> + <span class="modifier"></span> + <span class="kind">def</span> + </span> + <span class="symbol"> + <span class="name">logWarning</span><span class="params">(<span name="msg">msg: ⇒ String</span>, <span name="throwable">throwable: Throwable</span>)</span><span class="result">: <span name="scala.Unit" class="extype">Unit</span></span> + </span> + </h4> + <div class="fullcomment"><dl class="attributes block"> <dt>Attributes</dt><dd>protected </dd><dt>Definition Classes</dt><dd><a name="spark.Logging" class="extype" href="Logging.html">Logging</a></dd></dl></div> + </li><li visbl="prt" name="spark.Logging#logWarning" data-isabs="false"> + <a id="logWarning(⇒ String):Unit"></a> + <h4 class="signature"> + <span class="modifier_kind"> + <span class="modifier"></span> + <span class="kind">def</span> + </span> + <span class="symbol"> + <span class="name">logWarning</span><span class="params">(<span name="msg">msg: ⇒ String</span>)</span><span class="result">: <span name="scala.Unit" class="extype">Unit</span></span> + </span> + </h4> + <div class="fullcomment"><dl class="attributes block"> <dt>Attributes</dt><dd>protected </dd><dt>Definition Classes</dt><dd><a name="spark.Logging" class="extype" href="Logging.html">Logging</a></dd></dl></div> + </li><li visbl="pub" name="spark.SparkContext#makeRDD" data-isabs="false"> + <a id="makeRDD[T](Seq[T],Int)(ClassManifest[T]):RDD[T]"></a> + <h4 class="signature"> + <span class="modifier_kind"> + <span class="modifier"></span> + <span class="kind">def</span> + </span> + <span class="symbol"> + <span class="name">makeRDD</span><span class="tparams">[<span name="T">T</span>]</span><span class="params">(<span name="seq">seq: Seq[T]</span>, <span name="numSlices">numSlices: <span name="scala.Int" class="extype">Int</span></span>)</span><span class="params">(<span class="implicit">implicit </span><span name="arg0">arg0: ClassManifest[T]</span>)</span><span class="result">: <a name="spark.RDD" class="extype" href="RDD.html">RDD</a>[T]</span> + </span> + </h4> + <p class="shortcomment cmt">Distribute a local Scala collection to form an RDD.</p> + </li><li visbl="pub" name="scala.AnyRef#ne" data-isabs="false"> + <a id="ne(AnyRef):Boolean"></a> + <h4 class="signature"> + <span class="modifier_kind"> + <span class="modifier">final </span> + <span class="kind">def</span> + </span> + <span class="symbol"> + <span class="name">ne</span><span class="params">(<span name="arg0">arg0: AnyRef</span>)</span><span class="result">: <span name="scala.Boolean" class="extype">Boolean</span></span> + </span> + </h4> + <div class="fullcomment"><dl class="attributes block"> <dt>Definition Classes</dt><dd>AnyRef</dd></dl></div> + </li><li visbl="pub" name="spark.SparkContext#newAPIHadoopFile" data-isabs="false"> + <a id="newAPIHadoopFile[K, V, F<:InputFormat[K, V]](String,Class[F],Class[K],Class[V],Configuration):RDD[(K, V)]"></a> + <h4 class="signature"> + <span class="modifier_kind"> + <span class="modifier"></span> + <span class="kind">def</span> + </span> + <span class="symbol"> + <span class="name">newAPIHadoopFile</span><span class="tparams">[<span name="K">K</span>, <span name="V">V</span>, <span name="F">F <: <span name="org.apache.hadoop.mapreduce.InputFormat" class="extype">InputFormat</span>[K, V]</span>]</span><span class="params">(<span name="path">path: String</span>, <span name="fClass">fClass: Class[F]</span>, <span name="kClass">kClass: Class[K]</span>, <span name="vClass">vClass: Class[V]</span>, <span name="conf">conf: <span name="org.apache.hadoop.conf.Configuration" class="extype">Configuration</span></span>)</span><span class="result">: <a name="spark.RDD" class="extype" href="RDD.html">RDD</a>[(K, V)]</span> + </span> + </h4> + <p class="shortcomment cmt">Get an RDD for a given Hadoop file with an arbitrary new API InputFormat +and extra configuration options to pass to the input format.</p> + </li><li visbl="pub" name="spark.SparkContext#newAPIHadoopFile" data-isabs="false"> + <a id="newAPIHadoopFile[K, V, F<:InputFormat[K, V]](String)(ClassManifest[K],ClassManifest[V],ClassManifest[F]):RDD[(K, V)]"></a> + <h4 class="signature"> + <span class="modifier_kind"> + <span class="modifier"></span> + <span class="kind">def</span> + </span> + <span class="symbol"> + <span class="name">newAPIHadoopFile</span><span class="tparams">[<span name="K">K</span>, <span name="V">V</span>, <span name="F">F <: <span name="org.apache.hadoop.mapreduce.InputFormat" class="extype">InputFormat</span>[K, V]</span>]</span><span class="params">(<span name="path">path: String</span>)</span><span class="params">(<span class="implicit">implicit </span><span name="km">km: ClassManifest[K]</span>, <span name="vm">vm: ClassManifest[V]</span>, <span name="fm">fm: ClassManifest[F]</span>)</span><span class="result">: <a name="spark.RDD" class="extype" href="RDD.html">RDD</a>[(K, V)]</span> + </span> + </h4> + <p class="shortcomment cmt">Get an RDD for a Hadoop file with an arbitrary new API InputFormat.</p> + </li><li visbl="pub" name="spark.SparkContext#newAPIHadoopRDD" data-isabs="false"> + <a id="newAPIHadoopRDD[K, V, F<:InputFormat[K, V]](Configuration,Class[F],Class[K],Class[V]):RDD[(K, V)]"></a> + <h4 class="signature"> + <span class="modifier_kind"> + <span class="modifier"></span> + <span class="kind">def</span> + </span> + <span class="symbol"> + <span class="name">newAPIHadoopRDD</span><span class="tparams">[<span name="K">K</span>, <span name="V">V</span>, <span name="F">F <: <span name="org.apache.hadoop.mapreduce.InputFormat" class="extype">InputFormat</span>[K, V]</span>]</span><span class="params">(<span name="conf">conf: <span name="org.apache.hadoop.conf.Configuration" class="extype">Configuration</span></span>, <span name="fClass">fClass: Class[F]</span>, <span name="kClass">kClass: Class[K]</span>, <span name="vClass">vClass: Class[V]</span>)</span><span class="result">: <a name="spark.RDD" class="extype" href="RDD.html">RDD</a>[(K, V)]</span> + </span> + </h4> + <p class="shortcomment cmt">Get an RDD for a given Hadoop file with an arbitrary new API InputFormat +and extra configuration options to pass to the input format.</p> + </li><li visbl="pub" name="scala.AnyRef#notify" data-isabs="false"> + <a id="notify():Unit"></a> + <h4 class="signature"> + <span class="modifier_kind"> + <span class="modifier">final </span> + <span class="kind">def</span> + </span> + <span class="symbol"> + <span class="name">notify</span><span class="params">()</span><span class="result">: <span name="scala.Unit" class="extype">Unit</span></span> + </span> + </h4> + <div class="fullcomment"><dl class="attributes block"> <dt>Definition Classes</dt><dd>AnyRef</dd></dl></div> + </li><li visbl="pub" name="scala.AnyRef#notifyAll" data-isabs="false"> + <a id="notifyAll():Unit"></a> + <h4 class="signature"> + <span class="modifier_kind"> + <span class="modifier">final </span> + <span class="kind">def</span> + </span> + <span class="symbol"> + <span class="name">notifyAll</span><span class="params">()</span><span class="result">: <span name="scala.Unit" class="extype">Unit</span></span> + </span> + </h4> + <div class="fullcomment"><dl class="attributes block"> <dt>Definition Classes</dt><dd>AnyRef</dd></dl></div> + </li><li visbl="pub" name="spark.SparkContext#objectFile" data-isabs="false"> + <a id="objectFile[T](String,Int)(ClassManifest[T]):RDD[T]"></a> + <h4 class="signature"> + <span class="modifier_kind"> + <span class="modifier"></span> + <span class="kind">def</span> + </span> + <span class="symbol"> + <span class="name">objectFile</span><span class="tparams">[<span name="T">T</span>]</span><span class="params">(<span name="path">path: String</span>, <span name="minSplits">minSplits: <span name="scala.Int" class="extype">Int</span> = <span class="symbol"><span class="name"><a href="#defaultMinSplits:Int">defaultMinSplits</a></span></span></span>)</span><span class="params">(<span class="implicit">implicit </span><span name="arg0">arg0: ClassManifest[T]</span>)</span><span class="result">: <a name="spark.RDD" class="extype" href="RDD.html">RDD</a>[T]</span> + </span> + </h4> + <p class="shortcomment cmt">Load an RDD saved as a SequenceFile containing serialized objects, with NullWritable keys and +BytesWritable values that contain a serialized partition.</p><div class="fullcomment"><div class="comment cmt"><p>Load an RDD saved as a SequenceFile containing serialized objects, with NullWritable keys and +BytesWritable values that contain a serialized partition. This is still an experimental storage +format and may not be supported exactly as is in future Spark releases. It will also be pretty +slow if you use the default serializer (Java serialization), though the nice thing about it is +that there's very little effort required to save arbitrary objects. +</p></div></div> + </li><li visbl="pub" name="spark.SparkContext#parallelize" data-isabs="false"> + <a id="parallelize[T](Seq[T],Int)(ClassManifest[T]):RDD[T]"></a> + <h4 class="signature"> + <span class="modifier_kind"> + <span class="modifier"></span> + <span class="kind">def</span> + </span> + <span class="symbol"> + <span class="name">parallelize</span><span class="tparams">[<span name="T">T</span>]</span><span class="params">(<span name="seq">seq: Seq[T]</span>, <span name="numSlices">numSlices: <span name="scala.Int" class="extype">Int</span></span>)</span><span class="params">(<span class="implicit">implicit </span><span name="arg0">arg0: ClassManifest[T]</span>)</span><span class="result">: <a name="spark.RDD" class="extype" href="RDD.html">RDD</a>[T]</span> + </span> + </h4> + <p class="shortcomment cmt">Distribute a local Scala collection to form an RDD.</p> + </li><li visbl="pub" name="spark.SparkContext#runApproximateJob" data-isabs="false"> + <a id="runApproximateJob[T, U, R](RDD[T],(TaskContext, Iterator[T]) ⇒ U,ApproximateEvaluator[U, R],Long):PartialResult[R]"></a> + <h4 class="signature"> + <span class="modifier_kind"> + <span class="modifier"></span> + <span class="kind">def</span> + </span> + <span class="symbol"> + <span class="name">runApproximateJob</span><span class="tparams">[<span name="T">T</span>, <span name="U">U</span>, <span name="R">R</span>]</span><span class="params">(<span name="rdd">rdd: <a name="spark.RDD" class="extype" href="RDD.html">RDD</a>[T]</span>, <span name="func">func: (<a name="spark.TaskContext" class="extype" href="TaskContext.html">TaskContext</a>, Iterator[T]) ⇒ U</span>, <span name="evaluator">evaluator: <span name="spark.partial.ApproximateEvaluator" class="extype">ApproximateEvaluator</span>[U, R]</span>, <span name="timeout">timeout: <span name="scala.Long" class="extype">Long</span></span>)</span><span class="result">: <a name="spark.partial.PartialResult" class="extype" href="partial/PartialResult.html">PartialResult</a>[R]</span> + </span> + </h4> + <p class="shortcomment cmt">Run a job that can return approximate results.</p> + </li><li visbl="pub" name="spark.SparkContext#runJob" data-isabs="false"> + <a id="runJob[T, U](RDD[T],(Iterator[T]) ⇒ U)(ClassManifest[U]):Array[U]"></a> + <h4 class="signature"> + <span class="modifier_kind"> + <span class="modifier"></span> + <span class="kind">def</span> + </span> + <span class="symbol"> + <span class="name">runJob</span><span class="tparams">[<span name="T">T</span>, <span name="U">U</span>]</span><span class="params">(<span name="rdd">rdd: <a name="spark.RDD" class="extype" href="RDD.html">RDD</a>[T]</span>, <span name="func">func: (Iterator[T]) ⇒ U</span>)</span><span class="params">(<span class="implicit">implicit </span><span name="arg0">arg0: ClassManifest[U]</span>)</span><span class="result">: <span name="scala.Array" class="extype">Array</span>[U]</span> + </span> + </h4> + <p class="shortcomment cmt">Run a job on all partitions in an RDD and return the results in an array.</p> + </li><li visbl="pub" name="spark.SparkContext#runJob" data-isabs="false"> + <a id="runJob[T, U](RDD[T],(TaskContext, Iterator[T]) ⇒ U)(ClassManifest[U]):Array[U]"></a> + <h4 class="signature"> + <span class="modifier_kind"> + <span class="modifier"></span> + <span class="kind">def</span> + </span> + <span class="symbol"> + <span class="name">runJob</span><span class="tparams">[<span name="T">T</span>, <span name="U">U</span>]</span><span class="params">(<span name="rdd">rdd: <a name="spark.RDD" class="extype" href="RDD.html">RDD</a>[T]</span>, <span name="func">func: (<a name="spark.TaskContext" class="extype" href="TaskContext.html">TaskContext</a>, Iterator[T]) ⇒ U</span>)</span><span class="params">(<span class="implicit">implicit </span><span name="arg0">arg0: ClassManifest[U]</span>)</span><span class="result">: <span name="scala.Array" class="extype">Array</span>[U]</span> + </span> + </h4> + <p class="shortcomment cmt">Run a job on all partitions in an RDD and return the results in an array.</p> + </li><li visbl="pub" name="spark.SparkContext#runJob" data-isabs="false"> + <a id="runJob[T, U](RDD[T],(Iterator[T]) ⇒ U,Seq[Int],Boolean)(ClassManifest[U]):Array[U]"></a> + <h4 class="signature"> + <span class="modifier_kind"> + <span class="modifier"></span> + <span class="kind">def</span> + </span> + <span class="symbol"> + <span class="name">runJob</span><span class="tparams">[<span name="T">T</span>, <span name="U">U</span>]</span><span class="params">(<span name="rdd">rdd: <a name="spark.RDD" class="extype" href="RDD.html">RDD</a>[T]</span>, <span name="func">func: (Iterator[T]) ⇒ U</span>, <span name="partitions">partitions: Seq[<span name="scala.Int" class="extype">Int</span>]</span>, <span name="allowLocal">allowLocal: <span name="scala.Boolean" class="extype">Boolean</span></span>)</span><span class="params">(<span class="implicit">implicit </span><span name="arg0">arg0: ClassManifest[U]</span>)</span><span class="result">: <span name="scala.Array" class="extype">Array</span>[U]</span> + </span> + </h4> + <p class="shortcomment cmt">Run a job on a given set of partitions of an RDD, but take a function of type +<code>Iterator[T] => U</code> instead of <code>(TaskContext, Iterator[T]) => U</code>.</p> + </li><li visbl="pub" name="spark.SparkContext#runJob" data-isabs="false"> + <a id="runJob[T, U](RDD[T],(TaskContext, Iterator[T]) ⇒ U,Seq[Int],Boolean)(ClassManifest[U]):Array[U]"></a> + <h4 class="signature"> + <span class="modifier_kind"> + <span class="modifier"></span> + <span class="kind">def</span> + </span> + <span class="symbol"> + <span class="name">runJob</span><span class="tparams">[<span name="T">T</span>, <span name="U">U</span>]</span><span class="params">(<span name="rdd">rdd: <a name="spark.RDD" class="extype" href="RDD.html">RDD</a>[T]</span>, <span name="func">func: (<a name="spark.TaskContext" class="extype" href="TaskContext.html">TaskContext</a>, Iterator[T]) ⇒ U</span>, <span name="partitions">partitions: Seq[<span name="scala.Int" class="extype">Int</span>]</span>, <span name="allowLocal">allowLocal: <span name="scala.Boolean" class="extype">Boolean</span></span>)</span><span class="params">(<span class="implicit">implicit </span><span name="arg0">arg0: ClassManifest[U]</span>)</span><span class="result">: <span name="scala.Array" class="extype">Array</span>[U]</span> + </span> + </h4> + <p class="shortcomment cmt">Run a function on a given set of partitions in an RDD and return the results.</p><div class="fullcomment"><div class="comment cmt"><p>Run a function on a given set of partitions in an RDD and return the results. This is the main +entry point to the scheduler, by which all actions get launched. The allowLocal flag specifies +whether the scheduler can run the computation on the master rather than shipping it out to the +cluster, for short actions like first(). +</p></div></div> + </li><li visbl="pub" name="spark.SparkContext#sequenceFile" data-isabs="false"> + <a id="sequenceFile[K, V](String,Int)(ClassManifest[K],ClassManifest[V],() ⇒ WritableConverter[K],() ⇒ WritableConverter[V]):RDD[(K, V)]"></a> + <h4 class="signature"> + <span class="modifier_kind"> + <span class="modifier"></span> + <span class="kind">def</span> + </span> + <span class="symbol"> + <span class="name">sequenceFile</span><span class="tparams">[<span name="K">K</span>, <span name="V">V</span>]</span><span class="params">(<span name="path">path: String</span>, <span name="minSplits">minSplits: <span name="scala.Int" class="extype">Int</span> = <span class="symbol"><span class="name"><a href="#defaultMinSplits:Int">defaultMinSplits</a></span></span></span>)</span><span class="params">(<span class="implicit">implicit </span><span name="km">km: ClassManifest[K]</span>, <span name="vm">vm: ClassManifest[V]</span>, <span name="kcf">kcf: () ⇒ <span name="spark.WritableConverter" class="extype">WritableConverter</span>[K]</span>, <span name="vcf">vcf: () ⇒ <span name="spark.WritableConverter" class="extype">WritableConverter</span>[V]</span>)</span><span class="result">: <a name="spark.RDD" class="extype" href="RDD.html">RDD</a>[(K, V)]</span> + </span> + </h4> + <p class="shortcomment cmt">Version of sequenceFile() for types implicitly convertible to Writables through a +WritableConverter.</p><div class="fullcomment"><div class="comment cmt"><p>Version of sequenceFile() for types implicitly convertible to Writables through a +WritableConverter. For example, to access a SequenceFile where the keys are Text and the +values are IntWritable, you could simply write</p><pre> +sparkContext.sequenceFile[<span class="std">String</span>, <span class="std">Int</span>](path, ...) +</pre><p>WritableConverters are provided in a somewhat strange way (by an implicit function) to support +both subclasses of Writable and types for which we define a converter (e.g. Int to +IntWritable). The most natural thing would've been to have implicit objects for the +converters, but then we couldn't have an object for every subclass of Writable (you can't +have a parameterized singleton object). We use functions instead to create a new converter +for the appropriate type. In addition, we pass the converter a ClassManifest of its type to +allow it to figure out the Writable class to use in the subclass case. +</p></div></div> + </li><li visbl="pub" name="spark.SparkContext#sequenceFile" data-isabs="false"> + <a id="sequenceFile[K, V](String,Class[K],Class[V]):RDD[(K, V)]"></a> + <h4 class="signature"> + <span class="modifier_kind"> + <span class="modifier"></span> + <span class="kind">def</span> + </span> + <span class="symbol"> + <span class="name">sequenceFile</span><span class="tparams">[<span name="K">K</span>, <span name="V">V</span>]</span><span class="params">(<span name="path">path: String</span>, <span name="keyClass">keyClass: Class[K]</span>, <span name="valueClass">valueClass: Class[V]</span>)</span><span class="result">: <a name="spark.RDD" class="extype" href="RDD.html">RDD</a>[(K, V)]</span> + </span> + </h4> + <p class="shortcomment cmt">Get an RDD for a Hadoop SequenceFile with given key and value types.</p> + </li><li visbl="pub" name="spark.SparkContext#sequenceFile" data-isabs="false"> + <a id="sequenceFile[K, V](String,Class[K],Class[V],Int):RDD[(K, V)]"></a> + <h4 class="signature"> + <span class="modifier_kind"> + <span class="modifier"></span> + <span class="kind">def</span> + </span> + <span class="symbol"> + <span class="name">sequenceFile</span><span class="tparams">[<span name="K">K</span>, <span name="V">V</span>]</span><span class="params">(<span name="path">path: String</span>, <span name="keyClass">keyClass: Class[K]</span>, <span name="valueClass">valueClass: Class[V]</span>, <span name="minSplits">minSplits: <span name="scala.Int" class="extype">Int</span></span>)</span><span class="result">: <a name="spark.RDD" class="extype" href="RDD.html">RDD</a>[(K, V)]</span> + </span> + </h4> + <p class="shortcomment cmt">Get an RDD for a Hadoop SequenceFile with given key and value types.</p> + </li><li visbl="pub" name="spark.SparkContext#sparkHome" data-isabs="false"> + <a id="sparkHome:String"></a> + <h4 class="signature"> + <span class="modifier_kind"> + <span class="modifier"></span> + <span class="kind">val</span> + </span> + <span class="symbol"> + <span class="name">sparkHome</span><span class="result">: String</span> + </span> + </h4> + <p class="shortcomment cmt">Location where Spark is installed on cluster nodes.</p> + </li><li visbl="pub" name="spark.SparkContext#stop" data-isabs="false"> + <a id="stop():Unit"></a> + <h4 class="signature"> + <span class="modifier_kind"> + <span class="modifier"></span> + <span class="kind">def</span> + </span> + <span class="symbol"> + <span class="name">stop</span><span class="params">()</span><span class="result">: <span name="scala.Unit" class="extype">Unit</span></span> + </span> + </h4> + <p class="shortcomment cmt">Shut down the SparkContext.</p> + </li><li visbl="pub" name="scala.AnyRef#synchronized" data-isabs="false"> + <a id="synchronized[T0](⇒ T0):T0"></a> + <h4 class="signature"> + <span class="modifier_kind"> + <span class="modifier">final </span> + <span class="kind">def</span> + </span> + <span class="symbol"> + <span class="name">synchronized</span><span class="tparams">[<span name="T0">T0</span>]</span><span class="params">(<span name="arg0">arg0: ⇒ T0</span>)</span><span class="result">: T0</span> + </span> + </h4> + <div class="fullcomment"><dl class="attributes block"> <dt>Definition Classes</dt><dd>AnyRef</dd></dl></div> + </li><li visbl="pub" name="spark.SparkContext#textFile" data-isabs="false"> + <a id="textFile(String,Int):RDD[String]"></a> + <h4 class="signature"> + <span class="modifier_kind"> + <span class="modifier"></span> + <span class="kind">def</span> + </span> + <span class="symbol"> + <span class="name">textFile</span><span class="params">(<span name="path">path: String</span>, <span name="minSplits">minSplits: <span name="scala.Int" class="extype">Int</span> = <span class="symbol"><span class="name"><a href="#defaultMinSplits:Int">defaultMinSplits</a></span></span></span>)</span><span class="result">: <a name="spark.RDD" class="extype" href="RDD.html">RDD</a>[String]</span> + </span> + </h4> + <p class="shortcomment cmt">Read a text file from HDFS, a local file system (available on all nodes), or any +Hadoop-supported file system URI, and return it as an RDD of Strings.</p> + </li><li visbl="pub" name="scala.AnyRef#toString" data-isabs="false"> + <a id="toString():String"></a> + <h4 class="signature"> + <span class="modifier_kind"> + <span class="modifier"></span> + <span class="kind">def</span> + </span> + <span class="symbol"> + <span class="name">toString</span><span class="params">()</span><span class="result">: <span name="java.lang.String" class="extype">String</span></span> + </span> + </h4> + <div class="fullcomment"><dl class="attributes block"> <dt>Definition Classes</dt><dd>AnyRef → Any</dd></dl></div> + </li><li visbl="pub" name="spark.SparkContext#union" data-isabs="false"> + <a id="union[T](RDD[T],RDD[T]*)(ClassManifest[T]):RDD[T]"></a> + <h4 class="signature"> + <span class="modifier_kind"> + <span class="modifier"></span> + <span class="kind">def</span> + </span> + <span class="symbol"> + <span class="name">union</span><span class="tparams">[<span name="T">T</span>]</span><span class="params">(<span name="first">first: <a name="spark.RDD" class="extype" href="RDD.html">RDD</a>[T]</span>, <span name="rest">rest: <a name="spark.RDD" class="extype" href="RDD.html">RDD</a>[T]*</span>)</span><span class="params">(<span class="implicit">implicit </span><span name="arg0">arg0: ClassManifest[T]</span>)</span><span class="result">: <a name="spark.RDD" class="extype" href="RDD.html">RDD</a>[T]</span> + </span> + </h4> + <p class="shortcomment cmt">Build the union of a list of RDDs passed as variable-length arguments.</p> + </li><li visbl="pub" name="spark.SparkContext#union" data-isabs="false"> + <a id="union[T](Seq[RDD[T]])(ClassManifest[T]):RDD[T]"></a> + <h4 class="signature"> + <span class="modifier_kind"> + <span class="modifier"></span> + <span class="kind">def</span> + </span> + <span class="symbol"> + <span class="name">union</span><span class="tparams">[<span name="T">T</span>]</span><span class="params">(<span name="rdds">rdds: Seq[<a name="spark.RDD" class="extype" href="RDD.html">RDD</a>[T]]</span>)</span><span class="params">(<span class="implicit">implicit </span><span name="arg0">arg0: ClassManifest[T]</span>)</span><span class="result">: <a name="spark.RDD" class="extype" href="RDD.html">RDD</a>[T]</span> + </span> + </h4> + <p class="shortcomment cmt">Build the union of a list of RDDs.</p> + </li><li visbl="pub" name="scala.AnyRef#wait" data-isabs="false"> + <a id="wait():Unit"></a> + <h4 class="signature"> + <span class="modifier_kind"> + <span class="modifier">final </span> + <span class="kind">def</span> + </span> + <span class="symbol"> + <span class="name">wait</span><span class="params">()</span><span class="result">: <span name="scala.Unit" class="extype">Unit</span></span> + </span> + </h4> + <div class="fullcomment"><dl class="attributes block"> <dt>Definition Classes</dt><dd>AnyRef</dd><dt>Annotations</dt><dd> + <span class="name">@throws</span><span class="args">()</span> + + </dd></dl></div> + </li><li visbl="pub" name="scala.AnyRef#wait" data-isabs="false"> + <a id="wait(Long,Int):Unit"></a> + <h4 class="signature"> + <span class="modifier_kind"> + <span class="modifier">final </span> + <span class="kind">def</span> + </span> + <span class="symbol"> + <span class="name">wait</span><span class="params">(<span name="arg0">arg0: <span name="scala.Long" class="extype">Long</span></span>, <span name="arg1">arg1: <span name="scala.Int" class="extype">Int</span></span>)</span><span class="result">: <span name="scala.Unit" class="extype">Unit</span></span> + </span> + </h4> + <div class="fullcomment"><dl class="attributes block"> <dt>Definition Classes</dt><dd>AnyRef</dd><dt>Annotations</dt><dd> + <span class="name">@throws</span><span class="args">()</span> + + </dd></dl></div> + </li><li visbl="pub" name="scala.AnyRef#wait" data-isabs="false"> + <a id="wait(Long):Unit"></a> + <h4 class="signature"> + <span class="modifier_kind"> + <span class="modifier">final </span> + <span class="kind">def</span> + </span> + <span class="symbol"> + <span class="name">wait</span><span class="params">(<span name="arg0">arg0: <span name="scala.Long" class="extype">Long</span></span>)</span><span class="result">: <span name="scala.Unit" class="extype">Unit</span></span> + </span> + </h4> + <div class="fullcomment"><dl class="attributes block"> <dt>Definition Classes</dt><dd>AnyRef</dd><dt>Annotations</dt><dd> + <span class="name">@throws</span><span class="args">()</span> + + </dd></dl></div> + </li></ol> + </div> + + + </div> + + <div id="inheritedMembers"> + <div name="spark.Logging" class="parent"> + <h3>Inherited from <a name="spark.Logging" class="extype" href="Logging.html">Logging</a></h3> + </div><div name="scala.AnyRef" class="parent"> + <h3>Inherited from AnyRef</h3> + </div><div name="scala.Any" class="parent"> + <h3>Inherited from <span name="scala.Any" class="extype">Any</span></h3> + </div> + </div> + + </div> + + <div id="tooltip"></div> + + <div id="footer"> </div> + + + </body> + </html>
\ No newline at end of file |