diff options
| -rw-r--r-- | README.md | 2 | ||||
| -rw-r--r-- | docs/index.md | 2 | ||||
| -rw-r--r-- | docs/mllib-guide.md | 2 | ||||
| -rw-r--r-- | docs/quick-start.md | 4 | ||||
| -rw-r--r-- | docs/running-on-yarn.md | 2 | ||||
| -rw-r--r-- | docs/scala-programming-guide.md | 10 | ||||
| -rw-r--r-- | docs/spark-debugger.md | 2 | ||||
| -rw-r--r-- | docs/spark-standalone.md | 4 | ||||
| -rwxr-xr-x | make-distribution.sh | 2 |
9 files changed, 15 insertions, 15 deletions
@@ -19,7 +19,7 @@ which is packaged with it. To build Spark and its example programs, run: Once you've built Spark, the easiest way to start using it is the shell: - ./spark-shell + ./bin/spark-shell Or, for the Python API, the Python shell (`./pyspark`). diff --git a/docs/index.md b/docs/index.md index d3ac696d1e..a5c179c19c 100644 --- a/docs/index.md +++ b/docs/index.md @@ -34,7 +34,7 @@ to connect to. This can be a [URL for a distributed cluster](scala-programming-g or `local` to run locally with one thread, or `local[N]` to run locally with N threads. You should start by using `local` for testing. -Finally, you can run Spark interactively through modified versions of the Scala shell (`./spark-shell`) or +Finally, you can run Spark interactively through modified versions of the Scala shell (`./bin/spark-shell`) or Python interpreter (`./pyspark`). These are a great way to learn the framework. # Launching on a Cluster diff --git a/docs/mllib-guide.md b/docs/mllib-guide.md index c1ff9c417c..95537ef185 100644 --- a/docs/mllib-guide.md +++ b/docs/mllib-guide.md @@ -87,7 +87,7 @@ svmAlg.optimizer.setNumIterations(200) val modelL1 = svmAlg.run(parsedData) {% endhighlight %} -Both of the code snippets above can be executed in `spark-shell` to generate a +Both of the code snippets above can be executed in `bin/spark-shell` to generate a classifier for the provided dataset. Available algorithms for binary classification: diff --git a/docs/quick-start.md b/docs/quick-start.md index bdf127d404..912f332236 100644 --- a/docs/quick-start.md +++ b/docs/quick-start.md @@ -20,7 +20,7 @@ $ sbt/sbt assembly ## Basics Spark's interactive shell provides a simple way to learn the API, as well as a powerful tool to analyze datasets interactively. -Start the shell by running `./spark-shell` in the Spark directory. +Start the shell by running `./bin/spark-shell` in the Spark directory. Spark's primary abstraction is a distributed collection of items called a Resilient Distributed Dataset (RDD). RDDs can be created from Hadoop InputFormats (such as HDFS files) or by transforming other RDDs. Let's make a new RDD from the text of the README file in the Spark source directory: @@ -99,7 +99,7 @@ scala> linesWithSpark.count() res9: Long = 15 {% endhighlight %} -It may seem silly to use Spark to explore and cache a 30-line text file. The interesting part is that these same functions can be used on very large data sets, even when they are striped across tens or hundreds of nodes. You can also do this interactively by connecting `spark-shell` to a cluster, as described in the [programming guide](scala-programming-guide.html#initializing-spark). +It may seem silly to use Spark to explore and cache a 30-line text file. The interesting part is that these same functions can be used on very large data sets, even when they are striped across tens or hundreds of nodes. You can also do this interactively by connecting `bin/spark-shell` to a cluster, as described in the [programming guide](scala-programming-guide.html#initializing-spark). # A Standalone App in Scala Now say we wanted to write a standalone application using the Spark API. We will walk through a simple application in both Scala (with SBT), Java (with Maven), and Python. If you are using other build systems, consider using the Spark assembly JAR described in the developer guide. diff --git a/docs/running-on-yarn.md b/docs/running-on-yarn.md index 00cad99881..be32034b3e 100644 --- a/docs/running-on-yarn.md +++ b/docs/running-on-yarn.md @@ -112,7 +112,7 @@ For example: SPARK_JAR=./assembly/target/scala-{{site.SCALA_VERSION}}/spark-assembly-{{site.SPARK_VERSION}}-hadoop2.0.5-alpha.jar \ SPARK_YARN_APP_JAR=examples/target/scala-{{site.SCALA_VERSION}}/spark-examples-assembly-{{site.SPARK_VERSION}}.jar \ - MASTER=yarn-client ./spark-shell + MASTER=yarn-client ./bin/spark-shell # Building Spark for Hadoop/YARN 2.2.x diff --git a/docs/scala-programming-guide.md b/docs/scala-programming-guide.md index 1db255ca53..16ae15046a 100644 --- a/docs/scala-programming-guide.md +++ b/docs/scala-programming-guide.md @@ -13,7 +13,7 @@ At a high level, every Spark application consists of a *driver program* that run A second abstraction in Spark is *shared variables* that can be used in parallel operations. By default, when Spark runs a function in parallel as a set of tasks on different nodes, it ships a copy of each variable used in the function to each task. Sometimes, a variable needs to be shared across tasks, or between tasks and the driver program. Spark supports two types of shared variables: *broadcast variables*, which can be used to cache a value in memory on all nodes, and *accumulators*, which are variables that are only "added" to, such as counters and sums. -This guide shows each of these features and walks through some samples. It assumes some familiarity with Scala, especially with the syntax for [closures](http://www.scala-lang.org/node/133). Note that you can also run Spark interactively using the `spark-shell` script. We highly recommend doing that to follow along! +This guide shows each of these features and walks through some samples. It assumes some familiarity with Scala, especially with the syntax for [closures](http://www.scala-lang.org/node/133). Note that you can also run Spark interactively using the `bin/spark-shell` script. We highly recommend doing that to follow along! # Linking with Spark @@ -54,16 +54,16 @@ object for more advanced configuration. The `master` parameter is a string specifying a [Spark or Mesos cluster URL](#master-urls) to connect to, or a special "local" string to run in local mode, as described below. `appName` is a name for your application, which will be shown in the cluster web UI. Finally, the last two parameters are needed to deploy your code to a cluster if running in distributed mode, as described later. -In the Spark shell, a special interpreter-aware SparkContext is already created for you, in the variable called `sc`. Making your own SparkContext will not work. You can set which master the context connects to using the `MASTER` environment variable, and you can add JARs to the classpath with the `ADD_JARS` variable. For example, to run `spark-shell` on four cores, use +In the Spark shell, a special interpreter-aware SparkContext is already created for you, in the variable called `sc`. Making your own SparkContext will not work. You can set which master the context connects to using the `MASTER` environment variable, and you can add JARs to the classpath with the `ADD_JARS` variable. For example, to run `bin/spark-shell` on four cores, use {% highlight bash %} -$ MASTER=local[4] ./spark-shell +$ MASTER=local[4] ./bin/spark-shell {% endhighlight %} Or, to also add `code.jar` to its classpath, use: {% highlight bash %} -$ MASTER=local[4] ADD_JARS=code.jar ./spark-shell +$ MASTER=local[4] ADD_JARS=code.jar ./bin/spark-shell {% endhighlight %} ### Master URLs @@ -95,7 +95,7 @@ If you want to run your application on a cluster, you will need to specify the t * `sparkHome`: The path at which Spark is installed on your worker machines (it should be the same on all of them). * `jars`: A list of JAR files on the local machine containing your application's code and any dependencies, which Spark will deploy to all the worker nodes. You'll need to package your application into a set of JARs using your build system. For example, if you're using SBT, the [sbt-assembly](https://github.com/sbt/sbt-assembly) plugin is a good way to make a single JAR with your code and dependencies. -If you run `spark-shell` on a cluster, you can add JARs to it by specifying the `ADD_JARS` environment variable before you launch it. This variable should contain a comma-separated list of JARs. For example, `ADD_JARS=a.jar,b.jar ./spark-shell` will launch a shell with `a.jar` and `b.jar` on its classpath. In addition, any new classes you define in the shell will automatically be distributed. +If you run `bin/spark-shell` on a cluster, you can add JARs to it by specifying the `ADD_JARS` environment variable before you launch it. This variable should contain a comma-separated list of JARs. For example, `ADD_JARS=a.jar,b.jar ./bin/spark-shell` will launch a shell with `a.jar` and `b.jar` on its classpath. In addition, any new classes you define in the shell will automatically be distributed. # Resilient Distributed Datasets (RDDs) diff --git a/docs/spark-debugger.md b/docs/spark-debugger.md index d6315d97f4..11c51d5cde 100644 --- a/docs/spark-debugger.md +++ b/docs/spark-debugger.md @@ -39,7 +39,7 @@ where `path/to/event-log` is where you want the event log to go relative to `$SP ### Loading the event log into the debugger -1. Run a Spark shell with `MASTER=<i>host</i> ./spark-shell`. +1. Run a Spark shell with `MASTER=<i>host</i> ./bin/spark-shell`. 2. Use `EventLogReader` to load the event log as follows: {% highlight scala %} spark> val r = new spark.EventLogReader(sc, Some("path/to/event-log")) diff --git a/docs/spark-standalone.md b/docs/spark-standalone.md index c2ce23081a..ec5ae5b4f7 100644 --- a/docs/spark-standalone.md +++ b/docs/spark-standalone.md @@ -143,9 +143,9 @@ constructor](scala-programming-guide.html#initializing-spark). To run an interactive Spark shell against the cluster, run the following command: - MASTER=spark://IP:PORT ./spark-shell + MASTER=spark://IP:PORT ./bin/spark-shell -Note that if you are running spark-shell from one of the spark cluster machines, the `spark-shell` script will +Note that if you are running spark-shell from one of the spark cluster machines, the `bin/spark-shell` script will automatically set MASTER from the `SPARK_MASTER_IP` and `SPARK_MASTER_PORT` variables in `conf/spark-env.sh`. You can also pass an option `-c <numCores>` to control the number of cores that spark-shell uses on the cluster. diff --git a/make-distribution.sh b/make-distribution.sh index 82638a23a0..8765c7e620 100755 --- a/make-distribution.sh +++ b/make-distribution.sh @@ -34,7 +34,7 @@ # 2) cd to deploy dir; ./bin/start-master.sh # 3) Verify master is up by visiting web page, ie http://master-ip:8080. Note the spark:// URL. # 4) ./bin/start-slave.sh 1 <<spark:// URL>> -# 5) MASTER="spark://my-master-ip:7077" ./spark-shell +# 5) MASTER="spark://my-master-ip:7077" ./bin/spark-shell # # Figure out where the Spark framework is installed |