diff options
author | Andy Konwinski <andyk@berkeley.edu> | 2012-09-02 23:05:40 -0700 |
---|---|---|
committer | Andy Konwinski <andyk@berkeley.edu> | 2012-09-12 13:03:43 -0700 |
commit | 16da942d66ad3d460889ffcb08ee8c82b1ea7936 (patch) | |
tree | d49349d1376fb070950473658a75a33cf51631e6 /docs/spark-simple-tutorial.md | |
parent | a29ac5f9cf3b63cdb0bdd864dc0fea3d3d8db095 (diff) | |
download | spark-16da942d66ad3d460889ffcb08ee8c82b1ea7936.tar.gz spark-16da942d66ad3d460889ffcb08ee8c82b1ea7936.tar.bz2 spark-16da942d66ad3d460889ffcb08ee8c82b1ea7936.zip |
Adding docs directory containing documentation currently on the wiki
which can be compiled via jekyll, using the command `jekyll`. To compile
and run a local webserver to serve the doc as a website, run
`jekyll --server`.
Diffstat (limited to 'docs/spark-simple-tutorial.md')
-rw-r--r-- | docs/spark-simple-tutorial.md | 41 |
1 files changed, 41 insertions, 0 deletions
diff --git a/docs/spark-simple-tutorial.md b/docs/spark-simple-tutorial.md new file mode 100644 index 0000000000..9875de62bd --- /dev/null +++ b/docs/spark-simple-tutorial.md @@ -0,0 +1,41 @@ +--- +layout: global +title: Tutorial - Running a Simple Spark Application +--- + +1. Create directory for spark demo: + + ~$ mkdir SparkTest + +2. Copy the sbt files in ~/spark/sbt directory: + + ~/SparkTest$ cp -r ../spark/sbt . + +3. Edit the ~/SparkTest/sbt/sbt file to look like this: + + #!/bin/bash + java -Xmx800M -XX:MaxPermSize=150m -jar $(dirname $0)/sbt-launch-*.jar "$@" + +4. To build a Spark application, you need Spark and its dependencies in a single Java archive (JAR) file. Create this JAR in Spark's main directory with sbt as: + + ~/spark$ sbt/sbt assembly + +5. create a source file in ~/SparkTest/src/main/scala directory: + + ~/SparkTest/src/main/scala$ vi Test1.scala + +6. Make the contain of the Test1.scala file like this: + + import spark.SparkContext + import spark.SparkContext._ + object Test1 { + def main(args: Array[String]) { + val sc = new SparkContext("local", "SparkTest") + println(sc.parallelize(1 to 10).reduce(_ + _)) + System.exit(0) + } + } + +7. Run the Test1.scala file: + + ~/SparkTest$ sbt/sbt run |