diff options
| author | Matei Zaharia <matei@eecs.berkeley.edu> | 2012-03-17 13:49:55 -0700 |
|---|---|---|
| committer | Matei Zaharia <matei@eecs.berkeley.edu> | 2012-03-17 13:49:55 -0700 |
| commit | 36c7db72bc172961b66cfa0b9741ac860cc03bb4 (patch) | |
| tree | c9ff07552bfb3d1f2760c964a2180f569e70ada7 /README.md | |
| parent | 08cda89e8a05caf453f46fa1dcf00d67535805f1 (diff) | |
| download | spark-36c7db72bc172961b66cfa0b9741ac860cc03bb4.tar.gz spark-36c7db72bc172961b66cfa0b9741ac860cc03bb4.tar.bz2 spark-36c7db72bc172961b66cfa0b9741ac860cc03bb4.zip | |
Documentation
Diffstat (limited to 'README.md')
| -rw-r--r-- | README.md | 12 |
1 files changed, 12 insertions, 0 deletions
@@ -37,6 +37,15 @@ to connect to. This can be a Mesos URL, or "local" to run locally with one thread, or "local[N]" to run locally with N threads. +## A Note About Hadoop + +Spark uses the Hadoop core library to talk to HDFS and other Hadoop-supported +storage systems. Because the HDFS API has changed in different versions of +Hadoop, you must build Spark against the same version that your cluster runs. +You can change the version by setting the `HADOOP_VERSION` variable at the top +of `project/SparkBuild.scala`, then rebuilding Spark. + + ## Configuration Spark can be configured through two files: `conf/java-opts` and @@ -58,5 +67,8 @@ several Spark-specific variables you can set: - `SPARK_JAVA_OPTS`: Extra options to pass to JVM. +- `MESOS_NATIVE_LIBRARY`: Your Mesos library, if you want to run on a Mesos + cluster. For example, this might be /usr/local/lib/libmesos.so on Linux. + Note that `spark-env.sh` must be a shell script (it must be executable and start with a `#!` header to specify the shell to use). |