How does Spark relate to Hadoop?
Spark is a fast and powerful engine for processing Hadoop data. It runs in Hadoop clusters through Hadoop YARN or Spark's standalone mode, and it can process data in HDFS, HBase, Cassandra, Hive, and any Hadoop InputFormat. It is designed to perform both general data processing (similar to MapReduce) and new workloads like streaming, interactive queries, and machine learning.
Which languages does Spark support?
Spark supports Scala, Java and Python.
How large a cluster can Spark scale to?
We have seen multiple deployments on over 1000 nodes.
What happens when a cached dataset does not fit in memory?
Spark can either spill it to disk or recompute the partitions that don't fit in RAM each time they are requested. By default, it uses recomputation, but you can set a dataset's storage level to MEMORY_AND_DISK to avoid this.
How can I run Spark on a cluster?
You can use either the standalone deploy mode, which only needs Java to be installed on each node, or the Mesos and YARN cluster managers. If you'd like to run on Amazon EC2, Spark provides EC2 scripts to automatically launch a cluster.
Note that you can also run Spark locally (possibly on multiple cores) without any special setup by just passing local[N] as the master URL, where N is the number of parallel threads you want.
Do I need Hadoop to run Spark?
No, but if you run on a cluster, you will need some form of shared file system (for example, NFS mounted at the same path on each node). If you have this type of filesystem, you can just deploy Spark in standalone mode.
How can I access data in S3?
Use the s3n:// URI scheme (s3n://bucket/path). You will also need to set your Amazon security credentials, either by setting the environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY before your program runs, or by setting fs.s3.awsAccessKeyId and fs.s3.awsSecretAccessKey in SparkContext.hadoopConfiguration.
Does Spark require modified versions of Scala or Python?
No. Spark requires no changes to Scala or compiler plugins. The Python API uses the standard CPython implementation, and can call into existing C libraries for Python such as NumPy.
What are good resources for learning Scala?
Check out First Steps to Scala for a quick introduction, the Scala tutorial for Java programmers, or the free online book Programming in Scala. Scala is easy to transition to if you have Java experience or experience in a similarly high-level language (e.g. Ruby).
In addition, Spark also has Java and Python APIs.
What license is Spark under?
Starting in version 0.8, Spark is under the Apache 2.0 license. Previous versions used the BSD license.
How can I contribute to Spark?
Contact the mailing list or send us a pull request on GitHub (instructions here). We're glad to hear about your experience using Spark and to accept patches.
If you would like to report an issue, post it to the Spark issue tracker.
Where can I get more help?
Please post on the Spark Users mailing list. We'll be glad to help!