From 63c7d387a2a69e5ccd23c3de0af0e4cc428fbfd4 Mon Sep 17 00:00:00 2001 From: Matei Alexandru Zaharia Date: Wed, 4 Jun 2014 20:18:25 +0000 Subject: website tweaks: release note links and scaling FAQ --- faq.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) (limited to 'faq.md') diff --git a/faq.md b/faq.md index 815313bbc..4aa96dcf7 100644 --- a/faq.md +++ b/faq.md @@ -22,8 +22,8 @@ streaming, interactive queries, and machine learning.

Which languages does Spark support?

Spark supports Scala, Java and Python.

-

Does Spark require modified versions of Scala or Python?

-

No. Spark requires no changes to Scala or compiler plugins. The Python API uses the standard CPython implementation, and can call into existing C libraries for Python such as NumPy.

+

How large a cluster can Spark scale to?

+

We are aware of multiple deployments on over 1000 nodes.

What happens when a cached dataset does not fit in memory?

Spark can either spill it to disk or recompute the partitions that don't fit in RAM each time they are requested. By default, it uses recomputation, but you can set a dataset's storage level to MEMORY_AND_DISK to avoid this.

@@ -39,6 +39,9 @@ streaming, interactive queries, and machine learning.

How can I access data in S3?

Use the s3n:// URI scheme (s3n://bucket/path). You will also need to set your Amazon security credentials, either by setting the environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY before your program runs, or by setting fs.s3.awsAccessKeyId and fs.s3.awsSecretAccessKey in SparkContext.hadoopConfiguration.

+

Does Spark require modified versions of Scala or Python?

+

No. Spark requires no changes to Scala or compiler plugins. The Python API uses the standard CPython implementation, and can call into existing C libraries for Python such as NumPy.

+

What are good resources for learning Scala?

Check out First Steps to Scala for a quick introduction, the Scala tutorial for Java programmers, or the free online book Programming in Scala. Scala is easy to transition to if you have Java experience or experience in a similarly high-level language (e.g. Ruby).

-- cgit v1.2.3