From 9700f2f4afe566412bdb73b443b3aad99b375af1 Mon Sep 17 00:00:00 2001
From: Matei Zaharia Do I need Hadoop to run Spark? No, but if you run on a cluster, you will need some form of shared file system (for example, NFS mounted at the same path on each node). If you have this type of filesystem, you can just deploy Spark in standalone mode. How can I access data in S3? Use the Does Spark require modified versions of Scala or Python? No. Spark requires no changes to Scala or compiler plugins. The Python API uses the standard CPython implementation, and can call into existing C libraries for Python such as NumPy. What are good resources for learning Scala? Check out First Steps to Scala for a quick introduction, the Scala tutorial for Java programmers, or the free online book Programming in Scala. Scala is easy to transition to if you have Java experience or experience in a similarly high-level language (e.g. Ruby). In addition, Spark also has Java and Python APIs. I understand Spark Streaming uses micro-batching. Does this increase latency?
+While Spark does use a micro-batch execution model, this does not have much impact on applications, because the batches can be as short as 0.5 seconds. In most applications of streaming big data, the analytics is done over a larger window (say 10 minutes), or the latency to get data in is higher (e.g. sensors collect readings every 10 seconds). Spark's model enables exactly-once semantics and consistency, meaning the system gives correct results despite slow nodes or failures.
+ Where can I find high-resolution versions of the Spark logo?s3n://
URI scheme (s3n://bucket/path
). You will also need to set your Amazon security credentials, either by setting the environment variables AWS_ACCESS_KEY_ID
and AWS_SECRET_ACCESS_KEY
before your program runs, or by setting fs.s3.awsAccessKeyId
and fs.s3.awsSecretAccessKey
in SparkContext.hadoopConfiguration
.
Can I provide commercial software or services based on Spark?
+ ++Yes, as long as you respect the Apache Software Foundation's +software license +and trademark policy. +In particular, note that there are strong restrictions about how third-party products +use the "Spark" name (names based on Spark are generally not allowed). +Please also refer to our +trademark policy summary. +
+How can I contribute to Spark?
See the Contributing to Spark wiki for more information.
-- cgit v1.2.3