Lightning-fast cluster computing
Latest News
- Spark 0.8.1 released (Dec 19, 2013)
- Spark Summit 2013 is a Wrap (Dec 15, 2013)
- Announcing the first Spark Summit: December 2, 2013 (Oct 08, 2013)
- Spark 0.8.0 released (Sep 25, 2013)
Speed
Run programs up to 100x faster than Hadoop MapReduce in memory, or 10x faster on disk.
Spark has an advanced DAG execution engine that supports cyclic data flow and in-memory computing.
Ease of Use
Write applications quickly in Java, Scala or Python.
Spark offers over 80 high-level operators that make it easy to build parallel apps. And you can use it interactively from the Scala and Python shells.
file.flatMap(line => line.split(" "))
.map(word => (word, 1))
.reduceByKey(_ + _)
Generality
Combine SQL, streaming, and complex analytics.
Spark powers a stack of high-level tools including Shark for SQL, MLlib for machine learning, GraphX, and Spark Streaming. You can combine these frameworks seamlessly in the same application.
Integrated with Hadoop
Spark can run on Hadoop 2's YARN cluster manager, and can read any existing Hadoop data.
If you have a Hadoop 2 cluster, you can run Spark without any installation needed. Otherwise, Spark is easy to run standalone or on EC2 or Mesos. It can read from HDFS, HBase, Cassandra, and any Hadoop data source.
Community
Spark is used at a wide range of organizations to process large datasets. You can find example use cases at the Spark Summit conference, or on the Powered By page.
There are many ways to reach the community:
- Use the mailing lists to ask questions.
- In-person events include the Bay Area Spark meetup and Spark Summit.
- We use JIRA for issue tracking.
Contributors
Apache Spark is built by a wide set of developers from over 25 companies. Since the project started in 2010, more than 120 developers have contributed to Spark!
The project's committers come from 12 organizations.
If you'd like to participate in Spark, or contribute to the libraries on top of it, learn how to contribute.
Getting Started
Learning Spark is easy whether you come from a Java or Python background:
- Download the latest release — you can run Spark locally on your laptop.
- Read the quick start guide.
- Spark Summit 2013 contained free training videos and exercises that you can run on Amazon EC2.
- Learn how to deploy Spark on a cluster.
Apache Incubator Notice
Apache Spark is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the Apache Incubator. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.