Seemlessly mix SQL queries with Spark programs.
Spark SQL lets you query structured data as a distributed dataset (RDD) in Spark, with integrated APIs in Python, Scala and Java. This tight integration makes it easy to run SQL queries alongside complex analytic algorithms.
Load and query data from a variety of sources.
SchemaRDDs provide a single interface for efficiently working with structured data, including Apache Hive tables, parquet files and JSON files.
Run unmodified Hive queries on existing warehouses.
Spark SQL reuses the Hive frontend and metastore, giving you full compatibility with existing Hive data, queries, and UDFs. Simply install it alongside Hive.
Connect through JDBC or ODBC.
Spark SQL includes a server mode with industry standard JDBC and ODBC connectivity.
Use the same engine for both interactive and long queries.
Spark SQL takes advantage of the RDD model to support mid-query fault tolerance, letting it scale to large jobs too. Don't worry about using a different engine for historical data.
Spark SQL is developed as part of Apache Spark. It thus gets tested and updated with each Spark release.
If you have questions about the system, ask on the Spark mailing lists.
The Spark SQL developers welcome contributions. If you'd like to help out, read how to contribute to Spark, and send us a patch!
To get started with Spark SQL: