diff --git a/site/examples.html b/site/examples.html
index 5431f5dda..1be96be50 100644
--- a/site/examples.html
+++ b/site/examples.html
@@ -213,11 +213,11 @@ In this page, we will show examples using RDD API as well as examples using high
-
text_file=sc.textFile("hdfs://...")
+
+counts.saveAsTextFile("hdfs://...")
@@ -225,11 +225,11 @@ In this page, we will show examples using RDD API as well as examples using high
-
valtextFile=sc.textFile("hdfs://...")
+
+counts.saveAsTextFile("hdfs://...")
@@ -237,7 +237,7 @@ In this page, we will show examples using RDD API as well as examples using high
We are happy to announce the availability of Spark 2.0.0! Visit the release notes to read about the new features, or download the release today.
-
@@ -211,7 +210,6 @@
June 16, 2016
Call for presentations is now open for Spark Summit EU! The event will take place on October 25-27 in Brussels. Submissions are welcome across a variety of Spark-related topics, including applications, development, data science, enterprise, spark ecosystem and research. Please submit by July 1 to be considered.
-
@@ -231,7 +229,6 @@
April 17, 2016
The agenda for Spark Summit 2016 is now available! The summit kicks off on June 6th with a full day of Spark training followed by over 90+ talks featuring speakers from Airbnb, Baidu, Bloomberg, Databricks, Duke, IBM, Microsoft, Netflix, Uber, UC Berkeley. Check out the full schedule and register to attend!
-
@@ -251,7 +248,6 @@
February 11, 2016
Call for presentations is now open for Spark Summit San Francisco! The event will take place on June 6-8 in San Francisco. Submissions are welcome across a variety of Spark-related topics, including applications, development, data science, business value, spark ecosystem and research. Please submit by February 29th to be considered.
-
@@ -261,7 +257,6 @@
January 14, 2016
The agenda for Spark Summit East is now posted, with 60 talks from organizations including Netflix, Comcast, Blackrock, Bloomberg and others. The 2nd annual Spark Summit East will run February 16-18th in NYC and feature a full program of speakers along with Spark training opportunities. More details are available on the Spark Summit East website, where you can also register to attend.
-
@@ -284,7 +279,6 @@ With this release the Spark community continues to grow, with contributions from
November 19, 2015
Call for presentations is closing soon for Spark Summit East! The event will take place on February 16th-18th in New York City. Submissions are welcome across a variety of Spark-related topics, including applications, development, data science, enterprise, and research. Please submit by November 22nd to be considered.
-
@@ -304,7 +298,6 @@ With this release the Spark community continues to grow, with contributions from
October 14, 2015
Abstract submissions are now open for the 2nd Spark Summit East! The event will take place on February 16th-18th in New York City. Submissions are welcome across a variety of Spark-related topics, including applications, development, data science, enterprise, and research.
-
@@ -334,7 +327,6 @@ With this release the Spark community continues to grow, with contributions from
September 7, 2015
The agenda for Spark Summit Europe is now posted, with 38 talks from organizations including Barclays, Netflix, Elsevier, Intel and others. This inaugural Spark conference in Europe will run October 27th-29th 2015 in Amsterdam and feature a full program of speakers along with Spark training opportunities. More details are available on the Spark Summit Europe website, where you can also register to attend.
-
@@ -354,7 +346,6 @@ With this release the Spark community continues to grow, with contributions from
June 29, 2015
The videos and slides for Spark Summit 2015 are now all available online! The talks include technical roadmap discussions, deep dives on Spark components, and use cases built on top of Spark.
-
@@ -371,7 +362,7 @@ With this release the Spark community continues to grow, with contributions from
There is one month left until Spark Summit 2015, which
will be held in San Francisco on June 15th to 17th.
@@ -383,10 +374,9 @@ The Summit will contain presen
Abstract submissions are now open for the first ever Spark Summit Europe. The event will take place on October 27th to 29th in Amsterdam. Submissions are welcome across a variety of Spark related topics, including use cases and ongoing development.
-
@@ -395,7 +385,7 @@ The Summit will contain presen
The videos and slides for Spark Summit East 2015 are now all available online. Watch them to get the latest news from the Spark community as well as use cases and applications built on top.
+
The videos and slides for Spark Summit East 2015 are now all available online. Watch them to get the latest news from the Spark community as well as use cases and applications built on top.
@@ -405,7 +395,7 @@ The Summit will contain presen
We are happy to announce the availability of Spark 1.2.2 and Spark 1.3.1! These are both maintenance releases that collectively feature the work of more than 90 developers.
+
We are happy to announce the availability of Spark 1.2.2 and Spark 1.3.1! These are both maintenance releases that collectively feature the work of more than 90 developers.
@@ -517,7 +507,7 @@ The Summit will contain presen
We are happy to announce the availability of
Spark 0.9.2! Apache Spark 0.9.2 is a maintenance release with bug fixes. We recommend all 0.9.x users to upgrade to this stable release.
-Contributions to this release came from 28 developers.
+Contributions to this release came from 28 developers.
@@ -588,7 +578,7 @@ about the latest happenings in Spark.
We are happy to announce the availability of
Spark 0.9.1! Apache Spark 0.9.1 is a maintenance release with bug fixes, performance improvements, better stability with YARN and
improved parity of the Scala and Python API. We recommend all 0.9.0 users to upgrade to this stable release.
-Contributions to this release came from 37 developers.
+Contributions to this release came from 37 developers.
@@ -636,7 +626,6 @@ hardened YARN support.
December 19, 2013
We’ve just posted Spark Release 0.8.1, a maintenance and performance release for the Scala 2.9 version of Spark. 0.8.1 includes support for YARN 2.2, a high availability mode for the standalone scheduler, optimizations to the shuffle, and many other improvements. We recommend that all users update to this release. Visit the release notes to read about the new features, or download the release today.
-
@@ -667,7 +656,6 @@ Over 450 Spark developers and enthusiasts from 13 countries and more than 180 co
September 25, 2013
We’re proud to announce the release of Apache Spark 0.8.0. Spark 0.8.0 is a major release that includes many new capabilities and usability improvements. It’s also our first release under the Apache incubator. It is the largest Spark release yet, with contributions from 67 developers and 24 companies. Major new features include an expanded monitoring framework and UI, a machine learning library, and support for running Spark inside of YARN.
-
@@ -697,7 +685,6 @@ Over 450 Spark developers and enthusiasts from 13 countries and more than 180 co
July 23, 2013
Want to learn how to use Spark, Shark, GraphX, and related technologies in person? The AMP Lab is hosting a two-day training workshop for them on August 29th and 30th in Berkeley. The workshop will include tutorials, talks from users, and over four hours of hands-on exercises. Registration is now open on the AMP Camp website, for a price of $250 per person. We recommend signing up early because last year’s workshop was sold out.
-
@@ -718,7 +705,6 @@ Over 450 Spark developers and enthusiasts from 13 countries and more than 180 co
Most users will probably want the User list, but individuals interested in contributing code to the project should also subscribe to the Dev list.
-
@@ -728,7 +714,6 @@ Over 450 Spark developers and enthusiasts from 13 countries and more than 180 co
July 16, 2013
We’ve just posted Spark Release 0.7.3, a maintenance release that contains several fixes, including streaming API updates and new functionality for adding JARs to a spark-shell session. We recommend that all users update to this release. Visit the release notes to read about the new features, or download the release today.
-
@@ -738,7 +723,6 @@ Over 450 Spark developers and enthusiasts from 13 countries and more than 180 co
June 21, 2013
Spark, its creators at the AMP Lab, and some of its users were featured in a Wired Enterprise article a few days ago. Read on to learn a little about how Spark is being used in industry.
-
@@ -748,7 +732,6 @@ Over 450 Spark developers and enthusiasts from 13 countries and more than 180 co
June 21, 2013
Spark was recently accepted into the Apache Incubator, which will serve as the long-term home for the project. While moving the source code and issue tracking to Apache will take some time, we are excited to be joining the community at Apache. Stay tuned on this site for updates on how the project hosting will change.
-
@@ -758,7 +741,6 @@ Over 450 Spark developers and enthusiasts from 13 countries and more than 180 co
June 2, 2013
We’re happy to announce the release of Spark 0.7.2, a new maintenance release that includes several bug fixes and improvements, as well as new code examples and API features. We recommend that all users update to this release. Head over to the release notes to read about the new features, or download the release today.
-
@@ -774,7 +756,6 @@ Over 450 Spark developers and enthusiasts from 13 countries and more than 180 co
@@ -784,7 +765,6 @@ Over 450 Spark developers and enthusiasts from 13 countries and more than 180 co
March 17, 2013
At this year’s Strata conference, the AMP Lab hosted a full day of tutorials on Spark, Shark, and Spark Streaming, including online exercises on Amazon EC2. Those exercises are now available online, letting you learn Spark and Shark at your own pace on an EC2 cluster with real data. They are a great resource for learning the systems. You can also find slides from the Strata tutorials online, as well as videos from the AMP Camp workshop we held at Berkeley in August.
-
@@ -794,7 +774,6 @@ Over 450 Spark developers and enthusiasts from 13 countries and more than 180 co
February 27, 2013
We’re proud to announce the release of Spark 0.7.0, a new major version of Spark that adds several key features, including a Python API for Spark and an alpha of Spark Streaming. This release is the result of the largest group of contributors yet behind a Spark release – 31 contributors from inside and outside Berkeley. Head over to the release notes to read more about the new features, or download the release today.
-
@@ -804,7 +783,6 @@ Over 450 Spark developers and enthusiasts from 13 countries and more than 180 co
February 24, 2013
This weekend, Amazon posted an article and code that make it easy to launch Spark and Shark on Elastic MapReduce. The article includes examples of how to run both interactive Scala commands and SQL queries from Shark on data in S3. Head over to the Amazon article for details. We’re very excited because, to our knowledge, this makes Spark the first non-Hadoop engine that you can launch with EMR.
-
@@ -814,7 +792,6 @@ Over 450 Spark developers and enthusiasts from 13 countries and more than 180 co
February 7, 2013
We recently released Spark 0.6.2, a new version of Spark. This is a maintenance release that includes several bug fixes and usability improvements (see the release notes). We recommend that all users upgrade to this release.
-
@@ -829,7 +806,6 @@ Over 450 Spark developers and enthusiasts from 13 countries and more than 180 co
Thanks for sharing this, and looking forward to see others!
-
@@ -839,7 +815,6 @@ Over 450 Spark developers and enthusiasts from 13 countries and more than 180 co
December 21, 2012
On December 18th, we held the first of a series of Spark development meetups, for people interested in learning the Spark codebase and contributing to the project. There was quite a bit more demand than we anticipated, with over 80 people signing up and 64 attending. The first meetup was an introduction to Spark internals. Thanks to one of the attendees, there’s now a video of the meetup on YouTube. We’ve also posted the slides. Look to see more development meetups on Spark and Shark in the future.
-
@@ -858,8 +833,7 @@ Over 450 Spark developers and enthusiasts from 13 countries and more than 180 co
DataInformed interviewed two Spark users and wrote about their applications in anomaly detection, predictive analytics and data mining.
@@ -869,7 +843,6 @@ Over 450 Spark developers and enthusiasts from 13 countries and more than 180 co
November 22, 2012
Today we’ve made available two maintenance releases for Spark: 0.6.1 and 0.5.2. They both contain important bug fixes as well as some new features, such as the ability to build against Hadoop 2 distributions. We recommend that users update to the latest version for their branch; for new users, we recommend 0.6.1.
-
@@ -879,7 +852,6 @@ Over 450 Spark developers and enthusiasts from 13 countries and more than 180 co
October 15, 2012
Spark version 0.6.0 was released today, a major release that brings a wide range of performance improvements and new features, including a simpler standalone deploy mode and a Java API. Read more about it in the release notes.
-
@@ -889,7 +861,6 @@ Over 450 Spark developers and enthusiasts from 13 countries and more than 180 co
@@ -899,7 +870,6 @@ Over 450 Spark developers and enthusiasts from 13 countries and more than 180 co
January 10, 2012
We’ve started hosting a regular Bay Area Spark User Meetup. Sign up on the meetup.com page to be notified about events and meet other Spark developers and users.
We are happy to announce the availability of
Spark 0.9.1! Apache Spark 0.9.1 is a maintenance release with bug fixes, performance improvements, better stability with YARN and
improved parity of the Scala and Python API. We recommend all 0.9.0 users to upgrade to this stable release.
-Contributions to this release came from 37 developers.
+Contributions to this release came from 37 developers.
Visit the release notes
to read about the new features, or download the release today.
We are happy to announce the availability of
Spark 0.9.2! Apache Spark 0.9.2 is a maintenance release with bug fixes. We recommend all 0.9.x users to upgrade to this stable release.
-Contributions to this release came from 28 developers.
+Contributions to this release came from 28 developers.
Visit the release notes
to read about the new features, or download the release today.
We are happy to announce the availability of Spark 1.1.0! Spark 1.1.0 is the second release on the API-compatible 1.X line. It is Spark’s largest release ever, with contributions from 171 developers!
-
This release brings operational and performance improvements in Spark core including a new implementation of the Spark shuffle designed for very large scale workloads. Spark 1.1 adds significant extensions to the newest Spark modules, MLlib and Spark SQL. Spark SQL introduces a JDBC server, byte code generation for fast expression evaluation, a public types API, JSON support, and other features and optimizations. MLlib introduces a new statistics libary along with several new algorithms and optimizations. Spark 1.1 also builds out Spark’s Python support and adds new components to the Spark Streaming module.
+
This release brings operational and performance improvements in Spark core including a new implementation of the Spark shuffle designed for very large scale workloads. Spark 1.1 adds significant extensions to the newest Spark modules, MLlib and Spark SQL. Spark SQL introduces a JDBC server, byte code generation for fast expression evaluation, a public types API, JSON support, and other features and optimizations. MLlib introduces a new statistics libary along with several new algorithms and optimizations. Spark 1.1 also builds out Spark’s Python support and adds new components to the Spark Streaming module.
Visit the release notes to read about the new features, or download the release today.
We are happy to announce the availability of Spark 1.2.2 and Spark 1.3.1! These are both maintenance releases that collectively feature the work of more than 90 developers.
+
We are happy to announce the availability of Spark 1.2.2 and Spark 1.3.1! These are both maintenance releases that collectively feature the work of more than 90 developers.
To download either release, visit the downloads page.
The videos and slides for Spark Summit East 2015 are now all available online. Watch them to get the latest news from the Spark community as well as use cases and applications built on top.
+
The videos and slides for Spark Summit East 2015 are now all available online. Watch them to get the latest news from the Spark community as well as use cases and applications built on top.
If you like what you see, consider joining us at the 2015 Spark Summit in San Francisco.
Spark’s internal job scheduler has been refactored and extended to include more sophisticated scheduling policies. In particular, a fair scheduler implementation now allows multiple users to share an instance of Spark, which helps users running shorter jobs to achieve good performance, even when longer-running jobs are running in parallel. Support for topology-aware scheduling has been extended, including the ability to take into account rack locality and support for multiple executors on a single machine.
Easier Deployment and Linking
-
User programs can now link to Spark no matter which Hadoop version they need, without having to publish a version of spark-core specifically for that Hadoop version. An explanation of how to link against different Hadoop versions is provided here.
+
User programs can now link to Spark no matter which Hadoop version they need, without having to publish a version of spark-core specifically for that Hadoop version. An explanation of how to link against different Hadoop versions is provided here.
Expanded EC2 Capabilities
Spark’s EC2 scripts now support launching in any availability zone. Support has also been added for EC2 instance types which use the newer “HVM” architecture. This includes the cluster compute (cc1/cc2) family of instance types. We’ve also added support for running newer versions of HDFS alongside Spark. Finally, we’ve added the ability to launch clusters with maintenance releases of Spark in addition to launching the newest release.
Improved Documentation
-
This release adds documentation about cluster hardware provisioning and inter-operation with common Hadoop distributions. Docs are also included to cover the MLlib machine learning functions and new cluster monitoring features. Existing documentation has been updated to reflect changes in building and deploying Spark.
+
This release adds documentation about cluster hardware provisioning and inter-operation with common Hadoop distributions. Docs are also included to cover the MLlib machine learning functions and new cluster monitoring features. Existing documentation has been updated to reflect changes in building and deploying Spark.
Spark SQL adds a number of new features and performance improvements in this release. A JDBC/ODBC server allows users to connect to SparkSQL from many different applications and provides shared access to cached tables. A new module provides support for loading JSON data directly into Spark’s SchemaRDD format, including automatic schema inference. Spark SQL introduces dynamic bytecode generation in this release, a technique which significantly speeds up execution for queries that perform complex expression evaluation. This release also adds support for registering Python, Scala, and Java lambda functions as UDFs, which can then be called directly in SQL. Spark 1.1 adds a public types API to allow users to create SchemaRDD’s from custom data sources. Finally, many optimizations have been added to the native Parquet support as well as throughout the engine.
MLlib
-
MLlib adds several new algorithms and optimizations in this release. 1.1 introduces a new library of statistical packages which provides exploratory analytic functions. These include stratified sampling, correlations, chi-squared tests and support for creating random datasets. This release adds utilities for feature extraction (Word2Vec and TF-IDF) and feature transformation (normalization and standard scaling). Also new are support for nonnegative matrix factorization and SVD via Lanczos. The decision tree algorithm has been added in Python and Java. A tree aggregation primitive has been added to help optimize many existing algorithms. Performance improves across the board in MLlib 1.1, with improvements of around 2-3X for many algorithms and up to 5X for large scale decision tree problems.
+
MLlib adds several new algorithms and optimizations in this release. 1.1 introduces a new library of statistical packages which provides exploratory analytic functions. These include stratified sampling, correlations, chi-squared tests and support for creating random datasets. This release adds utilities for feature extraction (Word2Vec and TF-IDF) and feature transformation (normalization and standard scaling). Also new are support for nonnegative matrix factorization and SVD via Lanczos. The decision tree algorithm has been added in Python and Java. A tree aggregation primitive has been added to help optimize many existing algorithms. Performance improves across the board in MLlib 1.1, with improvements of around 2-3X for many algorithms and up to 5X for large scale decision tree problems.
The default value of spark.io.compression.codec is now snappy for improved memory usage. Old behavior can be restored by switching to lzf.
-
The default value of spark.broadcast.factory is now org.apache.spark.broadcast.TorrentBroadcastFactory for improved efficiency of broadcasts. Old behavior can be restored by switching to org.apache.spark.broadcast.HttpBroadcastFactory.
+
The default value of spark.broadcast.factory is now org.apache.spark.broadcast.TorrentBroadcastFactory for improved efficiency of broadcasts. Old behavior can be restored by switching to org.apache.spark.broadcast.HttpBroadcastFactory.
PySpark now performs external spilling during aggregations. Old behavior can be restored by setting spark.shuffle.spill to false.
PySpark uses a new heuristic for determining the parallelism of shuffle operations. Old behavior can be restored by setting spark.default.parallelism to the number of cores in the cluster.
@@ -275,7 +275,7 @@
Daneil Darabos – bug fixes and UI enhancements
Daoyuan Wang – SQL fixes
David Lemieux – bug fix
-
Davies Liu – PySpark fixes and spilling
+
Davies Liu – PySpark fixes and spilling
DB Tsai – online summaries in MLlib and other MLlib features
In 1.2 Spark core upgrades two major subsystems to improve the performance and stability of very large scale shuffles. The first is Spark’s communication manager used during bulk transfers, which upgrades to a netty-based implementation. The second is Spark’s shuffle mechanism, which upgrades to the “sort based” shuffle initially released in Spark 1.1. These both improve the performance and stability of very large scale shuffles. Spark also adds an elastic scaling mechanism designed to improve cluster utilization during long running ETL-style jobs. This is currently supported on YARN and will make its way to other cluster managers in future versions. Finally, Spark 1.2 adds support for Scala 2.11. For instructions on building for Scala 2.11 see the build documentation.
Spark Streaming
-
This release includes two major feature additions to Spark’s streaming library, a Python API and a write ahead log for full driver H/A. The Python API covers almost all the DStream transformations and output operations. Input sources based on text files and text over sockets are currently supported. Support for Kafka and Flume input streams in Python will be added in the next release. Second, Spark streaming now features H/A driver support through a write ahead log (WAL). In Spark 1.1 and earlier, some buffered (received but not yet processed) data can be lost during driver restarts. To prevent this Spark 1.2 adds an optional WAL, which buffers received data into a fault-tolerant file system (e.g. HDFS). See the streaming programming guide for more details.
+
This release includes two major feature additions to Spark’s streaming library, a Python API and a write ahead log for full driver H/A. The Python API covers almost all the DStream transformations and output operations. Input sources based on text files and text over sockets are currently supported. Support for Kafka and Flume input streams in Python will be added in the next release. Second, Spark streaming now features H/A driver support through a write ahead log (WAL). In Spark 1.1 and earlier, some buffered (received but not yet processed) data can be lost during driver restarts. To prevent this Spark 1.2 adds an optional WAL, which buffers received data into a fault-tolerant file system (e.g. HDFS). See the streaming programming guide for more details.
MLLib
Spark 1.2 previews a new set of machine learning API’s in a package called spark.ml that supports learning pipelines, where multiple algorithms are run in sequence with varying parameters. This type of pipeline is common in practical machine learning deployments. The new ML package uses Spark’s SchemaRDD to represent ML datasets, providing direct interoperability with Spark SQL. In addition to the new API, Spark 1.2 extends decision trees with two tree ensemble methods: random forests and gradient-boosted trees, among the most successful tree-based models for classification and regression. Finally, MLlib’s Python implementation receives a major update in 1.2 to simplify the process of adding Python APIs, along with better Python API coverage.
Spark 1.3 sees a handful of usability improvements in the core engine. The core API now supports multi level aggregation trees to help speed up expensive reduce operations. Improved error reporting has been added for certain gotcha operations. Spark’s Jetty dependency is now shaded to help avoid conflicts with user programs. Spark now supports SSL encryption for some communication endpoints. Finaly, realtime GC metrics and record counts have been added to the UI.
+
Spark 1.3 sees a handful of usability improvements in the core engine. The core API now supports multi level aggregation trees to help speed up expensive reduce operations. Improved error reporting has been added for certain gotcha operations. Spark’s Jetty dependency is now shaded to help avoid conflicts with user programs. Spark now supports SSL encryption for some communication endpoints. Finaly, realtime GC metrics and record counts have been added to the UI.
DataFrame API
Spark 1.3 adds a new DataFrames API that provides powerful and convenient operators when working with structured datasets. The DataFrame is an evolution of the base RDD API that includes named fields along with schema information. It’s easy to construct a DataFrame from sources such as Hive tables, JSON data, a JDBC database, or any implementation of Spark’s new data source API. Data frames will become a common interchange format between Spark components and when importing and exporting data to other systems. Data frames are supported in Python, Scala, and Java.
Spark 1.3 introduces a new direct Kafka API (docs) which enables exactly-once delivery without the use of write ahead logs. It also adds a Python Kafka API along with infrastructure for additional Python API’s in future releases. An online version of logistic regression and the ability to read binary records have also been added. For stateful operations, support has been added for loading of an initial state RDD. Finally, the streaming programming guide has been updated to include information about SQL and DataFrame operations within streaming applications, and important clarifications to the fault-tolerance semantics.
+
Spark 1.3 introduces a new direct Kafka API (docs) which enables exactly-once delivery without the use of write ahead logs. It also adds a Python Kafka API along with infrastructure for additional Python API’s in future releases. An online version of logistic regression and the ability to read binary records have also been added. For stateful operations, support has been added for loading of an initial state RDD. Finally, the streaming programming guide has been updated to include information about SQL and DataFrame operations within streaming applications, and important clarifications to the fault-tolerance semantics.
GraphX
GraphX adds a handful of utility functions in this release, including conversion into a canonical edge graph.
Unable to read parquet data generated by Spark 1.1.1 (SPARK-6315)
-
Parquet data source may use wrong Hadoop FileSystem (SPARK-6330)
+
Unable to read parquet data generated by Spark 1.1.1 (SPARK-6315)
+
Parquet data source may use wrong Hadoop FileSystem (SPARK-6330)
Spark Streaming
diff --git a/site/releases/spark-release-1-4-0.html b/site/releases/spark-release-1-4-0.html
index e6e1f0286..67f734c1c 100644
--- a/site/releases/spark-release-1-4-0.html
+++ b/site/releases/spark-release-1-4-0.html
@@ -250,7 +250,7 @@ Python coverage. MLlib also adds several new algorithms.
Spark Streaming
-
Spark streaming adds visual instrumentation graphs and significantly improved debugging information in the UI. It also enhances support for both Kafka and Kinesis.
+
Spark streaming adds visual instrumentation graphs and significantly improved debugging information in the UI. It also enhances support for both Kafka and Kinesis.
@@ -276,7 +276,7 @@ Python coverage. MLlib also adds several new algorithms.
Test Partners
-
Thanks to The following organizations, who helped benchmark or integration test release candidates: Intel, Palantir, Cloudera, Mesosphere, Huawei, Shopify, Netflix, Yahoo, UC Berkeley and Databricks.
+
Thanks to The following organizations, who helped benchmark or integration test release candidates: Intel, Palantir, Cloudera, Mesosphere, Huawei, Shopify, Netflix, Yahoo, UC Berkeley and Databricks.
SPARK-10000Unified Memory Management - Shared memory for execution and caching instead of exclusive division of the regions.
SPARK-11787Parquet Performance - Improve Parquet scan performance when using flat schemas.
-
SPARK-9241 Improved query planner for queries having distinct aggregations - Query plans of distinct aggregations are more robust when distinct columns have high cardinality.
+
SPARK-9241 Improved query planner for queries having distinct aggregations - Query plans of distinct aggregations are more robust when distinct columns have high cardinality.
SPARK-9858 Adaptive query execution - Initial support for automatically selecting the number of reducers for joins and aggregations.
SPARK-10978Avoiding double filters in Data Source API - When implementing a data source with filter pushdown, developers can now tell Spark SQL to avoid double evaluating a pushed-down filter.
SPARK-11111Fast null-safe joins - Joins using null-safe equality (<=>) will now execute using SortMergeJoin instead of computing a cartisian product.
@@ -233,7 +233,7 @@
Spark Streaming
-
API Updates
+
API Updates
SPARK-2629 New improved state management - mapWithState - a DStream transformation for stateful stream processing, supercedes updateStateByKey in functionality and performance.
SPARK-11198Kinesis record deaggregation - Kinesis streams have been upgraded to use KCL 1.4.0 and supports transparent deaggregation of KPL-aggregated records.
@@ -244,7 +244,7 @@
UI Improvements
Made failures visible in the streaming tab, in the timelines, batch list, and batch details page.
-
Made output operations visible in the streaming tab as progress bars.
+
Made output operations visible in the streaming tab as progress bars.
To download Apache Spark 2.0.0, visit the downloads page. You can consult JIRA for the detailed changes. We have curated a list of high level changes here, grouped by major modules.