We are happy to announce the availability of Spark 1.6.3! This maintenance release includes fixes across several areas of Spark.
+ +From b9aa4c3eefe4788fa97086ea87d92d8e3bfbc535 Mon Sep 17 00:00:00 2001
From: Reynold Xin In addition to the videos listed below, you can also view all slides from Bay Area meetups here. In addition to the videos listed below, you can also view all slides from Bay Area meetups here.
-
+ We are happy to announce the availability of Spark 1.6.3! This maintenance release includes fixes across several areas of Spark. There is one month left until Spark Summit 2015, which
will be held in San Francisco on June 15th to 17th.
@@ -383,7 +393,7 @@ The Summit will contain presen
Abstract submissions are now open for the first ever Spark Summit Europe. The event will take place on October 27th to 29th in Amsterdam. Submissions are welcome across a variety of Spark related topics, including use cases and ongoing development. The videos and slides for Spark Summit East 2015 are now all available online. Watch them to get the latest news from the Spark community as well as use cases and applications built on top. The videos and slides for Spark Summit East 2015 are now all available online. Watch them to get the latest news from the Spark community as well as use cases and applications built on top. We are happy to announce the availability of Spark 1.2.2 and Spark 1.3.1! These are both maintenance releases that collectively feature the work of more than 90 developers. We are happy to announce the availability of Spark 1.2.2 and Spark 1.3.1! These are both maintenance releases that collectively feature the work of more than 90 developers. We are happy to announce the availability of
Spark 0.9.2! Apache Spark 0.9.2 is a maintenance release with bug fixes. We recommend all 0.9.x users to upgrade to this stable release.
-Contributions to this release came from 28 developers. We are happy to announce the availability of
Spark 0.9.1! Apache Spark 0.9.1 is a maintenance release with bug fixes, performance improvements, better stability with YARN and
improved parity of the Scala and Python API. We recommend all 0.9.0 users to upgrade to this stable release.
-Contributions to this release came from 37 developers. In other news, there will be a full day of tutorials on Spark and Shark at the O’Reilly Strata conference in February. They include a three-hour introduction to Spark, Shark and BDAS Tuesday morning, and a three-hour hands-on exercise session. In other news, there will be a full day of tutorials on Spark and Shark at the O’Reilly Strata conference in February. They include a three-hour introduction to Spark, Shark and BDAS Tuesday morning, and a three-hour hands-on exercise session. We are happy to announce the availability of
Spark 0.9.1! Apache Spark 0.9.1 is a maintenance release with bug fixes, performance improvements, better stability with YARN and
improved parity of the Scala and Python API. We recommend all 0.9.0 users to upgrade to this stable release.
-Contributions to this release came from 37 developers.Latest News
+
diff --git a/site/documentation.html b/site/documentation.html
index ae706ec47..e4779567a 100644
--- a/site/documentation.html
+++ b/site/documentation.html
@@ -150,6 +150,9 @@
Latest News
+
@@ -190,6 +190,7 @@
Meetup Talk Videos
-
Latest News
+
diff --git a/site/examples.html b/site/examples.html
index 3f1e410b1..3e575a077 100644
--- a/site/examples.html
+++ b/site/examples.html
@@ -150,6 +150,9 @@
Latest News
+
diff --git a/site/faq.html b/site/faq.html
index 17b8ed7f7..c94f271f2 100644
--- a/site/faq.html
+++ b/site/faq.html
@@ -150,6 +150,9 @@
Latest News
+
diff --git a/site/graphx/index.html b/site/graphx/index.html
index 55fd33d2f..be3f9fce2 100644
--- a/site/graphx/index.html
+++ b/site/graphx/index.html
@@ -153,6 +153,9 @@
Latest News
+
diff --git a/site/index.html b/site/index.html
index dd24ec120..6709f6bd3 100644
--- a/site/index.html
+++ b/site/index.html
@@ -152,6 +152,9 @@
Latest News
+
diff --git a/site/js/downloads.js b/site/js/downloads.js
index 9e31b5068..77c263854 100644
--- a/site/js/downloads.js
+++ b/site/js/downloads.js
@@ -38,6 +38,7 @@ var packagesV7 = [hadoop2p7, hadoop2p6, hadoop2p4, hadoop2p3, hadoopFree, source
addRelease("2.0.1", new Date("10/03/2016"), packagesV7, true);
addRelease("2.0.0", new Date("07/26/2016"), packagesV7, true);
+addRelease("1.6.3", new Date("11/07/2016"), packagesV6, true);
addRelease("1.6.2", new Date("06/25/2016"), packagesV6, true);
addRelease("1.6.1", new Date("03/09/2016"), packagesV6, true);
addRelease("1.6.0", new Date("01/04/2016"), packagesV6, true);
@@ -195,7 +196,7 @@ function updateDownloadLink() {
if (pkg.toLowerCase().indexOf("mapr") > -1) {
link = "http://package.mapr.com/tools/apache-spark/$ver/$artifact"
} else if (download == "apache") {
- if (version < "1.6.2" ||
+ if (version < "1.6.3" ||
(version >= "2.0.0" && version < "2.0.1")) {
link = "http://archive.apache.org/dist/spark/spark-$ver/$artifact";
} else {
diff --git a/site/mailing-lists.html b/site/mailing-lists.html
index 74cf58c63..724a7eaf0 100644
--- a/site/mailing-lists.html
+++ b/site/mailing-lists.html
@@ -153,6 +153,9 @@
Latest News
+
diff --git a/site/mllib/index.html b/site/mllib/index.html
index 672909586..75c16bead 100644
--- a/site/mllib/index.html
+++ b/site/mllib/index.html
@@ -153,6 +153,9 @@
Latest News
+
diff --git a/site/news/amp-camp-2013-registration-ope.html b/site/news/amp-camp-2013-registration-ope.html
index ede67519f..27347e7f3 100644
--- a/site/news/amp-camp-2013-registration-ope.html
+++ b/site/news/amp-camp-2013-registration-ope.html
@@ -150,6 +150,9 @@
Latest News
+
diff --git a/site/news/announcing-the-first-spark-summit.html b/site/news/announcing-the-first-spark-summit.html
index 98aa7c101..cb8fa90a8 100644
--- a/site/news/announcing-the-first-spark-summit.html
+++ b/site/news/announcing-the-first-spark-summit.html
@@ -150,6 +150,9 @@
Latest News
+
diff --git a/site/news/fourth-spark-screencast-published.html b/site/news/fourth-spark-screencast-published.html
index 813dff3ab..c55341bca 100644
--- a/site/news/fourth-spark-screencast-published.html
+++ b/site/news/fourth-spark-screencast-published.html
@@ -150,6 +150,9 @@
Latest News
+
diff --git a/site/news/index.html b/site/news/index.html
index c75feb736..1b4d3fb7d 100644
--- a/site/news/index.html
+++ b/site/news/index.html
@@ -150,6 +150,9 @@
Latest News
+
@@ -185,6 +185,16 @@
Spark News
+Spark 1.6.3 released
+ Spark 2.0.1 released
@@ -371,7 +381,7 @@ With this release the Spark community continues to grow, with contributions from
One month to Spark Summit 2015 in San Francisco
- Announcing Spark Summit Europe
- Spark Summit East 2015 Videos Posted
Spark 1.2.2 and 1.3.1 released
Latest News
+
Latest News
+
Latest News
+
Latest News
+
diff --git a/site/news/run-spark-and-shark-on-amazon-emr.html b/site/news/run-spark-and-shark-on-amazon-emr.html
index 2e936e05f..d8a9a87ae 100644
--- a/site/news/run-spark-and-shark-on-amazon-emr.html
+++ b/site/news/run-spark-and-shark-on-amazon-emr.html
@@ -150,6 +150,9 @@
Latest News
+
diff --git a/site/news/spark-0-6-1-and-0-5-2-released.html b/site/news/spark-0-6-1-and-0-5-2-released.html
index 572ced7c3..c85d135c4 100644
--- a/site/news/spark-0-6-1-and-0-5-2-released.html
+++ b/site/news/spark-0-6-1-and-0-5-2-released.html
@@ -150,6 +150,9 @@
Latest News
+
diff --git a/site/news/spark-0-6-2-released.html b/site/news/spark-0-6-2-released.html
index 4814b40f8..ced84de4c 100644
--- a/site/news/spark-0-6-2-released.html
+++ b/site/news/spark-0-6-2-released.html
@@ -150,6 +150,9 @@
Latest News
+
diff --git a/site/news/spark-0-7-0-released.html b/site/news/spark-0-7-0-released.html
index cc67f77e4..97fd52d1f 100644
--- a/site/news/spark-0-7-0-released.html
+++ b/site/news/spark-0-7-0-released.html
@@ -150,6 +150,9 @@
Latest News
+
diff --git a/site/news/spark-0-7-2-released.html b/site/news/spark-0-7-2-released.html
index e6c783418..7da055a78 100644
--- a/site/news/spark-0-7-2-released.html
+++ b/site/news/spark-0-7-2-released.html
@@ -150,6 +150,9 @@
Latest News
+
diff --git a/site/news/spark-0-7-3-released.html b/site/news/spark-0-7-3-released.html
index 7e963df82..3e793c298 100644
--- a/site/news/spark-0-7-3-released.html
+++ b/site/news/spark-0-7-3-released.html
@@ -150,6 +150,9 @@
Latest News
+
diff --git a/site/news/spark-0-8-0-released.html b/site/news/spark-0-8-0-released.html
index c6bd17d96..8cbfed383 100644
--- a/site/news/spark-0-8-0-released.html
+++ b/site/news/spark-0-8-0-released.html
@@ -150,6 +150,9 @@
Latest News
+
diff --git a/site/news/spark-0-8-1-released.html b/site/news/spark-0-8-1-released.html
index b2bccb874..6e36603a4 100644
--- a/site/news/spark-0-8-1-released.html
+++ b/site/news/spark-0-8-1-released.html
@@ -150,6 +150,9 @@
Latest News
+
diff --git a/site/news/spark-0-9-0-released.html b/site/news/spark-0-9-0-released.html
index 120a06a27..f31323d9d 100644
--- a/site/news/spark-0-9-0-released.html
+++ b/site/news/spark-0-9-0-released.html
@@ -150,6 +150,9 @@
Latest News
+
diff --git a/site/news/spark-0-9-1-released.html b/site/news/spark-0-9-1-released.html
index efb2e66b5..f8f87b289 100644
--- a/site/news/spark-0-9-1-released.html
+++ b/site/news/spark-0-9-1-released.html
@@ -150,6 +150,9 @@
Latest News
+
@@ -189,7 +189,7 @@
Visit the release notes to read about the new features, or download the release today.
diff --git a/site/news/spark-0-9-2-released.html b/site/news/spark-0-9-2-released.html index 9c219d4a3..d4eab6b7d 100644 --- a/site/news/spark-0-9-2-released.html +++ b/site/news/spark-0-9-2-released.html @@ -150,6 +150,9 @@We are happy to announce the availability of Spark 0.9.2! Apache Spark 0.9.2 is a maintenance release with bug fixes. We recommend all 0.9.x users to upgrade to this stable release. -Contributions to this release came from 28 developers.
+Contributions to this release came from 28 developers.Visit the release notes to read about the new features, or download the release today.
diff --git a/site/news/spark-1-0-0-released.html b/site/news/spark-1-0-0-released.html index 1291c6f0e..b27817e94 100644 --- a/site/news/spark-1-0-0-released.html +++ b/site/news/spark-1-0-0-released.html @@ -150,6 +150,9 @@We are happy to announce the availability of Spark 1.1.0! Spark 1.1.0 is the second release on the API-compatible 1.X line. It is Spark’s largest release ever, with contributions from 171 developers!
-This release brings operational and performance improvements in Spark core including a new implementation of the Spark shuffle designed for very large scale workloads. Spark 1.1 adds significant extensions to the newest Spark modules, MLlib and Spark SQL. Spark SQL introduces a JDBC server, byte code generation for fast expression evaluation, a public types API, JSON support, and other features and optimizations. MLlib introduces a new statistics libary along with several new algorithms and optimizations. Spark 1.1 also builds out Spark’s Python support and adds new components to the Spark Streaming module.
+This release brings operational and performance improvements in Spark core including a new implementation of the Spark shuffle designed for very large scale workloads. Spark 1.1 adds significant extensions to the newest Spark modules, MLlib and Spark SQL. Spark SQL introduces a JDBC server, byte code generation for fast expression evaluation, a public types API, JSON support, and other features and optimizations. MLlib introduces a new statistics libary along with several new algorithms and optimizations. Spark 1.1 also builds out Spark’s Python support and adds new components to the Spark Streaming module.
Visit the release notes to read about the new features, or download the release today.
diff --git a/site/news/spark-1-1-1-released.html b/site/news/spark-1-1-1-released.html index 099ddc88d..9146d2cb4 100644 --- a/site/news/spark-1-1-1-released.html +++ b/site/news/spark-1-1-1-released.html @@ -150,6 +150,9 @@We are happy to announce the availability of Spark 1.2.2 and Spark 1.3.1! These are both maintenance releases that collectively feature the work of more than 90 developers.
+We are happy to announce the availability of Spark 1.2.2 and Spark 1.3.1! These are both maintenance releases that collectively feature the work of more than 90 developers.
To download either release, visit the downloads page.
diff --git a/site/news/spark-1-3-0-released.html b/site/news/spark-1-3-0-released.html index 567cb4c71..e750927ae 100644 --- a/site/news/spark-1-3-0-released.html +++ b/site/news/spark-1-3-0-released.html @@ -150,6 +150,9 @@+ + + Lightning-fast cluster computing + +
+ +We are happy to announce the availability of Spark 1.6.3! This maintenance release includes fixes across several areas of Spark.
+ +Visit the release notes to read about the new features, or download the release today.
+ + +
+
+Spark News Archive
+
In other news, there will be a full day of tutorials on Spark and Shark at the O’Reilly Strata conference in February. They include a three-hour introduction to Spark, Shark and BDAS Tuesday morning, and a three-hour hands-on exercise session.
+In other news, there will be a full day of tutorials on Spark and Shark at the O’Reilly Strata conference in February. They include a three-hour introduction to Spark, Shark and BDAS Tuesday morning, and a three-hour hands-on exercise session.
diff --git a/site/news/spark-becomes-tlp.html b/site/news/spark-becomes-tlp.html index f8c5c006d..a33f04572 100644 --- a/site/news/spark-becomes-tlp.html +++ b/site/news/spark-becomes-tlp.html @@ -150,6 +150,9 @@
The videos and slides for Spark Summit East 2015 are now all available online. Watch them to get the latest news from the Spark community as well as use cases and applications built on top.
+The videos and slides for Spark Summit East 2015 are now all available online. Watch them to get the latest news from the Spark community as well as use cases and applications built on top.
If you like what you see, consider joining us at the 2015 Spark Summit in San Francisco.
diff --git a/site/news/spark-summit-east-2016-cfp-closing.html b/site/news/spark-summit-east-2016-cfp-closing.html index d9d4e8f4c..1b84caace 100644 --- a/site/news/spark-summit-east-2016-cfp-closing.html +++ b/site/news/spark-summit-east-2016-cfp-closing.html @@ -150,6 +150,9 @@Spark’s internal job scheduler has been refactored and extended to include more sophisticated scheduling policies. In particular, a fair scheduler implementation now allows multiple users to share an instance of Spark, which helps users running shorter jobs to achieve good performance, even when longer-running jobs are running in parallel. Support for topology-aware scheduling has been extended, including the ability to take into account rack locality and support for multiple executors on a single machine.
User programs can now link to Spark no matter which Hadoop version they need, without having to publish a version of spark-core
specifically for that Hadoop version. An explanation of how to link against different Hadoop versions is provided here.
User programs can now link to Spark no matter which Hadoop version they need, without having to publish a version of spark-core
specifically for that Hadoop version. An explanation of how to link against different Hadoop versions is provided here.
Spark’s EC2 scripts now support launching in any availability zone. Support has also been added for EC2 instance types which use the newer “HVM” architecture. This includes the cluster compute (cc1/cc2) family of instance types. We’ve also added support for running newer versions of HDFS alongside Spark. Finally, we’ve added the ability to launch clusters with maintenance releases of Spark in addition to launching the newest release.
This release adds documentation about cluster hardware provisioning and inter-operation with common Hadoop distributions. Docs are also included to cover the MLlib machine learning functions and new cluster monitoring features. Existing documentation has been updated to reflect changes in building and deploying Spark.
+This release adds documentation about cluster hardware provisioning and inter-operation with common Hadoop distributions. Docs are also included to cover the MLlib machine learning functions and new cluster monitoring features. Existing documentation has been updated to reflect changes in building and deploying Spark.
Spark SQL adds a number of new features and performance improvements in this release. A JDBC/ODBC server allows users to connect to SparkSQL from many different applications and provides shared access to cached tables. A new module provides support for loading JSON data directly into Spark’s SchemaRDD format, including automatic schema inference. Spark SQL introduces dynamic bytecode generation in this release, a technique which significantly speeds up execution for queries that perform complex expression evaluation. This release also adds support for registering Python, Scala, and Java lambda functions as UDFs, which can then be called directly in SQL. Spark 1.1 adds a public types API to allow users to create SchemaRDD’s from custom data sources. Finally, many optimizations have been added to the native Parquet support as well as throughout the engine.
MLlib adds several new algorithms and optimizations in this release. 1.1 introduces a new library of statistical packages which provides exploratory analytic functions. These include stratified sampling, correlations, chi-squared tests and support for creating random datasets. This release adds utilities for feature extraction (Word2Vec and TF-IDF) and feature transformation (normalization and standard scaling). Also new are support for nonnegative matrix factorization and SVD via Lanczos. The decision tree algorithm has been added in Python and Java. A tree aggregation primitive has been added to help optimize many existing algorithms. Performance improves across the board in MLlib 1.1, with improvements of around 2-3X for many algorithms and up to 5X for large scale decision tree problems.
+MLlib adds several new algorithms and optimizations in this release. 1.1 introduces a new library of statistical packages which provides exploratory analytic functions. These include stratified sampling, correlations, chi-squared tests and support for creating random datasets. This release adds utilities for feature extraction (Word2Vec and TF-IDF) and feature transformation (normalization and standard scaling). Also new are support for nonnegative matrix factorization and SVD via Lanczos. The decision tree algorithm has been added in Python and Java. A tree aggregation primitive has been added to help optimize many existing algorithms. Performance improves across the board in MLlib 1.1, with improvements of around 2-3X for many algorithms and up to 5X for large scale decision tree problems.
Spark streaming adds a new data source Amazon Kinesis. For the Apache Flume, a new mode is supported which pulls data from Flume, simplifying deployment and providing high availability. The first of a set of streaming machine learning algorithms is introduced with streaming linear regression. Finally, rate limiting has been added for streaming inputs. GraphX adds custom storage levels for vertices and edges along with improved numerical precision across the board. Finally, GraphX adds a new label propagation algorithm.
@@ -215,7 +215,7 @@spark.io.compression.codec
is now snappy
for improved memory usage. Old behavior can be restored by switching to lzf
.spark.broadcast.factory
is now org.apache.spark.broadcast.TorrentBroadcastFactory
for improved efficiency of broadcasts. Old behavior can be restored by switching to org.apache.spark.broadcast.HttpBroadcastFactory
.spark.broadcast.factory
is now org.apache.spark.broadcast.TorrentBroadcastFactory
for improved efficiency of broadcasts. Old behavior can be restored by switching to org.apache.spark.broadcast.HttpBroadcastFactory
. spark.shuffle.spill
to false
.spark.default.parallelism
to the number of cores in the cluster.In 1.2 Spark core upgrades two major subsystems to improve the performance and stability of very large scale shuffles. The first is Spark’s communication manager used during bulk transfers, which upgrades to a netty-based implementation. The second is Spark’s shuffle mechanism, which upgrades to the “sort based” shuffle initially released in Spark 1.1. These both improve the performance and stability of very large scale shuffles. Spark also adds an elastic scaling mechanism designed to improve cluster utilization during long running ETL-style jobs. This is currently supported on YARN and will make its way to other cluster managers in future versions. Finally, Spark 1.2 adds support for Scala 2.11. For instructions on building for Scala 2.11 see the build documentation.
This release includes two major feature additions to Spark’s streaming library, a Python API and a write ahead log for full driver H/A. The Python API covers almost all the DStream transformations and output operations. Input sources based on text files and text over sockets are currently supported. Support for Kafka and Flume input streams in Python will be added in the next release. Second, Spark streaming now features H/A driver support through a write ahead log (WAL). In Spark 1.1 and earlier, some buffered (received but not yet processed) data can be lost during driver restarts. To prevent this Spark 1.2 adds an optional WAL, which buffers received data into a fault-tolerant file system (e.g. HDFS). See the streaming programming guide for more details.
+This release includes two major feature additions to Spark’s streaming library, a Python API and a write ahead log for full driver H/A. The Python API covers almost all the DStream transformations and output operations. Input sources based on text files and text over sockets are currently supported. Support for Kafka and Flume input streams in Python will be added in the next release. Second, Spark streaming now features H/A driver support through a write ahead log (WAL). In Spark 1.1 and earlier, some buffered (received but not yet processed) data can be lost during driver restarts. To prevent this Spark 1.2 adds an optional WAL, which buffers received data into a fault-tolerant file system (e.g. HDFS). See the streaming programming guide for more details.
Spark 1.2 previews a new set of machine learning API’s in a package called spark.ml that supports learning pipelines, where multiple algorithms are run in sequence with varying parameters. This type of pipeline is common in practical machine learning deployments. The new ML package uses Spark’s SchemaRDD to represent ML datasets, providing direct interoperability with Spark SQL. In addition to the new API, Spark 1.2 extends decision trees with two tree ensemble methods: random forests and gradient-boosted trees, among the most successful tree-based models for classification and regression. Finally, MLlib’s Python implementation receives a major update in 1.2 to simplify the process of adding Python APIs, along with better Python API coverage.
diff --git a/site/releases/spark-release-1-2-1.html b/site/releases/spark-release-1-2-1.html index 5581c5441..c9efc6a55 100644 --- a/site/releases/spark-release-1-2-1.html +++ b/site/releases/spark-release-1-2-1.html @@ -150,6 +150,9 @@To download Spark 1.3 visit the downloads page.
Spark 1.3 sees a handful of usability improvements in the core engine. The core API now supports multi level aggregation trees to help speed up expensive reduce operations. Improved error reporting has been added for certain gotcha operations. Spark’s Jetty dependency is now shaded to help avoid conflicts with user programs. Spark now supports SSL encryption for some communication endpoints. Finaly, realtime GC metrics and record counts have been added to the UI.
+Spark 1.3 sees a handful of usability improvements in the core engine. The core API now supports multi level aggregation trees to help speed up expensive reduce operations. Improved error reporting has been added for certain gotcha operations. Spark’s Jetty dependency is now shaded to help avoid conflicts with user programs. Spark now supports SSL encryption for some communication endpoints. Finaly, realtime GC metrics and record counts have been added to the UI.
Spark 1.3 adds a new DataFrames API that provides powerful and convenient operators when working with structured datasets. The DataFrame is an evolution of the base RDD API that includes named fields along with schema information. It’s easy to construct a DataFrame from sources such as Hive tables, JSON data, a JDBC database, or any implementation of Spark’s new data source API. Data frames will become a common interchange format between Spark components and when importing and exporting data to other systems. Data frames are supported in Python, Scala, and Java.
@@ -203,7 +203,7 @@In this release Spark MLlib introduces several new algorithms: latent Dirichlet allocation (LDA) for topic modeling, multinomial logistic regression for multiclass classification, Gaussian mixture model (GMM) and power iteration clustering for clustering, FP-growth for frequent pattern mining, and block matrix abstraction for distributed linear algebra. Initial support has been added for model import/export in exchangeable format, which will be expanded in future versions to cover more model types in Java/Python/Scala. The implementations of k-means and ALS receive updates that lead to significant performance gain. PySpark now supports the ML pipeline API added in Spark 1.2, and gradient boosted trees and Gaussian mixture model. Finally, the ML pipeline API has been ported to support the new DataFrames abstraction.
Spark 1.3 introduces a new direct Kafka API (docs) which enables exactly-once delivery without the use of write ahead logs. It also adds a Python Kafka API along with infrastructure for additional Python API’s in future releases. An online version of logistic regression and the ability to read binary records have also been added. For stateful operations, support has been added for loading of an initial state RDD. Finally, the streaming programming guide has been updated to include information about SQL and DataFrame operations within streaming applications, and important clarifications to the fault-tolerance semantics.
+Spark 1.3 introduces a new direct Kafka API (docs) which enables exactly-once delivery without the use of write ahead logs. It also adds a Python Kafka API along with infrastructure for additional Python API’s in future releases. An online version of logistic regression and the ability to read binary records have also been added. For stateful operations, support has been added for loading of an initial state RDD. Finally, the streaming programming guide has been updated to include information about SQL and DataFrame operations within streaming applications, and important clarifications to the fault-tolerance semantics.
GraphX adds a handful of utility functions in this release, including conversion into a canonical edge graph.
@@ -219,7 +219,7 @@collect()
.Spark streaming adds visual instrumentation graphs and significantly improved debugging information in the UI. It also enhances support for both Kafka and Kinesis.
+Spark streaming adds visual instrumentation graphs and significantly improved debugging information in the UI. It also enhances support for both Kafka and Kinesis.
Thanks to The following organizations, who helped benchmark or integration test release candidates:
Intel, Palantir, Cloudera, Mesosphere, Huawei, Shopify, Netflix, Yahoo, UC Berkeley and Databricks.
Thanks to The following organizations, who helped benchmark or integration test release candidates:
Intel, Palantir, Cloudera, Mesosphere, Huawei, Shopify, Netflix, Yahoo, UC Berkeley and Databricks.
You can consult JIRA for the detailed changes. We have curated a list of high level changes here:
You can consult JIRA for the detailed changes. We have curated a list of high level changes here:
<=>
) will now execute using SortMergeJoin instead of computing a cartisian product.mapWithState
- a DStream transformation for stateful stream processing, supercedes updateStateByKey
in functionality and performance.+ + + Lightning-fast cluster computing + +
+ +Spark 1.6.3 is a maintenance release containing stability fixes. This release is based on the branch-1.6 maintenance branch of Spark. We strongly recommend all 1.6.x users to upgrade to this stable release.
+ +You can consult JIRA for the detailed changes.
+ +We would like to acknowledge all community members for contributing patches to this release.
+ + +
+
+Spark News Archive
+
To download Apache Spark 2.0.0, visit the downloads page. You can consult JIRA for the detailed changes. We have curated a list of high level changes here, grouped by major modules.