From a8dce9912f8dacaffba91155b8673e6c700e6c17 Mon Sep 17 00:00:00 2001 From: Reynold Xin Date: Mon, 3 Oct 2016 12:08:10 -0700 Subject: Add Spark 2.0.1 release. --- downloads.md | 10 +- js/downloads.js | 1 + news/_posts/2016-10-03-spark-2-0-1-released.md | 14 ++ releases/_posts/2016-10-03-spark-release-2-0-1.md | 18 ++ site/community.html | 6 +- site/documentation.html | 11 +- site/downloads.html | 16 +- site/examples.html | 6 +- site/faq.html | 6 +- site/graphx/index.html | 6 +- site/index.html | 6 +- site/js/downloads.js | 1 + site/mailing-lists.html | 6 +- site/mllib/index.html | 6 +- site/news/amp-camp-2013-registration-ope.html | 6 +- site/news/announcing-the-first-spark-summit.html | 6 +- site/news/fourth-spark-screencast-published.html | 6 +- site/news/index.html | 29 ++- site/news/nsdi-paper.html | 6 +- site/news/one-month-to-spark-summit-2015.html | 6 +- .../news/proposals-open-for-spark-summit-east.html | 6 +- .../registration-open-for-spark-summit-east.html | 6 +- site/news/run-spark-and-shark-on-amazon-emr.html | 6 +- site/news/spark-0-6-1-and-0-5-2-released.html | 6 +- site/news/spark-0-6-2-released.html | 6 +- site/news/spark-0-7-0-released.html | 6 +- site/news/spark-0-7-2-released.html | 6 +- site/news/spark-0-7-3-released.html | 6 +- site/news/spark-0-8-0-released.html | 6 +- site/news/spark-0-8-1-released.html | 6 +- site/news/spark-0-9-0-released.html | 6 +- site/news/spark-0-9-1-released.html | 8 +- site/news/spark-0-9-2-released.html | 8 +- site/news/spark-1-0-0-released.html | 6 +- site/news/spark-1-0-1-released.html | 6 +- site/news/spark-1-0-2-released.html | 6 +- site/news/spark-1-1-0-released.html | 8 +- site/news/spark-1-1-1-released.html | 6 +- site/news/spark-1-2-0-released.html | 6 +- site/news/spark-1-2-1-released.html | 6 +- site/news/spark-1-2-2-released.html | 8 +- site/news/spark-1-3-0-released.html | 6 +- site/news/spark-1-4-0-released.html | 6 +- site/news/spark-1-4-1-released.html | 6 +- site/news/spark-1-5-0-released.html | 6 +- site/news/spark-1-5-1-released.html | 6 +- site/news/spark-1-5-2-released.html | 6 +- site/news/spark-1-6-0-released.html | 6 +- site/news/spark-1-6-1-released.html | 6 +- site/news/spark-1-6-2-released.html | 6 +- site/news/spark-2-0-0-released.html | 6 +- site/news/spark-2-0-1-released.html | 211 ++++++++++++++++++++ site/news/spark-2.0.0-preview.html | 6 +- .../news/spark-accepted-into-apache-incubator.html | 6 +- site/news/spark-and-shark-in-the-news.html | 8 +- site/news/spark-becomes-tlp.html | 6 +- site/news/spark-featured-in-wired.html | 6 +- .../news/spark-mailing-lists-moving-to-apache.html | 6 +- site/news/spark-meetups.html | 6 +- site/news/spark-screencasts-published.html | 6 +- site/news/spark-summit-2013-is-a-wrap.html | 6 +- site/news/spark-summit-2014-videos-posted.html | 6 +- site/news/spark-summit-2015-videos-posted.html | 6 +- site/news/spark-summit-agenda-posted.html | 6 +- .../news/spark-summit-east-2015-videos-posted.html | 8 +- site/news/spark-summit-east-2016-cfp-closing.html | 6 +- site/news/spark-summit-east-agenda-posted.html | 6 +- site/news/spark-summit-europe-agenda-posted.html | 6 +- site/news/spark-summit-europe.html | 6 +- .../news/spark-summit-june-2016-agenda-posted.html | 6 +- site/news/spark-tips-from-quantifind.html | 6 +- .../spark-user-survey-and-powered-by-page.html | 6 +- site/news/spark-version-0-6-0-released.html | 6 +- ...ark-wins-daytona-gray-sort-100tb-benchmark.html | 6 +- .../strata-exercises-now-available-online.html | 6 +- site/news/submit-talks-to-spark-summit-2014.html | 6 +- site/news/submit-talks-to-spark-summit-2016.html | 6 +- .../submit-talks-to-spark-summit-east-2016.html | 6 +- .../news/submit-talks-to-spark-summit-eu-2016.html | 6 +- site/news/two-weeks-to-spark-summit-2014.html | 6 +- .../video-from-first-spark-development-meetup.html | 6 +- site/releases/spark-release-0-3.html | 6 +- site/releases/spark-release-0-5-0.html | 6 +- site/releases/spark-release-0-5-1.html | 6 +- site/releases/spark-release-0-5-2.html | 6 +- site/releases/spark-release-0-6-0.html | 6 +- site/releases/spark-release-0-6-1.html | 6 +- site/releases/spark-release-0-6-2.html | 6 +- site/releases/spark-release-0-7-0.html | 6 +- site/releases/spark-release-0-7-2.html | 6 +- site/releases/spark-release-0-7-3.html | 6 +- site/releases/spark-release-0-8-0.html | 10 +- site/releases/spark-release-0-8-1.html | 6 +- site/releases/spark-release-0-9-0.html | 6 +- site/releases/spark-release-0-9-1.html | 26 +-- site/releases/spark-release-0-9-2.html | 6 +- site/releases/spark-release-1-0-0.html | 6 +- site/releases/spark-release-1-0-1.html | 14 +- site/releases/spark-release-1-0-2.html | 8 +- site/releases/spark-release-1-1-0.html | 12 +- site/releases/spark-release-1-1-1.html | 6 +- site/releases/spark-release-1-2-0.html | 8 +- site/releases/spark-release-1-2-1.html | 6 +- site/releases/spark-release-1-2-2.html | 6 +- site/releases/spark-release-1-3-0.html | 12 +- site/releases/spark-release-1-3-1.html | 12 +- site/releases/spark-release-1-4-0.html | 10 +- site/releases/spark-release-1-4-1.html | 6 +- site/releases/spark-release-1-5-0.html | 36 ++-- site/releases/spark-release-1-5-1.html | 6 +- site/releases/spark-release-1-5-2.html | 6 +- site/releases/spark-release-1-6-0.html | 26 +-- site/releases/spark-release-1-6-1.html | 6 +- site/releases/spark-release-1-6-2.html | 6 +- site/releases/spark-release-2-0-0.html | 42 ++-- site/releases/spark-release-2-0-1.html | 215 +++++++++++++++++++++ site/research.html | 6 +- site/screencasts/1-first-steps-with-spark.html | 6 +- .../2-spark-documentation-overview.html | 6 +- .../screencasts/3-transformations-and-caching.html | 6 +- site/screencasts/4-a-standalone-job-in-spark.html | 6 +- site/screencasts/index.html | 6 +- site/sql/index.html | 6 +- site/streaming/index.html | 6 +- site/trademarks.html | 6 +- 125 files changed, 920 insertions(+), 452 deletions(-) create mode 100644 news/_posts/2016-10-03-spark-2-0-1-released.md create mode 100644 releases/_posts/2016-10-03-spark-release-2-0-1.md create mode 100644 site/news/spark-2-0-1-released.html create mode 100644 site/releases/spark-release-2-0-1.html diff --git a/downloads.md b/downloads.md index da21cb528..e8fc30207 100644 --- a/downloads.md +++ b/downloads.md @@ -16,9 +16,9 @@ $(document).ready(function() { ## Download Apache Spark™ -Our latest stable version is Apache Spark 2.0.0, released on July 26, 2016 -(release notes) -(git tag)
+Our latest stable version is Apache Spark 2.0.1, released on Oct 3, 2016 +(release notes) +(git tag)
1. Choose a Spark release:
@@ -55,7 +55,7 @@ Spark artifacts are [hosted in Maven Central](http://search.maven.org/#search%7C groupId: org.apache.spark artifactId: spark-core_2.11 - version: 2.0.0 + version: 2.0.1 ### Spark Source Code Management If you are interested in working with the newest under-development code or contributing to Apache Spark development, you can also check out the master branch from Git: @@ -63,7 +63,7 @@ If you are interested in working with the newest under-development code or contr # Master development branch git clone git://github.com/apache/spark.git - # 2.0 maintenance branch with stability fixes on top of Spark 2.0.0 + # 2.0 maintenance branch with stability fixes on top of Spark 2.0.1 git clone git://github.com/apache/spark.git -b branch-2.0 Once you've downloaded Spark, you can find instructions for installing and building it on the documentation page. diff --git a/js/downloads.js b/js/downloads.js index bdf2cf08e..e04352fd1 100644 --- a/js/downloads.js +++ b/js/downloads.js @@ -36,6 +36,7 @@ var packagesV7 = [hadoop2p7, hadoop2p6, hadoop2p4, hadoop2p3, hadoopFree, source // addRelease("2.0.0-preview", new Date("05/24/2016"), sources.concat(packagesV7), true, false); +addRelease("2.0.1", new Date("10/03/2016"), packagesV7, true, true); addRelease("2.0.0", new Date("07/26/2016"), packagesV7, true, true); addRelease("1.6.2", new Date("06/25/2016"), packagesV6, true, true); addRelease("1.6.1", new Date("03/09/2016"), packagesV6, true, true); diff --git a/news/_posts/2016-10-03-spark-2-0-1-released.md b/news/_posts/2016-10-03-spark-2-0-1-released.md new file mode 100644 index 000000000..b13fb182d --- /dev/null +++ b/news/_posts/2016-10-03-spark-2-0-1-released.md @@ -0,0 +1,14 @@ +--- +layout: post +title: Spark 2.0.1 released +categories: +- News +tags: [] +status: publish +type: post +published: true +meta: + _edit_last: '4' + _wpas_done_all: '1' +--- +We are happy to announce the availability of Apache Spark 2.0.1! Visit the release notes to read about the new features, or download the release today. diff --git a/releases/_posts/2016-10-03-spark-release-2-0-1.md b/releases/_posts/2016-10-03-spark-release-2-0-1.md new file mode 100644 index 000000000..53a61b35a --- /dev/null +++ b/releases/_posts/2016-10-03-spark-release-2-0-1.md @@ -0,0 +1,18 @@ +--- +layout: post +title: Spark Release 2.0.1 +categories: [] +tags: [] +status: publish +type: post +published: true +meta: + _edit_last: '4' + _wpas_done_all: '1' +--- + +Apache Spark 2.0.1 is a maintenance release containing 300 stability and bug fixes. This release is based on the branch-2.0 maintenance branch of Spark. We strongly recommend all 2.0.0 users to upgrade to this stable release. + +To download Apache Spark 2.0.1, visit the [downloads](http://spark.apache.org/downloads.html) page. You can consult JIRA for the [detailed changes](https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315420&version=12336857). + +We would like to acknowledge all community members for contributing patches to this release. diff --git a/site/community.html b/site/community.html index 90390b872..521af8312 100644 --- a/site/community.html +++ b/site/community.html @@ -150,6 +150,9 @@
Latest News

Archive

diff --git a/site/documentation.html b/site/documentation.html index 33113fb90..b64820d78 100644 --- a/site/documentation.html +++ b/site/documentation.html @@ -150,6 +150,9 @@
Latest News

Archive

@@ -253,13 +253,12 @@

Meetup Talk Videos

-

In addition to the videos listed below, you can also view all slides from Bay Area meetups here.

+

In addition to the videos listed below, you can also view all slides from Bay Area meetups here. - +

-

In other news, there will be a full day of tutorials on Spark and Shark at the O’Reilly Strata conference in February. They include a three-hour introduction to Spark, Shark and BDAS Tuesday morning, and a three-hour hands-on exercise session.

+

In other news, there will be a full day of tutorials on Spark and Shark at the O’Reilly Strata conference in February. They include a three-hour introduction to Spark, Shark and BDAS Tuesday morning, and a three-hour hands-on exercise session.

diff --git a/site/news/nsdi-paper.html b/site/news/nsdi-paper.html index 9807b70a1..2c9eca17a 100644 --- a/site/news/nsdi-paper.html +++ b/site/news/nsdi-paper.html @@ -150,6 +150,9 @@
Latest News

Archive

diff --git a/site/news/one-month-to-spark-summit-2015.html b/site/news/one-month-to-spark-summit-2015.html index bf3170907..9b1a277d5 100644 --- a/site/news/one-month-to-spark-summit-2015.html +++ b/site/news/one-month-to-spark-summit-2015.html @@ -150,6 +150,9 @@
Latest News

Archive

diff --git a/site/news/proposals-open-for-spark-summit-east.html b/site/news/proposals-open-for-spark-summit-east.html index 2d5665e41..e24dd889b 100644 --- a/site/news/proposals-open-for-spark-summit-east.html +++ b/site/news/proposals-open-for-spark-summit-east.html @@ -150,6 +150,9 @@
Latest News

Archive

diff --git a/site/news/registration-open-for-spark-summit-east.html b/site/news/registration-open-for-spark-summit-east.html index 048f133fd..f89f05e12 100644 --- a/site/news/registration-open-for-spark-summit-east.html +++ b/site/news/registration-open-for-spark-summit-east.html @@ -150,6 +150,9 @@
Latest News

Archive

diff --git a/site/news/run-spark-and-shark-on-amazon-emr.html b/site/news/run-spark-and-shark-on-amazon-emr.html index 8b44b3dcd..9a2a07303 100644 --- a/site/news/run-spark-and-shark-on-amazon-emr.html +++ b/site/news/run-spark-and-shark-on-amazon-emr.html @@ -150,6 +150,9 @@
Latest News

Archive

diff --git a/site/news/spark-0-6-1-and-0-5-2-released.html b/site/news/spark-0-6-1-and-0-5-2-released.html index 906676ba3..fa2785289 100644 --- a/site/news/spark-0-6-1-and-0-5-2-released.html +++ b/site/news/spark-0-6-1-and-0-5-2-released.html @@ -150,6 +150,9 @@
Latest News

Archive

diff --git a/site/news/spark-0-6-2-released.html b/site/news/spark-0-6-2-released.html index 46472e393..686c29feb 100644 --- a/site/news/spark-0-6-2-released.html +++ b/site/news/spark-0-6-2-released.html @@ -150,6 +150,9 @@
Latest News

Archive

diff --git a/site/news/spark-0-7-0-released.html b/site/news/spark-0-7-0-released.html index a00b0e7e7..18b236618 100644 --- a/site/news/spark-0-7-0-released.html +++ b/site/news/spark-0-7-0-released.html @@ -150,6 +150,9 @@
Latest News

Archive

diff --git a/site/news/spark-0-7-2-released.html b/site/news/spark-0-7-2-released.html index 803faf93a..a6ffb15d3 100644 --- a/site/news/spark-0-7-2-released.html +++ b/site/news/spark-0-7-2-released.html @@ -150,6 +150,9 @@
Latest News

Archive

diff --git a/site/news/spark-0-7-3-released.html b/site/news/spark-0-7-3-released.html index 769f4c1f6..01eaa6fd7 100644 --- a/site/news/spark-0-7-3-released.html +++ b/site/news/spark-0-7-3-released.html @@ -150,6 +150,9 @@
Latest News

Archive

diff --git a/site/news/spark-0-8-0-released.html b/site/news/spark-0-8-0-released.html index 658fabee8..1ee261e55 100644 --- a/site/news/spark-0-8-0-released.html +++ b/site/news/spark-0-8-0-released.html @@ -150,6 +150,9 @@
Latest News

Archive

diff --git a/site/news/spark-0-8-1-released.html b/site/news/spark-0-8-1-released.html index ec7230377..292e03582 100644 --- a/site/news/spark-0-8-1-released.html +++ b/site/news/spark-0-8-1-released.html @@ -150,6 +150,9 @@
Latest News

Archive

diff --git a/site/news/spark-0-9-0-released.html b/site/news/spark-0-9-0-released.html index 2d930b531..5aaba7eb9 100644 --- a/site/news/spark-0-9-0-released.html +++ b/site/news/spark-0-9-0-released.html @@ -150,6 +150,9 @@
Latest News

Archive

diff --git a/site/news/spark-0-9-1-released.html b/site/news/spark-0-9-1-released.html index f35e414dc..43873713e 100644 --- a/site/news/spark-0-9-1-released.html +++ b/site/news/spark-0-9-1-released.html @@ -150,6 +150,9 @@
Latest News

Archive

@@ -189,7 +189,7 @@

We are happy to announce the availability of Spark 0.9.1! Apache Spark 0.9.1 is a maintenance release with bug fixes, performance improvements, better stability with YARN and improved parity of the Scala and Python API. We recommend all 0.9.0 users to upgrade to this stable release. -Contributions to this release came from 37 developers.

+Contributions to this release came from 37 developers.

Visit the release notes to read about the new features, or download the release today.

diff --git a/site/news/spark-0-9-2-released.html b/site/news/spark-0-9-2-released.html index b5fc38d9e..0f971aa83 100644 --- a/site/news/spark-0-9-2-released.html +++ b/site/news/spark-0-9-2-released.html @@ -150,6 +150,9 @@
Latest News

Archive

@@ -188,7 +188,7 @@

We are happy to announce the availability of Spark 0.9.2! Apache Spark 0.9.2 is a maintenance release with bug fixes. We recommend all 0.9.x users to upgrade to this stable release. -Contributions to this release came from 28 developers.

+Contributions to this release came from 28 developers.

Visit the release notes to read about the new features, or download the release today.

diff --git a/site/news/spark-1-0-0-released.html b/site/news/spark-1-0-0-released.html index 32864c7ef..e44fb4264 100644 --- a/site/news/spark-1-0-0-released.html +++ b/site/news/spark-1-0-0-released.html @@ -150,6 +150,9 @@
Latest News

Archive

diff --git a/site/news/spark-1-0-1-released.html b/site/news/spark-1-0-1-released.html index ca2c6b273..3e958a669 100644 --- a/site/news/spark-1-0-1-released.html +++ b/site/news/spark-1-0-1-released.html @@ -150,6 +150,9 @@
Latest News

Archive

diff --git a/site/news/spark-1-0-2-released.html b/site/news/spark-1-0-2-released.html index 797fadc20..ea59225b8 100644 --- a/site/news/spark-1-0-2-released.html +++ b/site/news/spark-1-0-2-released.html @@ -150,6 +150,9 @@
Latest News

Archive

diff --git a/site/news/spark-1-1-0-released.html b/site/news/spark-1-1-0-released.html index fcfa8ab2a..c54940a9a 100644 --- a/site/news/spark-1-1-0-released.html +++ b/site/news/spark-1-1-0-released.html @@ -150,6 +150,9 @@
Latest News

Archive

@@ -188,7 +188,7 @@

We are happy to announce the availability of Spark 1.1.0! Spark 1.1.0 is the second release on the API-compatible 1.X line. It is Spark’s largest release ever, with contributions from 171 developers!

-

This release brings operational and performance improvements in Spark core including a new implementation of the Spark shuffle designed for very large scale workloads. Spark 1.1 adds significant extensions to the newest Spark modules, MLlib and Spark SQL. Spark SQL introduces a JDBC server, byte code generation for fast expression evaluation, a public types API, JSON support, and other features and optimizations. MLlib introduces a new statistics libary along with several new algorithms and optimizations. Spark 1.1 also builds out Spark’s Python support and adds new components to the Spark Streaming module.

+

This release brings operational and performance improvements in Spark core including a new implementation of the Spark shuffle designed for very large scale workloads. Spark 1.1 adds significant extensions to the newest Spark modules, MLlib and Spark SQL. Spark SQL introduces a JDBC server, byte code generation for fast expression evaluation, a public types API, JSON support, and other features and optimizations. MLlib introduces a new statistics libary along with several new algorithms and optimizations. Spark 1.1 also builds out Spark’s Python support and adds new components to the Spark Streaming module.

Visit the release notes to read about the new features, or download the release today.

diff --git a/site/news/spark-1-1-1-released.html b/site/news/spark-1-1-1-released.html index 29fd8ca2d..f51cd26ae 100644 --- a/site/news/spark-1-1-1-released.html +++ b/site/news/spark-1-1-1-released.html @@ -150,6 +150,9 @@
Latest News

Archive

diff --git a/site/news/spark-1-2-0-released.html b/site/news/spark-1-2-0-released.html index 4db9a9ca1..6f16882f0 100644 --- a/site/news/spark-1-2-0-released.html +++ b/site/news/spark-1-2-0-released.html @@ -150,6 +150,9 @@
Latest News

Archive

diff --git a/site/news/spark-1-2-1-released.html b/site/news/spark-1-2-1-released.html index 0e26643e4..72cdfd6d2 100644 --- a/site/news/spark-1-2-1-released.html +++ b/site/news/spark-1-2-1-released.html @@ -150,6 +150,9 @@
Latest News

Archive

diff --git a/site/news/spark-1-2-2-released.html b/site/news/spark-1-2-2-released.html index 29502a938..29f512e19 100644 --- a/site/news/spark-1-2-2-released.html +++ b/site/news/spark-1-2-2-released.html @@ -150,6 +150,9 @@
Latest News

Archive

@@ -186,7 +186,7 @@

Spark 1.2.2 and 1.3.1 released

-

We are happy to announce the availability of Spark 1.2.2 and Spark 1.3.1! These are both maintenance releases that collectively feature the work of more than 90 developers.

+

We are happy to announce the availability of Spark 1.2.2 and Spark 1.3.1! These are both maintenance releases that collectively feature the work of more than 90 developers.

To download either release, visit the downloads page.

diff --git a/site/news/spark-1-3-0-released.html b/site/news/spark-1-3-0-released.html index d9173de67..8700a644d 100644 --- a/site/news/spark-1-3-0-released.html +++ b/site/news/spark-1-3-0-released.html @@ -150,6 +150,9 @@
Latest News

Archive

diff --git a/site/news/spark-1-4-0-released.html b/site/news/spark-1-4-0-released.html index 8464cebb6..436b2b93d 100644 --- a/site/news/spark-1-4-0-released.html +++ b/site/news/spark-1-4-0-released.html @@ -150,6 +150,9 @@
Latest News

Archive

diff --git a/site/news/spark-1-4-1-released.html b/site/news/spark-1-4-1-released.html index c5084324b..a1e640f87 100644 --- a/site/news/spark-1-4-1-released.html +++ b/site/news/spark-1-4-1-released.html @@ -150,6 +150,9 @@
Latest News

Archive

diff --git a/site/news/spark-1-5-0-released.html b/site/news/spark-1-5-0-released.html index 4526f0383..86e5fca44 100644 --- a/site/news/spark-1-5-0-released.html +++ b/site/news/spark-1-5-0-released.html @@ -150,6 +150,9 @@
Latest News

Archive

diff --git a/site/news/spark-1-5-1-released.html b/site/news/spark-1-5-1-released.html index f0cd893b8..6c9b880d6 100644 --- a/site/news/spark-1-5-1-released.html +++ b/site/news/spark-1-5-1-released.html @@ -150,6 +150,9 @@
Latest News

Archive

diff --git a/site/news/spark-1-5-2-released.html b/site/news/spark-1-5-2-released.html index debe6e066..858cf47df 100644 --- a/site/news/spark-1-5-2-released.html +++ b/site/news/spark-1-5-2-released.html @@ -150,6 +150,9 @@
Latest News

Archive

diff --git a/site/news/spark-1-6-0-released.html b/site/news/spark-1-6-0-released.html index ab26251be..3dd5c77ab 100644 --- a/site/news/spark-1-6-0-released.html +++ b/site/news/spark-1-6-0-released.html @@ -150,6 +150,9 @@
Latest News

Archive

diff --git a/site/news/spark-1-6-1-released.html b/site/news/spark-1-6-1-released.html index 3e73bfe4f..dc48c3c85 100644 --- a/site/news/spark-1-6-1-released.html +++ b/site/news/spark-1-6-1-released.html @@ -150,6 +150,9 @@
Latest News

Archive

diff --git a/site/news/spark-1-6-2-released.html b/site/news/spark-1-6-2-released.html index 1c66b19c9..adaf08970 100644 --- a/site/news/spark-1-6-2-released.html +++ b/site/news/spark-1-6-2-released.html @@ -150,6 +150,9 @@
Latest News

Archive

diff --git a/site/news/spark-2-0-0-released.html b/site/news/spark-2-0-0-released.html index 4b8d1e689..7e23dd079 100644 --- a/site/news/spark-2-0-0-released.html +++ b/site/news/spark-2-0-0-released.html @@ -150,6 +150,9 @@
Latest News

Archive

diff --git a/site/news/spark-2-0-1-released.html b/site/news/spark-2-0-1-released.html new file mode 100644 index 000000000..91847cea0 --- /dev/null +++ b/site/news/spark-2-0-1-released.html @@ -0,0 +1,211 @@ + + + + + + + + + Spark 2.0.1 released | Apache Spark + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ +
+ +

+ + + Lightning-fast cluster computing + +

+ +
+ + + + +
+
+
+
Latest News
+ +

Archive

+
+ +
+ +
+

Spark 2.0.1 released

+ + +

We are happy to announce the availability of Apache Spark 2.0.1! Visit the release notes to read about the new features, or download the release today.

+ + +

+
+Spark News Archive +

+ +
+
+ + + + + +
+ + + diff --git a/site/news/spark-2.0.0-preview.html b/site/news/spark-2.0.0-preview.html index c0730c37a..969a07908 100644 --- a/site/news/spark-2.0.0-preview.html +++ b/site/news/spark-2.0.0-preview.html @@ -150,6 +150,9 @@
Latest News

Archive

diff --git a/site/news/spark-accepted-into-apache-incubator.html b/site/news/spark-accepted-into-apache-incubator.html index e5f13313e..64b03238d 100644 --- a/site/news/spark-accepted-into-apache-incubator.html +++ b/site/news/spark-accepted-into-apache-incubator.html @@ -150,6 +150,9 @@
Latest News

Archive

diff --git a/site/news/spark-and-shark-in-the-news.html b/site/news/spark-and-shark-in-the-news.html index 528bb7579..f30a9c5bc 100644 --- a/site/news/spark-and-shark-in-the-news.html +++ b/site/news/spark-and-shark-in-the-news.html @@ -150,6 +150,9 @@
Latest News

Archive

@@ -196,7 +196,7 @@
  • DataInformed interviewed two Spark users and wrote about their applications in anomaly detection, predictive analytics and data mining.
    • -

    In other news, there will be a full day of tutorials on Spark and Shark at the O’Reilly Strata conference in February. They include a three-hour introduction to Spark, Shark and BDAS Tuesday morning, and a three-hour hands-on exercise session.

    +

    In other news, there will be a full day of tutorials on Spark and Shark at the O’Reilly Strata conference in February. They include a three-hour introduction to Spark, Shark and BDAS Tuesday morning, and a three-hour hands-on exercise session.

    diff --git a/site/news/spark-becomes-tlp.html b/site/news/spark-becomes-tlp.html index 0a7ce67d3..51d644dcb 100644 --- a/site/news/spark-becomes-tlp.html +++ b/site/news/spark-becomes-tlp.html @@ -150,6 +150,9 @@

    Latest News

    Archive

    diff --git a/site/news/spark-featured-in-wired.html b/site/news/spark-featured-in-wired.html index 79bf5ca31..3ccb7dc0c 100644 --- a/site/news/spark-featured-in-wired.html +++ b/site/news/spark-featured-in-wired.html @@ -150,6 +150,9 @@
    Latest News

    Archive

    diff --git a/site/news/spark-mailing-lists-moving-to-apache.html b/site/news/spark-mailing-lists-moving-to-apache.html index a3d756897..f8991062c 100644 --- a/site/news/spark-mailing-lists-moving-to-apache.html +++ b/site/news/spark-mailing-lists-moving-to-apache.html @@ -150,6 +150,9 @@
    Latest News

    Archive

    diff --git a/site/news/spark-meetups.html b/site/news/spark-meetups.html index 6cfce8fb3..0959ebfca 100644 --- a/site/news/spark-meetups.html +++ b/site/news/spark-meetups.html @@ -150,6 +150,9 @@
    Latest News

    Archive

    diff --git a/site/news/spark-screencasts-published.html b/site/news/spark-screencasts-published.html index d83c05f79..fb3eede41 100644 --- a/site/news/spark-screencasts-published.html +++ b/site/news/spark-screencasts-published.html @@ -150,6 +150,9 @@
    Latest News

    Archive

    diff --git a/site/news/spark-summit-2013-is-a-wrap.html b/site/news/spark-summit-2013-is-a-wrap.html index c95e365be..c143d2eef 100644 --- a/site/news/spark-summit-2013-is-a-wrap.html +++ b/site/news/spark-summit-2013-is-a-wrap.html @@ -150,6 +150,9 @@
    Latest News

    Archive

    diff --git a/site/news/spark-summit-2014-videos-posted.html b/site/news/spark-summit-2014-videos-posted.html index 67ee5b60a..312719c2b 100644 --- a/site/news/spark-summit-2014-videos-posted.html +++ b/site/news/spark-summit-2014-videos-posted.html @@ -150,6 +150,9 @@
    Latest News

    Archive

    diff --git a/site/news/spark-summit-2015-videos-posted.html b/site/news/spark-summit-2015-videos-posted.html index f2f0040b9..0bf46d91c 100644 --- a/site/news/spark-summit-2015-videos-posted.html +++ b/site/news/spark-summit-2015-videos-posted.html @@ -150,6 +150,9 @@
    Latest News

    Archive

    diff --git a/site/news/spark-summit-agenda-posted.html b/site/news/spark-summit-agenda-posted.html index 50e89e883..f9eba01be 100644 --- a/site/news/spark-summit-agenda-posted.html +++ b/site/news/spark-summit-agenda-posted.html @@ -150,6 +150,9 @@
    Latest News

    Archive

    diff --git a/site/news/spark-summit-east-2015-videos-posted.html b/site/news/spark-summit-east-2015-videos-posted.html index c5365268c..4dd71a4d9 100644 --- a/site/news/spark-summit-east-2015-videos-posted.html +++ b/site/news/spark-summit-east-2015-videos-posted.html @@ -150,6 +150,9 @@
    Latest News

    Archive

    @@ -186,7 +186,7 @@

    Spark Summit East 2015 Videos Posted

    -

    The videos and slides for Spark Summit East 2015 are now all available online. Watch them to get the latest news from the Spark community as well as use cases and applications built on top.

    +

    The videos and slides for Spark Summit East 2015 are now all available online. Watch them to get the latest news from the Spark community as well as use cases and applications built on top.

    If you like what you see, consider joining us at the 2015 Spark Summit in San Francisco.

    diff --git a/site/news/spark-summit-east-2016-cfp-closing.html b/site/news/spark-summit-east-2016-cfp-closing.html index 1147d071a..497fbf657 100644 --- a/site/news/spark-summit-east-2016-cfp-closing.html +++ b/site/news/spark-summit-east-2016-cfp-closing.html @@ -150,6 +150,9 @@
    Latest News

    Archive

    diff --git a/site/news/spark-summit-east-agenda-posted.html b/site/news/spark-summit-east-agenda-posted.html index 1d1f8bb81..4bdef4fb4 100644 --- a/site/news/spark-summit-east-agenda-posted.html +++ b/site/news/spark-summit-east-agenda-posted.html @@ -150,6 +150,9 @@
    Latest News

    Archive

    diff --git a/site/news/spark-summit-europe-agenda-posted.html b/site/news/spark-summit-europe-agenda-posted.html index b37387d56..429c15702 100644 --- a/site/news/spark-summit-europe-agenda-posted.html +++ b/site/news/spark-summit-europe-agenda-posted.html @@ -150,6 +150,9 @@
    Latest News

    Archive

    diff --git a/site/news/spark-summit-europe.html b/site/news/spark-summit-europe.html index 9bf66ed0d..e7441a177 100644 --- a/site/news/spark-summit-europe.html +++ b/site/news/spark-summit-europe.html @@ -150,6 +150,9 @@
    Latest News

    Archive

    diff --git a/site/news/spark-summit-june-2016-agenda-posted.html b/site/news/spark-summit-june-2016-agenda-posted.html index 5aeff108f..78c12b9e4 100644 --- a/site/news/spark-summit-june-2016-agenda-posted.html +++ b/site/news/spark-summit-june-2016-agenda-posted.html @@ -150,6 +150,9 @@
    Latest News

    Archive

    diff --git a/site/news/spark-tips-from-quantifind.html b/site/news/spark-tips-from-quantifind.html index 8b6c7ce41..1e819f6b0 100644 --- a/site/news/spark-tips-from-quantifind.html +++ b/site/news/spark-tips-from-quantifind.html @@ -150,6 +150,9 @@
    Latest News

    Archive

    diff --git a/site/news/spark-user-survey-and-powered-by-page.html b/site/news/spark-user-survey-and-powered-by-page.html index 0f0b8ecb2..06c7e686d 100644 --- a/site/news/spark-user-survey-and-powered-by-page.html +++ b/site/news/spark-user-survey-and-powered-by-page.html @@ -150,6 +150,9 @@
    Latest News

    Archive

    diff --git a/site/news/spark-version-0-6-0-released.html b/site/news/spark-version-0-6-0-released.html index b77e1ead0..cafd10922 100644 --- a/site/news/spark-version-0-6-0-released.html +++ b/site/news/spark-version-0-6-0-released.html @@ -150,6 +150,9 @@
    Latest News

    Archive

    diff --git a/site/news/spark-wins-daytona-gray-sort-100tb-benchmark.html b/site/news/spark-wins-daytona-gray-sort-100tb-benchmark.html index b168c6cd1..3f70dd1de 100644 --- a/site/news/spark-wins-daytona-gray-sort-100tb-benchmark.html +++ b/site/news/spark-wins-daytona-gray-sort-100tb-benchmark.html @@ -150,6 +150,9 @@
    Latest News

    Archive

    diff --git a/site/news/strata-exercises-now-available-online.html b/site/news/strata-exercises-now-available-online.html index fec18a0a3..08ce3a9ae 100644 --- a/site/news/strata-exercises-now-available-online.html +++ b/site/news/strata-exercises-now-available-online.html @@ -150,6 +150,9 @@
    Latest News

    Archive

    diff --git a/site/news/submit-talks-to-spark-summit-2014.html b/site/news/submit-talks-to-spark-summit-2014.html index ffe0dc245..f91b74302 100644 --- a/site/news/submit-talks-to-spark-summit-2014.html +++ b/site/news/submit-talks-to-spark-summit-2014.html @@ -150,6 +150,9 @@
    Latest News

    Archive

    diff --git a/site/news/submit-talks-to-spark-summit-2016.html b/site/news/submit-talks-to-spark-summit-2016.html index f4c1cb6af..919c1158b 100644 --- a/site/news/submit-talks-to-spark-summit-2016.html +++ b/site/news/submit-talks-to-spark-summit-2016.html @@ -150,6 +150,9 @@
    Latest News

    Archive

    diff --git a/site/news/submit-talks-to-spark-summit-east-2016.html b/site/news/submit-talks-to-spark-summit-east-2016.html index 4858b9d5b..8a8a9061e 100644 --- a/site/news/submit-talks-to-spark-summit-east-2016.html +++ b/site/news/submit-talks-to-spark-summit-east-2016.html @@ -150,6 +150,9 @@
    Latest News

    Archive

    diff --git a/site/news/submit-talks-to-spark-summit-eu-2016.html b/site/news/submit-talks-to-spark-summit-eu-2016.html index c3dbfff5d..5189ac618 100644 --- a/site/news/submit-talks-to-spark-summit-eu-2016.html +++ b/site/news/submit-talks-to-spark-summit-eu-2016.html @@ -150,6 +150,9 @@
    Latest News

    Archive

    diff --git a/site/news/two-weeks-to-spark-summit-2014.html b/site/news/two-weeks-to-spark-summit-2014.html index e1725353a..e20ecdd4f 100644 --- a/site/news/two-weeks-to-spark-summit-2014.html +++ b/site/news/two-weeks-to-spark-summit-2014.html @@ -150,6 +150,9 @@
    Latest News

    Archive

    diff --git a/site/news/video-from-first-spark-development-meetup.html b/site/news/video-from-first-spark-development-meetup.html index 8dcfbf7cf..0f84136cb 100644 --- a/site/news/video-from-first-spark-development-meetup.html +++ b/site/news/video-from-first-spark-development-meetup.html @@ -150,6 +150,9 @@
    Latest News

    Archive

    diff --git a/site/releases/spark-release-0-3.html b/site/releases/spark-release-0-3.html index 26e021366..4b01191b4 100644 --- a/site/releases/spark-release-0-3.html +++ b/site/releases/spark-release-0-3.html @@ -150,6 +150,9 @@
    Latest News

    Archive

    diff --git a/site/releases/spark-release-0-5-0.html b/site/releases/spark-release-0-5-0.html index a7f678492..da0eddc35 100644 --- a/site/releases/spark-release-0-5-0.html +++ b/site/releases/spark-release-0-5-0.html @@ -150,6 +150,9 @@
    Latest News

    Archive

    diff --git a/site/releases/spark-release-0-5-1.html b/site/releases/spark-release-0-5-1.html index fd5bcf7df..aebf9819c 100644 --- a/site/releases/spark-release-0-5-1.html +++ b/site/releases/spark-release-0-5-1.html @@ -150,6 +150,9 @@
    Latest News

    Archive

    diff --git a/site/releases/spark-release-0-5-2.html b/site/releases/spark-release-0-5-2.html index ebf45d9a6..66fd5d609 100644 --- a/site/releases/spark-release-0-5-2.html +++ b/site/releases/spark-release-0-5-2.html @@ -150,6 +150,9 @@
    Latest News

    Archive

    diff --git a/site/releases/spark-release-0-6-0.html b/site/releases/spark-release-0-6-0.html index ee102928f..1f5fd327d 100644 --- a/site/releases/spark-release-0-6-0.html +++ b/site/releases/spark-release-0-6-0.html @@ -150,6 +150,9 @@
    Latest News

    Archive

    diff --git a/site/releases/spark-release-0-6-1.html b/site/releases/spark-release-0-6-1.html index b0a22c1f3..482af1fda 100644 --- a/site/releases/spark-release-0-6-1.html +++ b/site/releases/spark-release-0-6-1.html @@ -150,6 +150,9 @@
    Latest News

    Archive

    diff --git a/site/releases/spark-release-0-6-2.html b/site/releases/spark-release-0-6-2.html index 2afdef329..739b55069 100644 --- a/site/releases/spark-release-0-6-2.html +++ b/site/releases/spark-release-0-6-2.html @@ -150,6 +150,9 @@
    Latest News

    Archive

    diff --git a/site/releases/spark-release-0-7-0.html b/site/releases/spark-release-0-7-0.html index ac7b0f93c..c0574df75 100644 --- a/site/releases/spark-release-0-7-0.html +++ b/site/releases/spark-release-0-7-0.html @@ -150,6 +150,9 @@
    Latest News

    Archive

    diff --git a/site/releases/spark-release-0-7-2.html b/site/releases/spark-release-0-7-2.html index 51b975a4e..686ddf39c 100644 --- a/site/releases/spark-release-0-7-2.html +++ b/site/releases/spark-release-0-7-2.html @@ -150,6 +150,9 @@
    Latest News

    Archive

    diff --git a/site/releases/spark-release-0-7-3.html b/site/releases/spark-release-0-7-3.html index 5694df047..376f889cc 100644 --- a/site/releases/spark-release-0-7-3.html +++ b/site/releases/spark-release-0-7-3.html @@ -150,6 +150,9 @@
    Latest News

    Archive

    diff --git a/site/releases/spark-release-0-8-0.html b/site/releases/spark-release-0-8-0.html index 61d7c573e..2f56fecde 100644 --- a/site/releases/spark-release-0-8-0.html +++ b/site/releases/spark-release-0-8-0.html @@ -150,6 +150,9 @@
    Latest News

    Archive

    @@ -210,13 +210,13 @@

    Spark’s internal job scheduler has been refactored and extended to include more sophisticated scheduling policies. In particular, a fair scheduler implementation now allows multiple users to share an instance of Spark, which helps users running shorter jobs to achieve good performance, even when longer-running jobs are running in parallel. Support for topology-aware scheduling has been extended, including the ability to take into account rack locality and support for multiple executors on a single machine.

    Easier Deployment and Linking

    -

    User programs can now link to Spark no matter which Hadoop version they need, without having to publish a version of spark-core specifically for that Hadoop version. An explanation of how to link against different Hadoop versions is provided here.

    +

    User programs can now link to Spark no matter which Hadoop version they need, without having to publish a version of spark-core specifically for that Hadoop version. An explanation of how to link against different Hadoop versions is provided here.

    Expanded EC2 Capabilities

    Spark’s EC2 scripts now support launching in any availability zone. Support has also been added for EC2 instance types which use the newer “HVM” architecture. This includes the cluster compute (cc1/cc2) family of instance types. We’ve also added support for running newer versions of HDFS alongside Spark. Finally, we’ve added the ability to launch clusters with maintenance releases of Spark in addition to launching the newest release.

    Improved Documentation

    -

    This release adds documentation about cluster hardware provisioning and inter-operation with common Hadoop distributions. Docs are also included to cover the MLlib machine learning functions and new cluster monitoring features. Existing documentation has been updated to reflect changes in building and deploying Spark.

    +

    This release adds documentation about cluster hardware provisioning and inter-operation with common Hadoop distributions. Docs are also included to cover the MLlib machine learning functions and new cluster monitoring features. Existing documentation has been updated to reflect changes in building and deploying Spark.

    Other Improvements

    Improvements to other deployment scenarios

    @@ -230,19 +230,19 @@

    Optimizations to MLLib

    Bug fixes and better API parity for PySpark

    @@ -274,13 +274,13 @@
  • Kay Ousterhout - Multiple bug fixes in scheduler’s handling of task failures
  • Kousuke Saruta - Use of https to access github
  • Mark Grover - Bug fix in distribution tar.gz
    • -
  • Matei Zaharia - Bug fixes in handling of task failures due to NPE, and cleaning up of scheduler data structures
    • +
  • Matei Zaharia - Bug fixes in handling of task failures due to NPE, and cleaning up of scheduler data structures
  • Nan Zhu - Bug fixes in PySpark RDD.takeSample and adding of JARs using ADD_JAR - and improvements to docs
  • Nick Lanham - Added ability to make distribution tarballs with Tachyon
  • Patrick Wendell - Bug fixes in ASM shading, fixes for log4j initialization, removing Ganglia due to LGPL license, and other miscallenous bug fixes
  • Prabin Banka - RDD.zip and other missing RDD operations in PySpark
  • Prashant Sharma - RDD.foldByKey in PySpark, and other PySpark doc improvements
    • -
  • Qiuzhuang - Bug fix in standalone worker
    • +
  • Qiuzhuang - Bug fix in standalone worker
  • Raymond Liu - Changed working directory in ZookeeperPersistenceEngine
  • Reynold Xin - Improvements to docs and test infrastructure
  • Sandy Ryza - Multiple important Yarn bug fixes and improvements
    • diff --git a/site/releases/spark-release-0-9-2.html b/site/releases/spark-release-0-9-2.html index eea5f60fa..58b7d23b5 100644 --- a/site/releases/spark-release-0-9-2.html +++ b/site/releases/spark-release-0-9-2.html @@ -150,6 +150,9 @@
    Latest News

    Archive

    diff --git a/site/releases/spark-release-1-0-0.html b/site/releases/spark-release-1-0-0.html index ad62eddde..a89f79f4f 100644 --- a/site/releases/spark-release-1-0-0.html +++ b/site/releases/spark-release-1-0-0.html @@ -150,6 +150,9 @@
    Latest News

    Archive

    diff --git a/site/releases/spark-release-1-0-1.html b/site/releases/spark-release-1-0-1.html index 4d1ae6ad5..408fb2dcc 100644 --- a/site/releases/spark-release-1-0-1.html +++ b/site/releases/spark-release-1-0-1.html @@ -150,6 +150,9 @@
    Latest News

    Archive

    @@ -258,8 +258,8 @@
  • Cheng Hao – SQL features
  • Cheng Lian – SQL features
  • Christian Tzolov – build improvmenet
    • -
  • Clément MATHIEU – doc updates
    • -
  • CodingCat – doc updates and bug fix
    • +
  • Clément MATHIEU – doc updates
    • +
  • CodingCat – doc updates and bug fix
  • Colin McCabe – bug fix
  • Daoyuan – SQL joins
  • David Lemieux – bug fix
    • @@ -275,7 +275,7 @@
  • Kan Zhang – PySpark SQL features
  • Kay Ousterhout – documentation fix
  • LY Lai – bug fix
    • -
  • Lars Albertsson – bug fix
    • +
  • Lars Albertsson – bug fix
  • Lei Zhang – SQL fix and feature
  • Mark Hamstra – bug fix
  • Matei Zaharia – doc updates and bug fix
    • @@ -297,7 +297,7 @@
  • Shixiong Zhu – code clean-up
  • Szul, Piotr – bug fix
  • Takuya UESHIN – bug fixes and SQL features
    • -
  • Thomas Graves – bug fix
    • +
  • Thomas Graves – bug fix
  • Uri Laserson – bug fix
  • Vadim Chekan – bug fix
  • Varakhedi Sujeet – ec2 r3 support
    • diff --git a/site/releases/spark-release-1-0-2.html b/site/releases/spark-release-1-0-2.html index d23897867..9e3559d41 100644 --- a/site/releases/spark-release-1-0-2.html +++ b/site/releases/spark-release-1-0-2.html @@ -150,6 +150,9 @@
    Latest News

    Archive

    @@ -268,7 +268,7 @@
  • johnnywalleye - Bug fixes in MLlib
  • joyyoj - Bug fix in Streaming
  • kballou - Doc fix
    • -
  • lianhuiwang - Doc fix
    • +
  • lianhuiwang - Doc fix
  • witgo - Bug fix in sbt
    • diff --git a/site/releases/spark-release-1-1-0.html b/site/releases/spark-release-1-1-0.html index 71d38a160..5c43cd7ed 100644 --- a/site/releases/spark-release-1-1-0.html +++ b/site/releases/spark-release-1-1-0.html @@ -150,6 +150,9 @@
    Latest News

    Archive

    @@ -197,7 +197,7 @@

    Spark SQL adds a number of new features and performance improvements in this release. A JDBC/ODBC server allows users to connect to SparkSQL from many different applications and provides shared access to cached tables. A new module provides support for loading JSON data directly into Spark’s SchemaRDD format, including automatic schema inference. Spark SQL introduces dynamic bytecode generation in this release, a technique which significantly speeds up execution for queries that perform complex expression evaluation. This release also adds support for registering Python, Scala, and Java lambda functions as UDFs, which can then be called directly in SQL. Spark 1.1 adds a public types API to allow users to create SchemaRDD’s from custom data sources. Finally, many optimizations have been added to the native Parquet support as well as throughout the engine.

    MLlib

    -

    MLlib adds several new algorithms and optimizations in this release. 1.1 introduces a new library of statistical packages which provides exploratory analytic functions. These include stratified sampling, correlations, chi-squared tests and support for creating random datasets. This release adds utilities for feature extraction (Word2Vec and TF-IDF) and feature transformation (normalization and standard scaling). Also new are support for nonnegative matrix factorization and SVD via Lanczos. The decision tree algorithm has been added in Python and Java. A tree aggregation primitive has been added to help optimize many existing algorithms. Performance improves across the board in MLlib 1.1, with improvements of around 2-3X for many algorithms and up to 5X for large scale decision tree problems.

    +

    MLlib adds several new algorithms and optimizations in this release. 1.1 introduces a new library of statistical packages which provides exploratory analytic functions. These include stratified sampling, correlations, chi-squared tests and support for creating random datasets. This release adds utilities for feature extraction (Word2Vec and TF-IDF) and feature transformation (normalization and standard scaling). Also new are support for nonnegative matrix factorization and SVD via Lanczos. The decision tree algorithm has been added in Python and Java. A tree aggregation primitive has been added to help optimize many existing algorithms. Performance improves across the board in MLlib 1.1, with improvements of around 2-3X for many algorithms and up to 5X for large scale decision tree problems.

    GraphX and Spark Streaming

    Spark streaming adds a new data source Amazon Kinesis. For the Apache Flume, a new mode is supported which pulls data from Flume, simplifying deployment and providing high availability. The first of a set of streaming machine learning algorithms is introduced with streaming linear regression. Finally, rate limiting has been added for streaming inputs. GraphX adds custom storage levels for vertices and edges along with improved numerical precision across the board. Finally, GraphX adds a new label propagation algorithm.

    @@ -215,7 +215,7 @@ @@ -275,7 +275,7 @@
  • Daneil Darabos – bug fixes and UI enhancements
  • Daoyuan Wang – SQL fixes
  • David Lemieux – bug fix
    • -
  • Davies Liu – PySpark fixes and spilling
    • +
  • Davies Liu – PySpark fixes and spilling
  • DB Tsai – online summaries in MLlib and other MLlib features
  • Derek Ma – bug fix
  • Doris Xin – MLlib stats library and several fixes
    • diff --git a/site/releases/spark-release-1-1-1.html b/site/releases/spark-release-1-1-1.html index b0ddad874..ff4ea4ff8 100644 --- a/site/releases/spark-release-1-1-1.html +++ b/site/releases/spark-release-1-1-1.html @@ -150,6 +150,9 @@
    Latest News

    Archive

    diff --git a/site/releases/spark-release-1-2-0.html b/site/releases/spark-release-1-2-0.html index 3eea59bc0..3c7475673 100644 --- a/site/releases/spark-release-1-2-0.html +++ b/site/releases/spark-release-1-2-0.html @@ -150,6 +150,9 @@
    Latest News

    Archive

    @@ -194,7 +194,7 @@

    In 1.2 Spark core upgrades two major subsystems to improve the performance and stability of very large scale shuffles. The first is Spark’s communication manager used during bulk transfers, which upgrades to a netty-based implementation. The second is Spark’s shuffle mechanism, which upgrades to the “sort based” shuffle initially released in Spark 1.1. These both improve the performance and stability of very large scale shuffles. Spark also adds an elastic scaling mechanism designed to improve cluster utilization during long running ETL-style jobs. This is currently supported on YARN and will make its way to other cluster managers in future versions. Finally, Spark 1.2 adds support for Scala 2.11. For instructions on building for Scala 2.11 see the build documentation.

    Spark Streaming

    -

    This release includes two major feature additions to Spark’s streaming library, a Python API and a write ahead log for full driver H/A. The Python API covers almost all the DStream transformations and output operations. Input sources based on text files and text over sockets are currently supported. Support for Kafka and Flume input streams in Python will be added in the next release. Second, Spark streaming now features H/A driver support through a write ahead log (WAL). In Spark 1.1 and earlier, some buffered (received but not yet processed) data can be lost during driver restarts. To prevent this Spark 1.2 adds an optional WAL, which buffers received data into a fault-tolerant file system (e.g. HDFS). See the streaming programming guide for more details.

    +

    This release includes two major feature additions to Spark’s streaming library, a Python API and a write ahead log for full driver H/A. The Python API covers almost all the DStream transformations and output operations. Input sources based on text files and text over sockets are currently supported. Support for Kafka and Flume input streams in Python will be added in the next release. Second, Spark streaming now features H/A driver support through a write ahead log (WAL). In Spark 1.1 and earlier, some buffered (received but not yet processed) data can be lost during driver restarts. To prevent this Spark 1.2 adds an optional WAL, which buffers received data into a fault-tolerant file system (e.g. HDFS). See the streaming programming guide for more details.

    MLLib

    Spark 1.2 previews a new set of machine learning API’s in a package called spark.ml that supports learning pipelines, where multiple algorithms are run in sequence with varying parameters. This type of pipeline is common in practical machine learning deployments. The new ML package uses Spark’s SchemaRDD to represent ML datasets, providing direct interoperability with Spark SQL. In addition to the new API, Spark 1.2 extends decision trees with two tree ensemble methods: random forests and gradient-boosted trees, among the most successful tree-based models for classification and regression. Finally, MLlib’s Python implementation receives a major update in 1.2 to simplify the process of adding Python APIs, along with better Python API coverage.

    diff --git a/site/releases/spark-release-1-2-1.html b/site/releases/spark-release-1-2-1.html index a4a1a67bd..d220fa285 100644 --- a/site/releases/spark-release-1-2-1.html +++ b/site/releases/spark-release-1-2-1.html @@ -150,6 +150,9 @@
    Latest News

    Archive

    diff --git a/site/releases/spark-release-1-2-2.html b/site/releases/spark-release-1-2-2.html index 58f7b87e8..7b9f3d756 100644 --- a/site/releases/spark-release-1-2-2.html +++ b/site/releases/spark-release-1-2-2.html @@ -150,6 +150,9 @@
    Latest News

    Archive

    diff --git a/site/releases/spark-release-1-3-0.html b/site/releases/spark-release-1-3-0.html index 1e673ff66..978d0fefc 100644 --- a/site/releases/spark-release-1-3-0.html +++ b/site/releases/spark-release-1-3-0.html @@ -150,6 +150,9 @@
    Latest News

    Archive

    @@ -191,7 +191,7 @@

    To download Spark 1.3 visit the downloads page.

    Spark Core

    -

    Spark 1.3 sees a handful of usability improvements in the core engine. The core API now supports multi level aggregation trees to help speed up expensive reduce operations. Improved error reporting has been added for certain gotcha operations. Spark’s Jetty dependency is now shaded to help avoid conflicts with user programs. Spark now supports SSL encryption for some communication endpoints. Finaly, realtime GC metrics and record counts have been added to the UI.

    +

    Spark 1.3 sees a handful of usability improvements in the core engine. The core API now supports multi level aggregation trees to help speed up expensive reduce operations. Improved error reporting has been added for certain gotcha operations. Spark’s Jetty dependency is now shaded to help avoid conflicts with user programs. Spark now supports SSL encryption for some communication endpoints. Finaly, realtime GC metrics and record counts have been added to the UI.

    DataFrame API

    Spark 1.3 adds a new DataFrames API that provides powerful and convenient operators when working with structured datasets. The DataFrame is an evolution of the base RDD API that includes named fields along with schema information. It’s easy to construct a DataFrame from sources such as Hive tables, JSON data, a JDBC database, or any implementation of Spark’s new data source API. Data frames will become a common interchange format between Spark components and when importing and exporting data to other systems. Data frames are supported in Python, Scala, and Java.

    @@ -203,7 +203,7 @@

    In this release Spark MLlib introduces several new algorithms: latent Dirichlet allocation (LDA) for topic modeling, multinomial logistic regression for multiclass classification, Gaussian mixture model (GMM) and power iteration clustering for clustering, FP-growth for frequent pattern mining, and block matrix abstraction for distributed linear algebra. Initial support has been added for model import/export in exchangeable format, which will be expanded in future versions to cover more model types in Java/Python/Scala. The implementations of k-means and ALS receive updates that lead to significant performance gain. PySpark now supports the ML pipeline API added in Spark 1.2, and gradient boosted trees and Gaussian mixture model. Finally, the ML pipeline API has been ported to support the new DataFrames abstraction.

    Spark Streaming

    -

    Spark 1.3 introduces a new direct Kafka API (docs) which enables exactly-once delivery without the use of write ahead logs. It also adds a Python Kafka API along with infrastructure for additional Python API’s in future releases. An online version of logistic regression and the ability to read binary records have also been added. For stateful operations, support has been added for loading of an initial state RDD. Finally, the streaming programming guide has been updated to include information about SQL and DataFrame operations within streaming applications, and important clarifications to the fault-tolerance semantics.

    +

    Spark 1.3 introduces a new direct Kafka API (docs) which enables exactly-once delivery without the use of write ahead logs. It also adds a Python Kafka API along with infrastructure for additional Python API’s in future releases. An online version of logistic regression and the ability to read binary records have also been added. For stateful operations, support has been added for loading of an initial state RDD. Finally, the streaming programming guide has been updated to include information about SQL and DataFrame operations within streaming applications, and important clarifications to the fault-tolerance semantics.

    GraphX

    GraphX adds a handful of utility functions in this release, including conversion into a canonical edge graph.

    @@ -219,7 +219,7 @@ diff --git a/site/releases/spark-release-1-3-1.html b/site/releases/spark-release-1-3-1.html index 027490bbe..d24675ab2 100644 --- a/site/releases/spark-release-1-3-1.html +++ b/site/releases/spark-release-1-3-1.html @@ -150,6 +150,9 @@
    Latest News

    Archive

    @@ -196,10 +196,10 @@

    Spark SQL

    Spark Streaming

    diff --git a/site/releases/spark-release-1-4-0.html b/site/releases/spark-release-1-4-0.html index 8d60c0f97..db4c88c46 100644 --- a/site/releases/spark-release-1-4-0.html +++ b/site/releases/spark-release-1-4-0.html @@ -150,6 +150,9 @@
    Latest News

    Archive

    @@ -250,7 +250,7 @@ Python coverage. MLlib also adds several new algorithms.

    Spark Streaming

    -

    Spark streaming adds visual instrumentation graphs and significantly improved debugging information in the UI. It also enhances support for both Kafka and Kinesis.

    +

    Spark streaming adds visual instrumentation graphs and significantly improved debugging information in the UI. It also enhances support for both Kafka and Kinesis.