diff options
Diffstat (limited to 'releases')
-rw-r--r-- | releases/_posts/2014-12-18-spark-release-1-2-0.md | 7 |
1 files changed, 3 insertions, 4 deletions
diff --git a/releases/_posts/2014-12-18-spark-release-1-2-0.md b/releases/_posts/2014-12-18-spark-release-1-2-0.md index 35d39f85d..747a4311a 100644 --- a/releases/_posts/2014-12-18-spark-release-1-2-0.md +++ b/releases/_posts/2014-12-18-spark-release-1-2-0.md @@ -16,13 +16,13 @@ Spark 1.2.0 is the third release on the 1.X line. This release brings performanc To download Spark 1.2 visit the <a href="{{site.url}}downloads.html">downloads</a> page. ### Spark Core -In 1.2 Spark core upgrades to major subsystems to improve the performance and stability of very large scale shuffles. The first is Spark’s communication manager used during bulk transfers, which upgrades to a [netty-based implementation](https://issues.apache.org/jira/browse/SPARK-2468). The second is Spark’s shuffle mechanism, which upgrades to the [“sort based” shuffle initially released in Spark 1.1](https://issues.apache.org/jira/browse/SPARK-3280). These both improve the performance and stability of very large scale shuffles. Spark also adds an [elastic scaling mechanism](https://issues.apache.org/jira/browse/SPARK-3174) designed to improve cluster utilization during long running ETL-style jobs. This is currently supported on YARN and will make its way to other cluster managers in future versions. Finally, Spark 1.2 adds support for Scala 2.11. For instructions on building for Scala 2.11 see the [build documentation](/docs/1.2.0/building-spark.html#building-for-scala-211). +In 1.2 Spark core upgrades two major subsystems to improve the performance and stability of very large scale shuffles. The first is Spark’s communication manager used during bulk transfers, which upgrades to a [netty-based implementation](https://issues.apache.org/jira/browse/SPARK-2468). The second is Spark’s shuffle mechanism, which upgrades to the [“sort based” shuffle initially released in Spark 1.1](https://issues.apache.org/jira/browse/SPARK-3280). These both improve the performance and stability of very large scale shuffles. Spark also adds an [elastic scaling mechanism](https://issues.apache.org/jira/browse/SPARK-3174) designed to improve cluster utilization during long running ETL-style jobs. This is currently supported on YARN and will make its way to other cluster managers in future versions. Finally, Spark 1.2 adds support for Scala 2.11. For instructions on building for Scala 2.11 see the [build documentation](/docs/1.2.0/building-spark.html#building-for-scala-211). ### Spark Streaming This release includes two major feature additions to Spark’s streaming library, a Python API and a write ahead log for full driver H/A. The [Python API](https://issues.apache.org/jira/browse/SPARK-2377) covers almost all the DStream transformations and output operations. Input sources based on text files and text over sockets are currently supported. Support for Kafka and Flume input streams in Python will be added in the next release. Second, Spark streaming now features H/A driver support through a [write ahead log (WAL)](https://issues.apache.org/jira/browse/SPARK-3129). In Spark 1.1 and earlier, some buffered (received but not yet processed) data can be lost during driver restarts. To prevent this Spark 1.2 adds an optional WAL, which buffers received data into a fault-tolerant file system (e.g. HDFS). See the [streaming programming guide](/docs/1.2.0/streaming-programming-guide.html) for more details. ### MLLib -Spark 1.2 previews a new set of machine learning API’s in a package called spark.ml that [supports learning pipelines](https://issues.apache.org/jira/browse/SPARK-3530), where multiple algorithms are run in sequence with varying parameters. This type of pipeline is common in practical machine learning deployments. The new ML package uses Spark’s SchemaRDD to represent [ML datasets](https://issues.apache.org/jira/browse/SPARK-3573), providing directly interoperability with Spark SQL. In addition to the new API, Spark 1.2 extends decision trees with two tree ensemble methods: [random forests](https://issues.apache.org/jira/browse/SPARK-1545) and [gradient-boosted trees](https://issues.apache.org/jira/browse/SPARK-1547), among the most successful tree-based models for classification and regression. Finally, MLlib's Python implementation receives a major update in 1.2 to simplify the process of adding Python APIs, along with better Python API coverage. +Spark 1.2 previews a new set of machine learning API’s in a package called spark.ml that [supports learning pipelines](https://issues.apache.org/jira/browse/SPARK-3530), where multiple algorithms are run in sequence with varying parameters. This type of pipeline is common in practical machine learning deployments. The new ML package uses Spark’s SchemaRDD to represent [ML datasets](https://issues.apache.org/jira/browse/SPARK-3573), providing direct interoperability with Spark SQL. In addition to the new API, Spark 1.2 extends decision trees with two tree ensemble methods: [random forests](https://issues.apache.org/jira/browse/SPARK-1545) and [gradient-boosted trees](https://issues.apache.org/jira/browse/SPARK-1547), among the most successful tree-based models for classification and regression. Finally, MLlib's Python implementation receives a major update in 1.2 to simplify the process of adding Python APIs, along with better Python API coverage. ### Spark SQL In this release Spark SQL adds a [new API for external data sources](https://issues.apache.org/jira/browse/SPARK-3247). This API supports mounting external data sources as temporary tables, with support for optimizations such as predicate pushdown. Spark’s [Parquet](https://issues.apache.org/jira/browse/SPARK-4413) and JSON bindings have been re-written to use this API and we expect a variety of community projects to integrate with other systems and formats during the 1.2 lifecycle. @@ -30,7 +30,6 @@ In this release Spark SQL adds a [new API for external data sources](https://iss Hive integration has been improved with support for the [fixed-precision decimal type](https://issues.apache.org/jira/browse/SPARK-3929) and [Hive 0.13](https://issues.apache.org/jira/browse/SPARK-2706). Spark SQL also adds [dynamically partitioned inserts](https://issues.apache.org/jira/browse/SPARK-3007), a popular Hive feature. An internal re-architecting around caching improves the performance and semantics of [caching SchemaRDD](https://issues.apache.org/jira/browse/SPARK-3212) instances and adds support for [statistics-based partition pruning](https://issues.apache.org/jira/browse/SPARK-2961) for cached data. ### GraphX -In 1.2 GraphX graduates from an alpha component and adds a stable API. This means applications written against GraphX are guaranteed to work with future Spark versions without code changes. In 1.2 GraphX graduates from an alpha component and adds a stable API. This means applications written against GraphX are guaranteed to work with future Spark versions without code changes. A new core API, aggregateMessages, is introduced to replace the now deprecated mapReduceTriplet API. The new aggregateMessages API features a more imperative programming model and improves performance. Some early test users found 20% - 1X performance improvement by switching to the new API. In addition, Spark now supports [graph checkpointing](https://issues.apache.org/jira/browse/SPARK-3623) and [lineage truncation](https://issues.apache.org/jira/browse/SPARK-4672) which are necessary to support large numbers of iterations in production jobs. Finally, a handful of performance improvements have been added for [PageRank](https://issues.apache.org/jira/browse/SPARK-3427) and [graph loading](https://issues.apache.org/jira/browse/SPARK-4646). @@ -43,7 +42,7 @@ In addition, Spark now supports [graph checkpointing](https://issues.apache.org/ ### Upgrading to Spark 1.2 -Spark 1.2 is binary compatible with Spark 1.0 and 1.1, so no code changes are necessary. This excludes API’s marked explicitly as unstable. Spark changes default configuration in a handful of cases for improved performance. Users who want to preserve identical configurations to Spark 1.1 can roll back these changes. +Spark 1.2 is binary compatible with Spark 1.0 and 1.1, so no code changes are necessary. This excludes APIs marked explicitly as unstable. Spark changes default configuration in a handful of cases for improved performance. Users who want to preserve identical configurations to Spark 1.1 can roll back these changes. 1. `spark.shuffle.blockTransferService` has been changed from `nio` to `netty` 2. `spark.shuffle.manager` has been changed from `hash` to `sort` |