diff options
| author | Xiangrui Meng <meng@databricks.com> | 2014-07-16 11:27:51 -0700 |
|---|---|---|
| committer | Reynold Xin <rxin@apache.org> | 2014-07-16 11:27:51 -0700 |
| commit | 96f28c9726d18f3b0d7a57b128c16ec9157f1532 (patch) | |
| tree | 8504c1594ffe9743909a3b36db1a0561e2b348db | |
| parent | ef48222c10be3d29a83dfc2329f455eba203cd38 (diff) | |
| download | spark-96f28c9726d18f3b0d7a57b128c16ec9157f1532.tar.gz spark-96f28c9726d18f3b0d7a57b128c16ec9157f1532.tar.bz2 spark-96f28c9726d18f3b0d7a57b128c16ec9157f1532.zip | |
[SPARK-2522] set default broadcast factory to torrent
HttpBroadcastFactory is the current default broadcast factory. It sends the broadcast data to each worker one by one, which is slow when the cluster is big. TorrentBroadcastFactory scales much better than http. Maybe we should make torrent the default broadcast method.
Author: Xiangrui Meng <meng@databricks.com>
Closes #1437 from mengxr/bt-broadcast and squashes the following commits:
ed492fe [Xiangrui Meng] set default broadcast factory to torrent
| -rw-r--r-- | core/src/main/scala/org/apache/spark/broadcast/BroadcastManager.scala | 2 | ||||
| -rw-r--r-- | docs/configuration.md | 2 |
2 files changed, 2 insertions, 2 deletions
diff --git a/core/src/main/scala/org/apache/spark/broadcast/BroadcastManager.scala b/core/src/main/scala/org/apache/spark/broadcast/BroadcastManager.scala index c88be6aba6..8f8a0b11f9 100644 --- a/core/src/main/scala/org/apache/spark/broadcast/BroadcastManager.scala +++ b/core/src/main/scala/org/apache/spark/broadcast/BroadcastManager.scala @@ -39,7 +39,7 @@ private[spark] class BroadcastManager( synchronized { if (!initialized) { val broadcastFactoryClass = - conf.get("spark.broadcast.factory", "org.apache.spark.broadcast.HttpBroadcastFactory") + conf.get("spark.broadcast.factory", "org.apache.spark.broadcast.TorrentBroadcastFactory") broadcastFactory = Class.forName(broadcastFactoryClass).newInstance.asInstanceOf[BroadcastFactory] diff --git a/docs/configuration.md b/docs/configuration.md index 9d3fe74414..a70007c165 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -419,7 +419,7 @@ Apart from these, the following properties are also available, and may be useful </tr> <tr> <td><code>spark.broadcast.factory</code></td> - <td>org.apache.spark.broadcast.<br />HttpBroadcastFactory</td> + <td>org.apache.spark.broadcast.<br />TorrentBroadcastFactory</td> <td> Which broadcast implementation to use. </td> |