diff options
| author | Zhan Zhang <zhazhan@gmail.com> | 2015-04-30 22:24:31 -0700 |
|---|---|---|
| committer | Patrick Wendell <patrick@databricks.com> | 2015-04-30 22:24:31 -0700 |
| commit | 36a7a6807ee58c8ba309c0e09e547919246b7f82 (patch) | |
| tree | 25863218dad04112ec03b08a83d91a7ec10a0b3b /docs | |
| parent | b5347a4664625ede6ab9d8ef6558457a34ae423f (diff) | |
| download | spark-36a7a6807ee58c8ba309c0e09e547919246b7f82.tar.gz spark-36a7a6807ee58c8ba309c0e09e547919246b7f82.tar.bz2 spark-36a7a6807ee58c8ba309c0e09e547919246b7f82.zip | |
[SPARK-6479] [BLOCK MANAGER] Create off-heap block storage API
This is the classes for creating off-heap block storage API. It also includes the migration for Tachyon. The diff seems to be big, but it mainly just rename tachyon to offheap. New implementation for hdfs will be submit for review in spark-6112.
Author: Zhan Zhang <zhazhan@gmail.com>
Closes #5430 from zhzhan/SPARK-6479 and squashes the following commits:
60acd84 [Zhan Zhang] minor change to kickoff the test
12f54c9 [Zhan Zhang] solve merge conflicts
a54132c [Zhan Zhang] solve review comments
ffb8e00 [Zhan Zhang] rebase to sparkcontext change
6e121e0 [Zhan Zhang] resolve review comments and restructure blockmanasger code
a7aed6c [Zhan Zhang] add Tachyon migration code
186de31 [Zhan Zhang] initial commit for off-heap block storage api
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/configuration.md | 22 |
1 files changed, 15 insertions, 7 deletions
diff --git a/docs/configuration.md b/docs/configuration.md index 72105feba4..7239b252a9 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -1,4 +1,4 @@ ---- +-- layout: global displayTitle: Spark Configuration title: Configuration @@ -843,19 +843,27 @@ Apart from these, the following properties are also available, and may be useful </td> </tr> <tr> - <td><code>spark.tachyonStore.baseDir</code></td> + <td><code>spark.externalBlockStore.blockManager</code></td> + <td>org.apache.spark.storage.TachyonBlockManager</td> + <td> + Implementation of external block manager (file system) that store RDDs. The file system's URL is set by + <code>spark.externalBlockStore.url</code>. + </td> +</tr> +<tr> + <td><code>spark.externalBlockStore.baseDir</code></td> <td>System.getProperty("java.io.tmpdir")</td> <td> - Directories of the Tachyon File System that store RDDs. The Tachyon file system's URL is set by - <code>spark.tachyonStore.url</code>. It can also be a comma-separated list of multiple + Directories of the external block store that store RDDs. The file system's URL is set by + <code>spark.externalBlockStore.url</code> It can also be a comma-separated list of multiple directories on Tachyon file system. </td> </tr> <tr> - <td><code>spark.tachyonStore.url</code></td> - <td>tachyon://localhost:19998</td> + <td><code>spark.externalBlockStore.url</code></td> + <td>tachyon://localhost:19998 for Tachyon</td> <td> - The URL of the underlying Tachyon file system in the TachyonStore. + The URL of the underlying external blocker file system in the external block store. </td> </tr> </table> |