diff options
| author | Reynold Xin <rxin@databricks.com> | 2014-11-11 00:25:31 -0800 |
|---|---|---|
| committer | Aaron Davidson <aaron@databricks.com> | 2014-11-11 00:25:49 -0800 |
| commit | e9d009dc348bc06198ed2c9e03f1ba870401e6df (patch) | |
| tree | 51fd7da463822b7bad2935b069e0929da5727499 /external | |
| parent | df8242c9b6307c085d4c1a7ec446b1701a7e7cde (diff) | |
| download | spark-e9d009dc348bc06198ed2c9e03f1ba870401e6df.tar.gz spark-e9d009dc348bc06198ed2c9e03f1ba870401e6df.tar.bz2 spark-e9d009dc348bc06198ed2c9e03f1ba870401e6df.zip | |
[SPARK-4307] Initialize FileDescriptor lazily in FileRegion.
Netty's DefaultFileRegion requires a FileDescriptor in its constructor, which means we need to have a opened file handle. In super large workloads, this could lead to too many open files due to the way these file descriptors are cleaned. This pull request creates a new LazyFileRegion that initializes the FileDescriptor when we are sending data for the first time.
Author: Reynold Xin <rxin@databricks.com>
Author: Reynold Xin <rxin@apache.org>
Closes #3172 from rxin/lazyFD and squashes the following commits:
0bdcdc6 [Reynold Xin] Added reference to Netty's DefaultFileRegion
d4564ae [Reynold Xin] Added SparkConf to the ctor argument of IndexShuffleBlockManager.
6ed369e [Reynold Xin] Code review feedback.
04cddc8 [Reynold Xin] [SPARK-4307] Initialize FileDescriptor lazily in FileRegion.
(cherry picked from commit ef29a9a9aa85468869eb67ca67b66c65f508d0ee)
Signed-off-by: Aaron Davidson <aaron@databricks.com>
Diffstat (limited to 'external')
0 files changed, 0 insertions, 0 deletions