diff options
| author | Kostas Sakellis <kostas@cloudera.com> | 2015-01-15 18:48:39 -0800 |
|---|---|---|
| committer | Patrick Wendell <pwendell@gmail.com> | 2015-01-15 18:48:39 -0800 |
| commit | a79a9f923c47f2ce7da93cf0ecfe2b66fcb9fdd4 (patch) | |
| tree | ed37829d87ea0568cd8978b785e2f44f43fd766a /docs/configuration.md | |
| parent | 96c2c714f4f9abe20d4c42d99ffaafcb269714a1 (diff) | |
| download | spark-a79a9f923c47f2ce7da93cf0ecfe2b66fcb9fdd4.tar.gz spark-a79a9f923c47f2ce7da93cf0ecfe2b66fcb9fdd4.tar.bz2 spark-a79a9f923c47f2ce7da93cf0ecfe2b66fcb9fdd4.zip | |
[SPARK-4092] [CORE] Fix InputMetrics for coalesce'd Rdds
When calculating the input metrics there was an assumption that one task only reads from one block - this is not true for some operations including coalesce. This patch simply increments the task's input metrics if previous ones existed of the same read method.
A limitation to this patch is that if a task reads from two different blocks of different read methods, one will override the other.
Author: Kostas Sakellis <kostas@cloudera.com>
Closes #3120 from ksakellis/kostas-spark-4092 and squashes the following commits:
54e6658 [Kostas Sakellis] Drops metrics if conflicting read methods exist
f0e0cc5 [Kostas Sakellis] Add bytesReadCallback to InputMetrics
a2a36d4 [Kostas Sakellis] CR feedback
5a0c770 [Kostas Sakellis] [SPARK-4092] [CORE] Fix InputMetrics for coalesce'd Rdds
Diffstat (limited to 'docs/configuration.md')
0 files changed, 0 insertions, 0 deletions