diff options
| author | CodingCat <zhunansjtu@gmail.com> | 2014-03-24 21:55:03 -0700 |
|---|---|---|
| committer | Aaron Davidson <aaron@databricks.com> | 2014-03-24 21:55:03 -0700 |
| commit | 5140598df889f7227c9d6a7953031eeef524badd (patch) | |
| tree | 129e43867802653f6e29d48ff1ee6c2396392929 /sbin | |
| parent | dc126f2121d0cd1dc0caa50ae0c4cb9137d42562 (diff) | |
| download | spark-5140598df889f7227c9d6a7953031eeef524badd.tar.gz spark-5140598df889f7227c9d6a7953031eeef524badd.tar.bz2 spark-5140598df889f7227c9d6a7953031eeef524badd.zip | |
SPARK-1128: set hadoop task properties when constructing HadoopRDD
https://spark-project.atlassian.net/browse/SPARK-1128
The task properties are not set when constructing HadoopRDD in current implementation, this may limit the implementation based on
```
mapred.tip.id
mapred.task.id
mapred.task.is.map
mapred.task.partition
mapred.job.id
```
This patch also contains a small fix in createJobID (SparkHadoopWriter.scala), where the current implementation actually is not using time parameter
Author: CodingCat <zhunansjtu@gmail.com>
Author: Nan Zhu <CodingCat@users.noreply.github.com>
Closes #101 from CodingCat/SPARK-1128 and squashes the following commits:
ed0980f [CodingCat] make SparkHiveHadoopWriter belongs to spark package
5b1ad7d [CodingCat] move SparkHiveHadoopWriter to org.apache.spark package
258f92c [CodingCat] code cleanup
af88939 [CodingCat] update the comments and permission of SparkHadoopWriter
9bd1fe3 [CodingCat] move configuration for jobConf to HadoopRDD
b7bdfa5 [Nan Zhu] style fix
a3153a8 [Nan Zhu] style fix
c3258d2 [CodingCat] set hadoop task properties while using InputFormat
Diffstat (limited to 'sbin')
0 files changed, 0 insertions, 0 deletions