diff options
author | Matei Zaharia <matei@eecs.berkeley.edu> | 2013-01-28 22:30:12 -0800 |
---|---|---|
committer | Matei Zaharia <matei@eecs.berkeley.edu> | 2013-01-28 22:30:12 -0800 |
commit | 64ba6a8c2c5f46e6de6deb6a6fd576a55cb3b198 (patch) | |
tree | ddf8897a1e3e3efa02bb8dedf6e8592bb9fb2494 /streaming/src | |
parent | b29599e5cf0272f0d0e3ceceebb473a8163eab8c (diff) | |
download | spark-64ba6a8c2c5f46e6de6deb6a6fd576a55cb3b198.tar.gz spark-64ba6a8c2c5f46e6de6deb6a6fd576a55cb3b198.tar.bz2 spark-64ba6a8c2c5f46e6de6deb6a6fd576a55cb3b198.zip |
Simplify checkpointing code and RDD class a little:
- RDD's getDependencies and getSplits methods are now guaranteed to be
called only once, so subclasses can safely do computation in there
without worrying about caching the results.
- The management of a "splits_" variable that is cleared out when we
checkpoint an RDD is now done in the RDD class.
- A few of the RDD subclasses are simpler.
- CheckpointRDD's compute() method no longer assumes that it is given a
CheckpointRDDSplit -- it can work just as well on a split from the
original RDD, because it only looks at its index. This is important
because things like UnionRDD and ZippedRDD remember the parent's
splits as part of their own and wouldn't work on checkpointed parents.
- RDD.iterator can now reuse cached data if an RDD is computed before it
is checkpointed. It seems like it wouldn't do this before (it always
called iterator() on the CheckpointRDD, which read from HDFS).
Diffstat (limited to 'streaming/src')
0 files changed, 0 insertions, 0 deletions