diff options
author | Jacker Hu <gt.hu.chang@gmail.com> | 2015-10-10 11:36:18 +0100 |
---|---|---|
committer | Sean Owen <sowen@cloudera.com> | 2015-10-10 11:36:18 +0100 |
commit | a16396df76cc27099011bfb96b28cbdd7f964ca8 (patch) | |
tree | fa24e961cd38f01993cee88208bf99acaf008444 /sql | |
parent | 864de3bf4041c829e95d278b9569e91448bab0cc (diff) | |
download | spark-a16396df76cc27099011bfb96b28cbdd7f964ca8.tar.gz spark-a16396df76cc27099011bfb96b28cbdd7f964ca8.tar.bz2 spark-a16396df76cc27099011bfb96b28cbdd7f964ca8.zip |
[SPARK-10772] [STREAMING] [SCALA] NullPointerException when transform function in DStream returns NULL
Currently, the ```TransformedDStream``` will using ```Some(transformFunc(parentRDDs, validTime))``` as compute return value, when the ```transformFunc``` somehow returns null as return value, the followed operator will have NullPointerExeception.
This fix uses the ```Option()``` instead of ```Some()``` to deal with the possible null value. When ```transformFunc``` returns ```null```, the option will transform null to ```None```, the downstream can handle ```None``` correctly.
NOTE (2015-09-25): The latest fix will check the return value of transform function, if it is ```NULL```, a spark exception will be thrown out
Author: Jacker Hu <gt.hu.chang@gmail.com>
Author: jhu-chang <gt.hu.chang@gmail.com>
Closes #8881 from jhu-chang/Fix_Transform.
Diffstat (limited to 'sql')
0 files changed, 0 insertions, 0 deletions