diff options
| author | Tarek Auel <tarek.auel@googlemail.com> | 2015-07-02 10:02:19 -0700 |
|---|---|---|
| committer | Davies Liu <davies@databricks.com> | 2015-07-02 10:02:19 -0700 |
| commit | 5b3338130dfd9db92c4894a348839a62ebb57ef3 (patch) | |
| tree | 13b6289381caca923444107940da2ed5abba7628 /python/pyspark/profiler.py | |
| parent | 0a468a46bf5b905e9b0205e98b862570b2ac556e (diff) | |
| download | spark-5b3338130dfd9db92c4894a348839a62ebb57ef3.tar.gz spark-5b3338130dfd9db92c4894a348839a62ebb57ef3.tar.bz2 spark-5b3338130dfd9db92c4894a348839a62ebb57ef3.zip | |
[SPARK-8223] [SPARK-8224] [SQL] shift left and shift right
Jira:
https://issues.apache.org/jira/browse/SPARK-8223
https://issues.apache.org/jira/browse/SPARK-8224
~~I am aware of #7174 and will update this pr, if it's merged.~~ Done
I don't know if #7034 can simplify this, but we can have a look on it, if it gets merged
rxin In the Jira ticket the function as no second argument. I added a `numBits` argument that allows to specify the number of bits. I guess this improves the usability. I wanted to add `shiftleft(value)` as well, but the `selectExpr` dataframe tests crashes, if I have both. I order to do this, I added the following to the functions.scala `def shiftRight(e: Column): Column = ShiftRight(e.expr, lit(1).expr)`, but as I mentioned this doesn't pass tests like `df.selectExpr("shiftRight(a)", ...` (not enough arguments exception).
If we need the bitwise shift in order to be hive compatible, I suggest to add `shiftLeft` and something like `shiftLeftX`
Author: Tarek Auel <tarek.auel@googlemail.com>
Closes #7178 from tarekauel/8223 and squashes the following commits:
8023bb5 [Tarek Auel] [SPARK-8223][SPARK-8224] fixed test
f3f64e6 [Tarek Auel] [SPARK-8223][SPARK-8224] Integer -> Int
f628706 [Tarek Auel] [SPARK-8223][SPARK-8224] removed toString; updated function description
3b56f2a [Tarek Auel] Merge remote-tracking branch 'origin/master' into 8223
5189690 [Tarek Auel] [SPARK-8223][SPARK-8224] minor fix and style fix
9434a28 [Tarek Auel] Merge remote-tracking branch 'origin/master' into 8223
44ee324 [Tarek Auel] [SPARK-8223][SPARK-8224] docu fix
ac7fe9d [Tarek Auel] [SPARK-8223][SPARK-8224] right and left bit shift
Diffstat (limited to 'python/pyspark/profiler.py')
0 files changed, 0 insertions, 0 deletions