diff options
| author | Joseph Batchik <josephbatchik@gmail.com> | 2015-07-29 23:35:55 -0700 |
|---|---|---|
| committer | Reynold Xin <rxin@databricks.com> | 2015-07-29 23:35:55 -0700 |
| commit | 1221849f91739454b8e495889cba7498ba8beea7 (patch) | |
| tree | 267b356aaa35da1189874af2a3dd1af3950cdbf4 /dev/sparktestsupport/modules.py | |
| parent | e127ec34d58ceb0a9d45748c2f2918786ba0a83d (diff) | |
| download | spark-1221849f91739454b8e495889cba7498ba8beea7.tar.gz spark-1221849f91739454b8e495889cba7498ba8beea7.tar.bz2 spark-1221849f91739454b8e495889cba7498ba8beea7.zip | |
[SPARK-8005][SQL] Input file name
Users can now get the file name of the partition being read in. A thread local variable is in `SQLNewHadoopRDD` and is set when the partition is computed. `SQLNewHadoopRDD` is moved to core so that the catalyst package can reach it.
This supports:
`df.select(inputFileName())`
and
`sqlContext.sql("select input_file_name() from table")`
Author: Joseph Batchik <josephbatchik@gmail.com>
Closes #7743 from JDrit/input_file_name and squashes the following commits:
abb8609 [Joseph Batchik] fixed failing test and changed the default value to be an empty string
d2f323d [Joseph Batchik] updates per review
102061f [Joseph Batchik] updates per review
75313f5 [Joseph Batchik] small fixes
c7f7b5a [Joseph Batchik] addeding input file name to Spark SQL
Diffstat (limited to 'dev/sparktestsupport/modules.py')
0 files changed, 0 insertions, 0 deletions