diff options
| author | Yin Huai <yhuai@databricks.com> | 2015-12-09 18:09:36 -0800 |
|---|---|---|
| committer | Yin Huai <yhuai@databricks.com> | 2015-12-09 18:09:36 -0800 |
| commit | ac8cdf1cdc148bd21290ecf4d4f9874f8c87cc14 (patch) | |
| tree | 159e45bbc24f9783807057ab06fceff5e31e1d00 /docs/sql-programming-guide.md | |
| parent | 8770bd1213f9b1051dabde9c5424ae7b32143a44 (diff) | |
| download | spark-ac8cdf1cdc148bd21290ecf4d4f9874f8c87cc14.tar.gz spark-ac8cdf1cdc148bd21290ecf4d4f9874f8c87cc14.tar.bz2 spark-ac8cdf1cdc148bd21290ecf4d4f9874f8c87cc14.zip | |
[SPARK-11678][SQL][DOCS] Document basePath in the programming guide.
This PR adds document for `basePath`, which is a new parameter used by `HadoopFsRelation`.
The compiled doc is shown below.
JIRA: https://issues.apache.org/jira/browse/SPARK-11678
Author: Yin Huai <yhuai@databricks.com>
Closes #10211 from yhuai/basePathDoc.
Diffstat (limited to 'docs/sql-programming-guide.md')
| -rw-r--r-- | docs/sql-programming-guide.md | 7 |
1 files changed, 7 insertions, 0 deletions
diff --git a/docs/sql-programming-guide.md b/docs/sql-programming-guide.md index 9f87accd30..3f9a831edd 100644 --- a/docs/sql-programming-guide.md +++ b/docs/sql-programming-guide.md @@ -1233,6 +1233,13 @@ infer the data types of the partitioning columns. For these use cases, the autom can be configured by `spark.sql.sources.partitionColumnTypeInference.enabled`, which is default to `true`. When type inference is disabled, string type will be used for the partitioning columns. +Starting from Spark 1.6.0, partition discovery only finds partitions under the given paths +by default. For the above example, if users pass `path/to/table/gender=male` to either +`SQLContext.read.parquet` or `SQLContext.read.load`, `gender` will not be considered as a +partitioning column. If users need to specify the base path that partition discovery +should start with, they can set `basePath` in the data source options. For example, +when `path/to/table/gender=male` is the path of the data and +users set `basePath` to `path/to/table/`, `gender` will be a partitioning column. ### Schema Merging |