diff options
author | Sean Owen <sowen@cloudera.com> | 2016-10-24 10:44:45 +0100 |
---|---|---|
committer | Sean Owen <sowen@cloudera.com> | 2016-10-24 10:44:45 +0100 |
commit | 4ecbe1b92f4c4c5b2d734895c09d8ded0ed48d4d (patch) | |
tree | c996509d584d6d5da25d647b7ac43d3eab9b5853 /docs/sql-programming-guide.md | |
parent | c64a8ff39794d60c596c0d34130019c09c9c8012 (diff) | |
download | spark-4ecbe1b92f4c4c5b2d734895c09d8ded0ed48d4d.tar.gz spark-4ecbe1b92f4c4c5b2d734895c09d8ded0ed48d4d.tar.bz2 spark-4ecbe1b92f4c4c5b2d734895c09d8ded0ed48d4d.zip |
[SPARK-17810][SQL] Default spark.sql.warehouse.dir is relative to local FS but can resolve as HDFS path
## What changes were proposed in this pull request?
Always resolve spark.sql.warehouse.dir as a local path, and as relative to working dir not home dir
## How was this patch tested?
Existing tests.
Author: Sean Owen <sowen@cloudera.com>
Closes #15382 from srowen/SPARK-17810.
Diffstat (limited to 'docs/sql-programming-guide.md')
-rw-r--r-- | docs/sql-programming-guide.md | 33 |
1 files changed, 5 insertions, 28 deletions
diff --git a/docs/sql-programming-guide.md b/docs/sql-programming-guide.md index d334a86bc7..064af41965 100644 --- a/docs/sql-programming-guide.md +++ b/docs/sql-programming-guide.md @@ -904,50 +904,27 @@ access data stored in Hive. Configuration of Hive is done by placing your `hive-site.xml`, `core-site.xml` (for security configuration), and `hdfs-site.xml` (for HDFS configuration) file in `conf/`. -<div class="codetabs"> - -<div data-lang="scala" markdown="1"> - When working with Hive, one must instantiate `SparkSession` with Hive support, including connectivity to a persistent Hive metastore, support for Hive serdes, and Hive user-defined functions. Users who do not have an existing Hive deployment can still enable Hive support. When not configured by the `hive-site.xml`, the context automatically creates `metastore_db` in the current directory and creates a directory configured by `spark.sql.warehouse.dir`, which defaults to the directory -`spark-warehouse` in the current directory that the spark application is started. Note that +`spark-warehouse` in the current directory that the Spark application is started. Note that the `hive.metastore.warehouse.dir` property in `hive-site.xml` is deprecated since Spark 2.0.0. Instead, use `spark.sql.warehouse.dir` to specify the default location of database in warehouse. -You may need to grant write privilege to the user who starts the spark application. +You may need to grant write privilege to the user who starts the Spark application. +<div class="codetabs"> + +<div data-lang="scala" markdown="1"> {% include_example spark_hive scala/org/apache/spark/examples/sql/hive/SparkHiveExample.scala %} </div> <div data-lang="java" markdown="1"> - -When working with Hive, one must instantiate `SparkSession` with Hive support, including -connectivity to a persistent Hive metastore, support for Hive serdes, and Hive user-defined functions. -Users who do not have an existing Hive deployment can still enable Hive support. When not configured -by the `hive-site.xml`, the context automatically creates `metastore_db` in the current directory and -creates a directory configured by `spark.sql.warehouse.dir`, which defaults to the directory -`spark-warehouse` in the current directory that the spark application is started. Note that -the `hive.metastore.warehouse.dir` property in `hive-site.xml` is deprecated since Spark 2.0.0. -Instead, use `spark.sql.warehouse.dir` to specify the default location of database in warehouse. -You may need to grant write privilege to the user who starts the spark application. - {% include_example spark_hive java/org/apache/spark/examples/sql/hive/JavaSparkHiveExample.java %} </div> <div data-lang="python" markdown="1"> - -When working with Hive, one must instantiate `SparkSession` with Hive support, including -connectivity to a persistent Hive metastore, support for Hive serdes, and Hive user-defined functions. -Users who do not have an existing Hive deployment can still enable Hive support. When not configured -by the `hive-site.xml`, the context automatically creates `metastore_db` in the current directory and -creates a directory configured by `spark.sql.warehouse.dir`, which defaults to the directory -`spark-warehouse` in the current directory that the spark application is started. Note that -the `hive.metastore.warehouse.dir` property in `hive-site.xml` is deprecated since Spark 2.0.0. -Instead, use `spark.sql.warehouse.dir` to specify the default location of database in warehouse. -You may need to grant write privilege to the user who starts the spark application. - {% include_example spark_hive python/sql/hive.py %} </div> |