diff options
| author | Cheng Hao <hao.cheng@intel.com> | 2014-09-04 19:16:12 -0700 |
|---|---|---|
| committer | Michael Armbrust <michael@databricks.com> | 2014-09-04 19:16:12 -0700 |
| commit | 1904bac38d97df5ae9fb193e92a83c7f8ff6d255 (patch) | |
| tree | 6d04c84847f4021c6dbf273bb0d1886179407136 /sql/core | |
| parent | ee575f12f2ab059d9c1b4fa8d6c1e62248c3d11b (diff) | |
| download | spark-1904bac38d97df5ae9fb193e92a83c7f8ff6d255.tar.gz spark-1904bac38d97df5ae9fb193e92a83c7f8ff6d255.tar.bz2 spark-1904bac38d97df5ae9fb193e92a83c7f8ff6d255.zip | |
[SPARK-3392] [SQL] Show value spark.sql.shuffle.partitions for mapred.reduce.tasks
This is a tiny fix for getting the value of "mapred.reduce.tasks", which make more sense for the hive user.
As well as the command "set -v", which should output verbose information for all of the key/values.
Author: Cheng Hao <hao.cheng@intel.com>
Closes #2261 from chenghao-intel/set_mapreduce_tasks and squashes the following commits:
653858a [Cheng Hao] show value spark.sql.shuffle.partitions for mapred.reduce.tasks
Diffstat (limited to 'sql/core')
| -rw-r--r-- | sql/core/src/main/scala/org/apache/spark/sql/execution/commands.scala | 16 |
1 files changed, 12 insertions, 4 deletions
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/commands.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/commands.scala index 286c6d264f..94543fc95b 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/commands.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/commands.scala @@ -60,10 +60,10 @@ case class SetCommand( logWarning(s"Property ${SQLConf.Deprecated.MAPRED_REDUCE_TASKS} is deprecated, " + s"automatically converted to ${SQLConf.SHUFFLE_PARTITIONS} instead.") context.setConf(SQLConf.SHUFFLE_PARTITIONS, v) - Array(Row(s"${SQLConf.SHUFFLE_PARTITIONS}=$v")) + Seq(Row(s"${SQLConf.SHUFFLE_PARTITIONS}=$v")) } else { context.setConf(k, v) - Array(Row(s"$k=$v")) + Seq(Row(s"$k=$v")) } // Query the value bound to key k. @@ -78,11 +78,19 @@ case class SetCommand( "hive-hwi-0.12.0.jar", "hive-0.12.0.jar").mkString(":") - Array( + context.getAllConfs.map { case (k, v) => + Row(s"$k=$v") + }.toSeq ++ Seq( Row("system:java.class.path=" + hiveJars), Row("system:sun.java.command=shark.SharkServer2")) } else { - Array(Row(s"$k=${context.getConf(k, "<undefined>")}")) + if (k == SQLConf.Deprecated.MAPRED_REDUCE_TASKS) { + logWarning(s"Property ${SQLConf.Deprecated.MAPRED_REDUCE_TASKS} is deprecated, " + + s"showing ${SQLConf.SHUFFLE_PARTITIONS} instead.") + Seq(Row(s"${SQLConf.SHUFFLE_PARTITIONS}=${context.numShufflePartitions}")) + } else { + Seq(Row(s"$k=${context.getConf(k, "<undefined>")}")) + } } // Query all key-value pairs that are set in the SQLConf of the context. |