diff options
| author | KaiXinXiaoLei <huleilei1@huawei.com> | 2015-02-09 20:58:58 -0800 |
|---|---|---|
| committer | Reynold Xin <rxin@databricks.com> | 2015-02-09 20:58:58 -0800 |
| commit | 31d435ecfdc24a788a6e38f4e82767bc275a3283 (patch) | |
| tree | 218fe6b63f2ef4b300c9a137cca383a4762a3ffd /core | |
| parent | 08488c175f2e8532cb6aab84da2abd9ad57179cc (diff) | |
| download | spark-31d435ecfdc24a788a6e38f4e82767bc275a3283.tar.gz spark-31d435ecfdc24a788a6e38f4e82767bc275a3283.tar.bz2 spark-31d435ecfdc24a788a6e38f4e82767bc275a3283.zip | |
Add a config option to print DAG.
Add a config option "spark.rddDebug.enable" to check whether to print DAG info. When "spark.rddDebug.enable" is true, it will print information about DAG in the log.
Author: KaiXinXiaoLei <huleilei1@huawei.com>
Closes #4257 from KaiXinXiaoLei/DAGprint and squashes the following commits:
d9fe42e [KaiXinXiaoLei] change log info
c27ee76 [KaiXinXiaoLei] change log info
83c2b32 [KaiXinXiaoLei] change config option
adcb14f [KaiXinXiaoLei] change the file.
f4e7b9e [KaiXinXiaoLei] add a option to print DAG
Diffstat (limited to 'core')
| -rw-r--r-- | core/src/main/scala/org/apache/spark/SparkContext.scala | 3 |
1 files changed, 3 insertions, 0 deletions
diff --git a/core/src/main/scala/org/apache/spark/SparkContext.scala b/core/src/main/scala/org/apache/spark/SparkContext.scala index 71bdbc9b38..8d3c3d000a 100644 --- a/core/src/main/scala/org/apache/spark/SparkContext.scala +++ b/core/src/main/scala/org/apache/spark/SparkContext.scala @@ -1420,6 +1420,9 @@ class SparkContext(config: SparkConf) extends Logging with ExecutorAllocationCli val callSite = getCallSite val cleanedFunc = clean(func) logInfo("Starting job: " + callSite.shortForm) + if (conf.getBoolean("spark.logLineage", false)) { + logInfo("RDD's recursive dependencies:\n" + rdd.toDebugString) + } dagScheduler.runJob(rdd, cleanedFunc, partitions, callSite, allowLocal, resultHandler, localProperties.get) progressBar.foreach(_.finishAll()) |