diff options
| author | lianhuiwang <lianhuiwang09@gmail.com> | 2015-02-02 12:32:28 -0800 |
|---|---|---|
| committer | Andrew Or <andrew@databricks.com> | 2015-02-02 12:32:28 -0800 |
| commit | f5e63751f0ed50ceafdc2ec5173b161a5155b646 (patch) | |
| tree | 75a199b5fa514ecd48ef634666e4d32d2036ef1b /sbin/start-history-server.sh | |
| parent | b2047b55c5fc85de6b63276d8ab9610d2496e08b (diff) | |
| download | spark-f5e63751f0ed50ceafdc2ec5173b161a5155b646.tar.gz spark-f5e63751f0ed50ceafdc2ec5173b161a5155b646.tar.bz2 spark-f5e63751f0ed50ceafdc2ec5173b161a5155b646.zip | |
[SPARK-5173]support python application running on yarn cluster mode
now when we run python application on yarn cluster mode through spark-submit, spark-submit does not support python application on yarn cluster mode. so i modify code of submit and yarn's AM in order to support it.
through specifying .py file or primaryResource file via spark-submit, we can make pyspark run in yarn-cluster mode.
example:spark-submit --master yarn-master --num-executors 1 --driver-memory 1g --executor-memory 1g xx.py --primaryResource yy.conf
this config is same as pyspark on yarn-client mode.
firstly,we put local path of .py or primaryResource to yarn's dist.files.that can be distributed on slave nodes.and then in spark-submit we transfer --py-files and --primaryResource to yarn.Client and use "org.apache.spark.deploy.PythonRunner" to user class that can run .py files on ApplicationMaster.
in yarn.Client we transfer --py-files and --primaryResource to ApplicationMaster.
in ApplicationMaster, user's class is org.apache.spark.deploy.PythonRunner, and user's args is primaryResource and -py-files. so that can make pyspark run on ApplicationMaster.
JoshRosen tgravescs sryza
Author: lianhuiwang <lianhuiwang09@gmail.com>
Author: Wang Lianhui <lianhuiwang09@gmail.com>
Closes #3976 from lianhuiwang/SPARK-5173 and squashes the following commits:
28a8a58 [lianhuiwang] fix variable name
67f8cee [lianhuiwang] update with andrewor's comments
0319ae3 [lianhuiwang] address with sryza's comments
2385ef6 [lianhuiwang] address with sryza's comments
03640ab [lianhuiwang] add sparkHome to env
47d2fc3 [lianhuiwang] fix test
2adc8f5 [lianhuiwang] add spark.test.home
d60bc60 [lianhuiwang] fix test
5b30064 [lianhuiwang] add test
097a5ec [lianhuiwang] fix line length exceeds 100
905a106 [lianhuiwang] update with sryza and andrewor 's comments
f1f55b6 [lianhuiwang] when yarn-cluster, all python files can be non-local
172eec1 [Wang Lianhui] fix a min submit's bug
9c941bc [lianhuiwang] support python application running on yarn cluster mode
Diffstat (limited to 'sbin/start-history-server.sh')
0 files changed, 0 insertions, 0 deletions