diff options
| author | Carson Wang <carson.wang@intel.com> | 2015-08-07 23:36:26 -0700 |
|---|---|---|
| committer | Andrew Or <andrew@databricks.com> | 2015-08-07 23:36:26 -0700 |
| commit | ef062c15992b0d08554495b8ea837bef3fabf6e9 (patch) | |
| tree | 617fad193916589da5194f829cff96b0800e5b4e /python/pyspark/sql/types.py | |
| parent | c564b27447ed99e55b359b3df1d586d5766b85ea (diff) | |
| download | spark-ef062c15992b0d08554495b8ea837bef3fabf6e9.tar.gz spark-ef062c15992b0d08554495b8ea837bef3fabf6e9.tar.bz2 spark-ef062c15992b0d08554495b8ea837bef3fabf6e9.zip | |
[SPARK-9731] Standalone scheduling incorrect cores if spark.executor.cores is not set
The issue only happens if `spark.executor.cores` is not set and executor memory is set to a high value.
For example, if we have a worker with 4G and 10 cores and we set `spark.executor.memory` to 3G, then only 1 core is assigned to the executor. The correct number should be 10 cores.
I've added a unit test to illustrate the issue.
Author: Carson Wang <carson.wang@intel.com>
Closes #8017 from carsonwang/SPARK-9731 and squashes the following commits:
d09ec48 [Carson Wang] Fix code style
86b651f [Carson Wang] Simplify the code
943cc4c [Carson Wang] fix scheduling correct cores to executors
Diffstat (limited to 'python/pyspark/sql/types.py')
0 files changed, 0 insertions, 0 deletions