diff options
| author | Bill Bejeck <bbejeck@gmail.com> | 2014-10-14 12:12:38 -0700 |
|---|---|---|
| committer | Josh Rosen <joshrosen@apache.org> | 2014-10-14 12:12:38 -0700 |
| commit | 9b6de6fbc00b184d81fc28ac160d03451fad80ec (patch) | |
| tree | f79c92561db8e861ac82a5f164be2fbad01dec3e /python/.gitignore | |
| parent | 186b497c945cc7bbe7a21fef56a948dd1fd10774 (diff) | |
| download | spark-9b6de6fbc00b184d81fc28ac160d03451fad80ec.tar.gz spark-9b6de6fbc00b184d81fc28ac160d03451fad80ec.tar.bz2 spark-9b6de6fbc00b184d81fc28ac160d03451fad80ec.zip | |
SPARK-3178 setting SPARK_WORKER_MEMORY to a value without a label (m or g) sets the worker memory limit to zero
Validate the memory is greater than zero when set from the SPARK_WORKER_MEMORY environment variable or command line without a g or m label. Added unit tests. If memory is 0 an IllegalStateException is thrown. Updated unit tests to mock environment variables by subclassing SparkConf (tip provided by Josh Rosen). Updated WorkerArguments to use SparkConf.getenv instead of System.getenv for reading the SPARK_WORKER_MEMORY environment variable.
Author: Bill Bejeck <bbejeck@gmail.com>
Closes #2309 from bbejeck/spark-memory-worker and squashes the following commits:
51cf915 [Bill Bejeck] SPARK-3178 - Validate the memory is greater than zero when set from the SPARK_WORKER_MEMORY environment variable or command line without a g or m label. Added unit tests. If memory is 0 an IllegalStateException is thrown. Updated unit tests to mock environment variables by subclassing SparkConf (tip provided by Josh Rosen). Updated WorkerArguments to use SparkConf.getenv instead of System.getenv for reading the SPARK_WORKER_MEMORY environment variable.
Diffstat (limited to 'python/.gitignore')
0 files changed, 0 insertions, 0 deletions