diff options
author | erenavsarogullari <erenavsarogullari@gmail.com> | 2017-02-06 08:24:17 -0600 |
---|---|---|
committer | Imran Rashid <irashid@cloudera.com> | 2017-02-06 08:24:17 -0600 |
commit | 7beb227cc8a4674e24cb1aaa278287ecc8194e5d (patch) | |
tree | 69465e065152f63260a660807011a93af6a1dfa1 /core/src/test/resources/fairscheduler-with-invalid-data.xml | |
parent | 7730426cb95eec2652a9ea979ae2c4faf7e585f2 (diff) | |
download | spark-7beb227cc8a4674e24cb1aaa278287ecc8194e5d.tar.gz spark-7beb227cc8a4674e24cb1aaa278287ecc8194e5d.tar.bz2 spark-7beb227cc8a4674e24cb1aaa278287ecc8194e5d.zip |
[SPARK-17663][CORE] SchedulableBuilder should handle invalid data access via scheduler.allocation.file
## What changes were proposed in this pull request?
If `spark.scheduler.allocation.file` has invalid `minShare` or/and `weight` values, these cause :
- `NumberFormatException` due to `toInt` function
- `SparkContext` can not be initialized.
- It does not show meaningful error message to user.
In a nutshell, this functionality can be more robust by selecting one of the following flows :
**1-** Currently, if `schedulingMode` has an invalid value, a warning message is logged and default value is set as `FIFO`. Same pattern can be used for `minShare`(default: 0) and `weight`(default: 1) as well
**2-** Meaningful error message can be shown to the user for all invalid cases.
PR offers :
- `schedulingMode` handles just empty values. It also needs to be supported for **whitespace**, **non-uppercase**(fair, FaIr etc...) or `SchedulingMode.NONE` cases by setting default value(`FIFO`)
- `minShare` and `weight` handle just empty values. They also need to be supported for **non-integer** cases by setting default values.
- Some refactoring of `PoolSuite`.
**Code to Reproduce :**
```
val conf = new SparkConf().setAppName("spark-fairscheduler").setMaster("local")
conf.set("spark.scheduler.mode", "FAIR")
conf.set("spark.scheduler.allocation.file", "src/main/resources/fairscheduler-invalid-data.xml")
val sc = new SparkContext(conf)
```
**fairscheduler-invalid-data.xml :**
```
<allocations>
<pool name="production">
<schedulingMode>FIFO</schedulingMode>
<weight>invalid_weight</weight>
<minShare>2</minShare>
</pool>
</allocations>
```
**Stacktrace :**
```
Exception in thread "main" java.lang.NumberFormatException: For input string: "invalid_weight"
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Integer.parseInt(Integer.java:580)
at java.lang.Integer.parseInt(Integer.java:615)
at scala.collection.immutable.StringLike$class.toInt(StringLike.scala:272)
at scala.collection.immutable.StringOps.toInt(StringOps.scala:29)
at org.apache.spark.scheduler.FairSchedulableBuilder$$anonfun$org$apache$spark$scheduler$FairSchedulableBuilder$$buildFairSchedulerPool$1.apply(SchedulableBuilder.scala:127)
at org.apache.spark.scheduler.FairSchedulableBuilder$$anonfun$org$apache$spark$scheduler$FairSchedulableBuilder$$buildFairSchedulerPool$1.apply(SchedulableBuilder.scala:102)
```
## How was this patch tested?
Added Unit Test Case.
Author: erenavsarogullari <erenavsarogullari@gmail.com>
Closes #15237 from erenavsarogullari/SPARK-17663.
Diffstat (limited to 'core/src/test/resources/fairscheduler-with-invalid-data.xml')
-rw-r--r-- | core/src/test/resources/fairscheduler-with-invalid-data.xml | 80 |
1 files changed, 80 insertions, 0 deletions
diff --git a/core/src/test/resources/fairscheduler-with-invalid-data.xml b/core/src/test/resources/fairscheduler-with-invalid-data.xml new file mode 100644 index 0000000000..a4d8d07b67 --- /dev/null +++ b/core/src/test/resources/fairscheduler-with-invalid-data.xml @@ -0,0 +1,80 @@ +<?xml version="1.0"?> +<!-- + ~ Licensed to the Apache Software Foundation (ASF) under one or more + ~ contributor license agreements. See the NOTICE file distributed with + ~ this work for additional information regarding copyright ownership. + ~ The ASF licenses this file to You under the Apache License, Version 2.0 + ~ (the "License"); you may not use this file except in compliance with + ~ the License. You may obtain a copy of the License at + ~ + ~ http://www.apache.org/licenses/LICENSE-2.0 + ~ + ~ Unless required by applicable law or agreed to in writing, software + ~ distributed under the License is distributed on an "AS IS" BASIS, + ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + ~ See the License for the specific language governing permissions and + ~ limitations under the License. + --> + +<allocations> + <pool name="pool_with_invalid_min_share"> + <minShare>INVALID_MIN_SHARE</minShare> + <weight>2</weight> + <schedulingMode>FAIR</schedulingMode> + </pool> + <pool name="pool_with_invalid_weight"> + <minShare>1</minShare> + <weight>INVALID_WEIGHT</weight> + <schedulingMode>FAIR</schedulingMode> + </pool> + <pool name="pool_with_invalid_scheduling_mode"> + <minShare>3</minShare> + <weight>2</weight> + <schedulingMode>INVALID_SCHEDULING_MODE</schedulingMode> + </pool> + <pool name="pool_with_non_uppercase_scheduling_mode"> + <minShare>2</minShare> + <weight>1</weight> + <schedulingMode>fair</schedulingMode> + </pool> + <pool name="pool_with_NONE_scheduling_mode"> + <minShare>1</minShare> + <weight>2</weight> + <schedulingMode>NONE</schedulingMode> + </pool> + <pool name="pool_with_whitespace_min_share"> + <minShare> </minShare> + <weight>2</weight> + <schedulingMode>FAIR</schedulingMode> + </pool> + <pool name="pool_with_whitespace_weight"> + <minShare>1</minShare> + <weight> </weight> + <schedulingMode>FAIR</schedulingMode> + </pool> + <pool name="pool_with_whitespace_scheduling_mode"> + <minShare>3</minShare> + <weight>2</weight> + <schedulingMode> </schedulingMode> + </pool> + <pool name="pool_with_empty_min_share"> + <minShare></minShare> + <weight>3</weight> + <schedulingMode>FAIR</schedulingMode> + </pool> + <pool name="pool_with_empty_weight"> + <minShare>2</minShare> + <weight></weight> + <schedulingMode>FAIR</schedulingMode> + </pool> + <pool name="pool_with_empty_scheduling_mode"> + <minShare>2</minShare> + <weight>2</weight> + <schedulingMode></schedulingMode> + </pool> + <pool name="pool_with_surrounded_whitespace"> + <minShare> 3 </minShare> + <weight> 2 </weight> + <schedulingMode> FAIR </schedulingMode> + </pool> +</allocations> |