diff options
author | hyukjinkwon <gurwls223@gmail.com> | 2017-01-02 15:23:19 +0000 |
---|---|---|
committer | Sean Owen <sowen@cloudera.com> | 2017-01-02 15:23:19 +0000 |
commit | 46b212602428f1f11c184c836b4e09c150d0ee30 (patch) | |
tree | b30420dbdfe979f65c390edbfe2d103572c07501 /dev/lint-python | |
parent | f1330b1d9e7b1d5de611e59eecae1bf0b0616d81 (diff) | |
download | spark-46b212602428f1f11c184c836b4e09c150d0ee30.tar.gz spark-46b212602428f1f11c184c836b4e09c150d0ee30.tar.bz2 spark-46b212602428f1f11c184c836b4e09c150d0ee30.zip |
[SPARK-19002][BUILD][PYTHON] Check pep8 against all Python scripts
## What changes were proposed in this pull request?
This PR proposes to check pep8 against all other Python scripts and fix the errors as below:
```bash
./dev/create-release/generate-contributors.py
./dev/create-release/releaseutils.py
./dev/create-release/translate-contributors.py
./dev/lint-python
./python/docs/epytext.py
./examples/src/main/python/mllib/decision_tree_classification_example.py
./examples/src/main/python/mllib/decision_tree_regression_example.py
./examples/src/main/python/mllib/gradient_boosting_classification_example.py
./examples/src/main/python/mllib/gradient_boosting_regression_example.py
./examples/src/main/python/mllib/linear_regression_with_sgd_example.py
./examples/src/main/python/mllib/logistic_regression_with_lbfgs_example.py
./examples/src/main/python/mllib/naive_bayes_example.py
./examples/src/main/python/mllib/random_forest_classification_example.py
./examples/src/main/python/mllib/random_forest_regression_example.py
./examples/src/main/python/mllib/svm_with_sgd_example.py
./examples/src/main/python/streaming/network_wordjoinsentiments.py
./sql/hive/src/test/resources/data/scripts/cat.py
./sql/hive/src/test/resources/data/scripts/cat_error.py
./sql/hive/src/test/resources/data/scripts/doubleescapedtab.py
./sql/hive/src/test/resources/data/scripts/dumpdata_script.py
./sql/hive/src/test/resources/data/scripts/escapedcarriagereturn.py
./sql/hive/src/test/resources/data/scripts/escapednewline.py
./sql/hive/src/test/resources/data/scripts/escapedtab.py
./sql/hive/src/test/resources/data/scripts/input20_script.py
./sql/hive/src/test/resources/data/scripts/newline.py
```
## How was this patch tested?
- `./python/docs/epytext.py`
```bash
cd ./python/docs $$ make html
```
- pep8 check (Python 2.7 / Python 3.3.6)
```
./dev/lint-python
```
- `./dev/merge_spark_pr.py` (Python 2.7 only / Python 3.3.6 not working)
```bash
python -m doctest -v ./dev/merge_spark_pr.py
```
- `./dev/create-release/releaseutils.py` `./dev/create-release/generate-contributors.py` `./dev/create-release/translate-contributors.py` (Python 2.7 only / Python 3.3.6 not working)
```bash
python generate-contributors.py
python translate-contributors.py
```
- Examples (Python 2.7 / Python 3.3.6)
```bash
./bin/spark-submit examples/src/main/python/mllib/decision_tree_classification_example.py
./bin/spark-submit examples/src/main/python/mllib/decision_tree_regression_example.py
./bin/spark-submit examples/src/main/python/mllib/gradient_boosting_classification_example.py
./bin/spark-submit examples/src/main/python/mllib/gradient_boosting_regression_example.p
./bin/spark-submit examples/src/main/python/mllib/random_forest_classification_example.py
./bin/spark-submit examples/src/main/python/mllib/random_forest_regression_example.py
```
- Examples (Python 2.7 only / Python 3.3.6 not working)
```
./bin/spark-submit examples/src/main/python/mllib/linear_regression_with_sgd_example.py
./bin/spark-submit examples/src/main/python/mllib/logistic_regression_with_lbfgs_example.py
./bin/spark-submit examples/src/main/python/mllib/naive_bayes_example.py
./bin/spark-submit examples/src/main/python/mllib/svm_with_sgd_example.py
```
- `sql/hive/src/test/resources/data/scripts/*.py` (Python 2.7 / Python 3.3.6 within suggested changes)
Manually tested only changed ones.
- `./dev/github_jira_sync.py` (Python 2.7 only / Python 3.3.6 not working)
Manually tested this after disabling actually adding comments and links.
And also via Jenkins tests.
Author: hyukjinkwon <gurwls223@gmail.com>
Closes #16405 from HyukjinKwon/minor-pep8.
Diffstat (limited to 'dev/lint-python')
-rwxr-xr-x | dev/lint-python | 6 |
1 files changed, 2 insertions, 4 deletions
diff --git a/dev/lint-python b/dev/lint-python index 3f878c2dad..c6f3fbfab8 100755 --- a/dev/lint-python +++ b/dev/lint-python @@ -19,10 +19,8 @@ SCRIPT_DIR="$( cd "$( dirname "$0" )" && pwd )" SPARK_ROOT_DIR="$(dirname "$SCRIPT_DIR")" -PATHS_TO_CHECK="./python/pyspark/ ./examples/src/main/python/ ./dev/sparktestsupport" -# TODO: fix pep8 errors with the rest of the Python scripts under dev -PATHS_TO_CHECK="$PATHS_TO_CHECK ./dev/run-tests.py ./python/*.py ./dev/run-tests-jenkins.py" -PATHS_TO_CHECK="$PATHS_TO_CHECK ./dev/pip-sanity-check.py" +# Exclude auto-geneated configuration file. +PATHS_TO_CHECK="$( cd "$SPARK_ROOT_DIR" && find . -name "*.py" -not -path "*python/docs/conf.py" )" PEP8_REPORT_PATH="$SPARK_ROOT_DIR/dev/pep8-report.txt" PYLINT_REPORT_PATH="$SPARK_ROOT_DIR/dev/pylint-report.txt" PYLINT_INSTALL_INFO="$SPARK_ROOT_DIR/dev/pylint-info.txt" |