Linting#
Before writing any code, set up formatters and linters.
Configuration#
New projects should use Black. All projects must use flake8 and isort with line lengths of 119 (the Django standard). If using Black, configure it as follows:
[build-system]
requires = ["setuptools>=61.2"]
build-backend = "setuptools.build_meta"
[tool.black]
line-length = 119
[tool.isort]
profile = 'black'
line_length = 119
[tool.pydocstyle]
match_dir = '(?!tests).*'
ignore = 'D100,D104,D200,D203,D205,D212,D400,D415'
[flake8]
max-line-length = 119
extend-ignore = E203
[metadata]
name = {{ cookiecutter.package_name }}
version = 0.0.1
author = Open Contracting Partnership
author_email = data@open-contracting.org
license = BSD
description = {{ cookiecutter.short_description }}
url = https://github.com/open-contracting/{{ cookiecutter.repository_name }}
long_description = file: README.rst
long_description_content_type = text/x-rst
classifiers =
License :: OSI Approved :: BSD License
{%- if cookiecutter.os_independent == "y" %}
Operating System :: OS Independent
{%- else %}
Operating System :: POSIX :: Linux
{%- endif %}
Programming Language :: Python :: 3.7
Programming Language :: Python :: 3.8
Programming Language :: Python :: 3.9
Programming Language :: Python :: 3.10
Programming Language :: Python :: 3.11
Programming Language :: Python :: Implementation :: CPython
{%- if cookiecutter.pypy == "y" %}
Programming Language :: Python :: Implementation :: PyPy
{%- endif %}
[options]
packages = find:
install_requires =
[options.packages.find]
exclude =
tests
tests.*
[options.extras_require]
test =
coveralls
pytest
pytest-cov
docs =
furo
sphinx
sphinx-autobuild
Repositories should not modify or otherwise use pyproject.toml
, setup.cfg
, .editorconfig
or tool-specific files, except to ignore generated files like database migrations. pyproject.toml
is preferred to tool-specific files; if it’s not supported, setup.cfg
is preferred – like for Flake8 and Babel.
Maintainers can find and compare configuration files with:
find . \( -name pyproject.toml -or -name setup.cfg -or -name .editorconfig -or -name .coveragerc -or -name .flake8 -or -name .isort.cfg -or -name .pylintrc -or -name pylintrc -or -name pytest.ini \) -not -path '*/node_modules/*' -exec bash -c 'sha=$(shasum {} | cut -d" " -f1); if [[ ! "45342d1e1c767ae5900edbcbde5c030adb30a753 ed723d5329bb74ab24e978c6b0ba6d2095e8fa1e 29418dd6acf27bb182036cf072790cb640f34c9c" =~ $sha ]]; then echo -e "\n\033[0;32m{}\033[0m"; echo $sha; cat {}; fi' \;
Pre-commit hooks#
To avoid pushing commits that fail formatting/linting checks, new projects should use pre-commit (add pre-commit
to the requirements_dev.in file. For example, if Black is configured as above, create a .pre-commit-config.yaml
file:
For an application:
ci: autoupdate_schedule: quarterly repos: - repo: https://github.com/psf/black rev: 22.6.0 hooks: - id: black - repo: https://github.com/pycqa/flake8 rev: 6.0.0 hooks: - id: flake8 additional_dependencies: [flake8-comprehensions] - repo: https://github.com/pycqa/isort rev: 5.12.0 hooks: - id: isort - repo: https://github.com/jazzband/pip-tools rev: 6.8.0 hooks: - id: pip-compile name: pip-compile requirements.in files: ^requirements\.(in|txt)$ - id: pip-compile name: pip-compile requirements_dev.in files: ^requirements(_dev)?\.(in|txt)$ args: [requirements_dev.in]
For a package:
ci: autoupdate_schedule: quarterly repos: - repo: https://github.com/psf/black rev: 22.6.0 hooks: - id: black - repo: https://github.com/pycqa/flake8 rev: 6.0.0 hooks: - id: flake8 additional_dependencies: [flake8-comprehensions] - repo: https://github.com/pycqa/isort rev: 5.12.0 hooks: - id: isort - repo: https://github.com/pycqa/pydocstyle rev: 6.1.1 hooks: - id: pydocstyle additional_dependencies: [toml] files: ^(?!tests)
To ignore generated files, you can add, for example, exclude: /migrations/
to the end of the file.
Note
pre-commit/pre-commit-hooks is not used in the templates, as the errors it covers are rarely encountered.
Skipping linting#
isort:skip
and noqa
comments should be kept to a minimum, and should reference the specific error, to avoid shadowing another error: for example, # noqa: E501
.
The errors that are allowed to be ignored are:
E501 line too long
for long strings, especially URLsF401 module imported but unused
in a library’s top-level__init__.py
fileE402 module level import not at top of file
in a Django project’sasgi.py
fileW291 Trailing whitespace
in tests relating to trailing whitespaceisort:skip
ifsys.path
needs to be changed before an import
Maintainers can find unwanted comments with this regular expression:
# noqa(?!(: (E501|F401|E402|W291)| isort:skip)\n)
Continuous integration#
Create a .github/workflows/lint.yml
file. As a base, use:
name: Lint
on: [push, pull_request]
env:
BASEDIR: https://raw.githubusercontent.com/open-contracting/standard-maintenance-scripts/main
jobs:
build:
if: github.event_name == 'push' || github.event.pull_request.head.repo.full_name != github.repository
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: actions/setup-python@v4
with:
python-version: '3.10'
cache: pip
cache-dependency-path: '**/requirements*.txt'
- shell: bash
run: curl -s -S --retry 3 $BASEDIR/tests/install.sh | bash -
- shell: bash
run: curl -s -S --retry 3 $BASEDIR/tests/script.sh | bash -
Note
If a repository – like an OCDS extension – has no dependency file, the actions/cache
action must be used instead of the cache
input:
- uses: actions/cache@v3
with:
path: ~/.cache/pip
key: ${{ runner.os }}-pip
See the documentation to learn about the Bash scripts.
If the project uses pre-commit, add the repository to pre-commit.ci to check and fix any issues.
Otherwise, if the project uses Black, add:
- run: pip install black
- run: black --check .
Unless the project is documentation only (like a handbook or a standard):
For an application, change the
python-version
to match the version used to compile the requirements_dev.txt file, and add:- run: pip install -r requirements_dev.txt - run: pytest /tmp/test_requirements.py
For a package, add:
- run: pip install .[test] - run: pytest /tmp/test_requirements.py
If the project is a package, add:
- run: pip install --upgrade check-manifest
- run: check-manifest
Finally, add any project-specific linting, like in notebooks-ocds.
See also
Workflow files for linting shell scripts and Javascript files
Optional linting#
Note
This section is provided for reference.
codespell finds typographical errors. It is especially useful in repositories with lengthy documentation. Otherwise, all repositories can be periodically checked with:
codespell -S '.git,.pytest_cache,cassettes,fixtures,_build,build,dist,target,locale,locales,vendor,node_modules,docson,htmlcov,redmine,schemaspy,*.csv,*.json,*.jsonl,*.map,*.po,european-union-support'
flake8’s --max-complexity
option (provided by mccabe) is deactivated by default. A threshold of 10 or 15 is recommended:
flake8 . --max-line-length 119 --max-complexity 10
Note
Complexity is best measured by the effort required to read and modify code. This cannot be measured using techniques like cyclomatic complexity. Reducing cyclomatic complexity typically means extracting single-caller methods and/or using object-oriented programming, which frequently increase cognitive complexity. See the note under Create products sustainably.
pylint and pylint-django provides useful, but noisy, feedback:
pip install pylint
pylint --max-line-length 119 directory
The Python Code Quality Authority maintains flake8
(which includes mccabe
, pycodestyle
and pyflakes
), isort
and pylint
.