Linting#

Before writing any code, set up formatters and linters.

Configuration#

New projects should use Black. All projects must use flake8 and isort with line lengths of 119 (the Django standard). If using Black, configure it as follows:

pyproject.toml#
[build-system]
requires = ["setuptools>=61.2"]
build-backend = "setuptools.build_meta"

[tool.black]
line-length = 119

[tool.isort]
profile = 'black'
line_length = 119

[tool.pydocstyle]
match_dir = '(?!tests).*'
ignore = 'D100,D104,D200,D203,D205,D212,D400,D415'
setup.cfg#
[flake8]
max-line-length = 119
extend-ignore = E203

[metadata]
name = {{ cookiecutter.package_name }}
version = 0.0.1
author = Open Contracting Partnership
author_email = data@open-contracting.org
license = BSD
description = {{ cookiecutter.short_description }}
url = https://github.com/open-contracting/{{ cookiecutter.repository_name }}
long_description = file: README.rst
long_description_content_type = text/x-rst
classifiers =
    License :: OSI Approved :: BSD License
{%- if cookiecutter.os_independent == "y" %}
    Operating System :: OS Independent
{%- else %}
    Operating System :: POSIX :: Linux
{%- endif %}
    Programming Language :: Python :: 3.7
    Programming Language :: Python :: 3.8
    Programming Language :: Python :: 3.9
    Programming Language :: Python :: 3.10
    Programming Language :: Python :: 3.11
    Programming Language :: Python :: Implementation :: CPython
{%- if cookiecutter.pypy == "y" %}
    Programming Language :: Python :: Implementation :: PyPy
{%- endif %}

[options]
packages = find:
install_requires =

[options.packages.find]
exclude =
    tests
    tests.*

[options.extras_require]
test =
    coveralls
    pytest
    pytest-cov
docs =
    furo
    sphinx
    sphinx-autobuild

Repositories should not modify or otherwise use pyproject.toml, setup.cfg, .editorconfig or tool-specific files, except to ignore generated files like database migrations. pyproject.toml is preferred to tool-specific files; if it’s not supported, setup.cfg is preferred – like for Flake8 and Babel.

Maintainers can find and compare configuration files with:

find . \( -name pyproject.toml -or -name setup.cfg -or -name .editorconfig -or -name .coveragerc -or -name .flake8 -or -name .isort.cfg -or -name .pylintrc -or -name pylintrc -or -name pytest.ini \) -not -path '*/node_modules/*' -exec bash -c 'sha=$(shasum {} | cut -d" " -f1); if [[ ! "45342d1e1c767ae5900edbcbde5c030adb30a753 ed723d5329bb74ab24e978c6b0ba6d2095e8fa1e 29418dd6acf27bb182036cf072790cb640f34c9c" =~ $sha ]]; then echo -e "\n\033[0;32m{}\033[0m"; echo $sha; cat {}; fi' \;

Pre-commit hooks#

To avoid pushing commits that fail formatting/linting checks, new projects should use pre-commit (add pre-commit to the requirements_dev.in file. For example, if Black is configured as above, create a .pre-commit-config.yaml file:

  • For an application:

    ci:
  autoupdate_schedule: quarterly
repos:
  - repo: https://github.com/psf/black
    rev: 22.6.0
    hooks:
      - id: black
  - repo: https://github.com/pycqa/flake8
    rev: 6.0.0
    hooks:
      - id: flake8
        additional_dependencies: [flake8-comprehensions]
  - repo: https://github.com/pycqa/isort
    rev: 5.12.0
    hooks:
      - id: isort
  - repo: https://github.com/jazzband/pip-tools
    rev: 6.8.0
    hooks:
      - id: pip-compile
        name: pip-compile requirements.in
        files: ^requirements\.(in|txt)$
      - id: pip-compile
        name: pip-compile requirements_dev.in
        files: ^requirements(_dev)?\.(in|txt)$
        args: [requirements_dev.in]

  • For a package:

    ci:
  autoupdate_schedule: quarterly
repos:
  - repo: https://github.com/psf/black
    rev: 22.6.0
    hooks:
      - id: black
  - repo: https://github.com/pycqa/flake8
    rev: 6.0.0
    hooks:
      - id: flake8
        additional_dependencies: [flake8-comprehensions]
  - repo: https://github.com/pycqa/isort
    rev: 5.12.0
    hooks:
      - id: isort
  - repo: https://github.com/pycqa/pydocstyle
    rev: 6.1.1
    hooks:
      - id: pydocstyle
        additional_dependencies: [toml]
        files: ^(?!tests)

To ignore generated files, you can add, for example, exclude: /migrations/ to the end of the file.

Note

pre-commit/pre-commit-hooks is not used in the templates, as the errors it covers are rarely encountered.

Skipping linting#

isort:skip and noqa comments should be kept to a minimum, and should reference the specific error, to avoid shadowing another error: for example, # noqa: E501.

The errors that are allowed to be ignored are:

  • E501 line too long for long strings, especially URLs

  • F401 module imported but unused in a library’s top-level __init__.py file

  • E402 module level import not at top of file in a Django project’s asgi.py file

  • W291 Trailing whitespace in tests relating to trailing whitespace

  • isort:skip if sys.path needs to be changed before an import

Maintainers can find unwanted comments with this regular expression:

# noqa(?!(: (E501|F401|E402|W291)| isort:skip)\n)

Continuous integration#

Create a .github/workflows/lint.yml file. As a base, use:

name: Lint
on: [push, pull_request]
env:
  BASEDIR: https://raw.githubusercontent.com/open-contracting/standard-maintenance-scripts/main
jobs:
  build:
    if: github.event_name == 'push' || github.event.pull_request.head.repo.full_name != github.repository
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-python@v4
        with:
          python-version: '3.10'
          cache: pip
          cache-dependency-path: '**/requirements*.txt'
      - shell: bash
        run: curl -s -S --retry 3 $BASEDIR/tests/install.sh | bash -
      - shell: bash
        run: curl -s -S --retry 3 $BASEDIR/tests/script.sh | bash -

Note

If a repository – like an OCDS extension – has no dependency file, the actions/cache action must be used instead of the cache input:

- uses: actions/cache@v3
  with:
    path: ~/.cache/pip
    key: ${{ runner.os }}-pip

See the documentation to learn about the Bash scripts.

If the project uses pre-commit, add the repository to pre-commit.ci to check and fix any issues.

Otherwise, if the project uses Black, add:

- run: pip install black
- run: black --check .

Unless the project is documentation only (like a handbook or a standard):

  • For an application, change the python-version to match the version used to compile the requirements_dev.txt file, and add:

    - run: pip install -r requirements_dev.txt
- run: pytest /tmp/test_requirements.py

  • For a package, add:

    - run: pip install .[test]
- run: pytest /tmp/test_requirements.py

If the project is a package, add:

- run: pip install --upgrade check-manifest
- run: check-manifest

Finally, add any project-specific linting, like in notebooks-ocds.

See also

Workflow files for linting shell scripts and Javascript files

Optional linting#

Note

This section is provided for reference.

codespell finds typographical errors. It is especially useful in repositories with lengthy documentation. Otherwise, all repositories can be periodically checked with:

codespell -S '.git,.pytest_cache,cassettes,fixtures,_build,build,dist,target,locale,locales,vendor,node_modules,docson,htmlcov,redmine,schemaspy,*.csv,*.json,*.jsonl,*.map,*.po,european-union-support'

flake8’s --max-complexity option (provided by mccabe) is deactivated by default. A threshold of 10 or 15 is recommended:

flake8 . --max-line-length 119 --max-complexity 10

Note

Complexity is best measured by the effort required to read and modify code. This cannot be measured using techniques like cyclomatic complexity. Reducing cyclomatic complexity typically means extracting single-caller methods and/or using object-oriented programming, which frequently increase cognitive complexity. See the note under Create products sustainably.

pylint and pylint-django provides useful, but noisy, feedback:

pip install pylint
pylint --max-line-length 119 directory

The Python Code Quality Authority maintains flake8 (which includes mccabe, pycodestyle and pyflakes), isort and pylint.