Logging

An application should use (but not over-use) logging to make it easier to debug and monitor it in production. The logging module is preferred.

See also

Logging section of the String formatting style guide

Name

In most cases, use logger = logging.getLogger(__name__), as recommended.

If a file is run directly, __name__ will be "__main__", which is less informative. In such cases, either use the name of the module, like "workers.check.dataset" in Pelican backend, or include the name of the sub-command, like "ocdskingfisher.summarize.add" in Kingfisher Summarize.

If a command-line tool logs messages to give user feedback, we typically use the name of the command, like "oc4ids", "ocdskit", "ocdsextensionregistry" and "spoonbill".

Format

In the context of web servers and worker daemons, the formatter’s format string should be set to:

"%(asctime)s %(levelname)s [%(name)s:%(lineno)s] %(message)s"

In most cases, the other placeholders are unnecessary:

• If the logger’s name is __name__, then %(name)s covers %(module)s (identical), %(pathname)s (too long) and %(filename)s (too short).

• %(funcName)s, unless %(name)s, %(lineno)s and %(message)s are insufficient to locate the relevant code.

• %(process)s and %(thread)s, unless the log messages from different processes/threads are written to the same location.

To configure the format in Django:

# https://docs.djangoproject.com/en/3.2/topics/logging/#django-security
LOGGING = {
    "version": 1,
    "disable_existing_loggers": False,
    "formatters": {
        "console": {
            "format": "%(asctime)s %(levelname)s [%(name)s:%(lineno)s] %(message)s",
        },
    },
    "handlers": {
        "console": {
            "class": "logging.StreamHandler",
            "formatter": "console",
        },
        "null": {
            "class": "logging.NullHandler",
        },
    },
    "loggers": {
        "": {
            "handlers": ["console"],
            "level": "INFO",
        },
        "django.security.DisallowedHost": {
            "handlers": ["null"],
            "propagate": False,
        },
        "mymodule": {
            "handlers": ["console"],
            "level": "DEBUG",
            "propagate": False,
        },
    },
}

Note

Django’s default logging configuration configures the django and django.server loggers. To change those (for example, to set the level to DEBUG in development to view database queries), add them to the above template.

To configure the format in general, replacing MYMODULE:

import logging
import logging.config

logging.config.dictConfig(
    {
        "version": 1,
        "disable_existing_loggers": False,
        "formatters": {
            "console": {
                "format": "%(asctime)s %(levelname)s [%(name)s:%(lineno)s] %(message)s",
            },
        },
        "handlers": {
            "console": {
                "class": "logging.StreamHandler",
                "formatter": "console",
            },
        },
        "loggers": {
            "MYMODULE": {
                "handlers": ["console"],
                "level": "INFO",
                "propagate": False,
            },
        },
    }
)

Reference: Python’s warnings.py format string

Methods

Use the corresponding method for the appropriate logging level.

When logging messages inside an except clause, if the error is unexpected and isn’t re-raised, use logger.exception(msg) to log the message at the ERROR level along with exception information. Do not bother with the traceback module.

Maintainers can review loggers inside except branches with this regular expression:

except (?!RecoverableException).+\n( +)(\S.*\n(\1.*\n)*\1)?.*log.*\.(?!exception\(|format\()

Configuration

Loggers are organized into a hierarchy. As such, you can configure only the root logger (its name is '' in Django, or None in general), or only the loggers for top-level modules (like only a, instead of both a.b and a.c).