File formats#

We read and write a lot of CSV and JSON files. Their format should be consistent.

JSON#

Input#

In most cases, simply use the standard library.

with open(path) as f:
    data = json.load(f)

For critical paths involving small files, use orjson.

Note

We can switch to the Python bindings for simdjson: either pysimdjson or libpy_simdjson. For JSON documents with known structures, JSON Link is fastest, but the files relevant to us have unknown structures.

For large files, use the same techniques as OCDS Kit to stream input using ijson, stream output using iterencode, and postpone evaluation using iterators. See its brief tutorial on streaming and reuse its default method.

Note

ijson uses Yajl. simdjson is limited to files smaller than 4 GB and has no streaming API.

Output#

Indent with 2 spaces and use UTF-8 characters. Example:

with open(path, "w") as f:
    json.dump(data, f, ensure_ascii=False, indent=2)
    f.write("\n")

Or, in a compact format:

with open(path, "w") as f:
    json.dump(data, f, separators=(",", ":"))

CSV#

Input#

with open(path) as f:
    reader = csv.DictReader(f)
    fieldnames = reader.fieldnames
    rows = [row for row in reader]

Output#

Use LF (\n) as the line terminator. Example:

with open(path, "w") as f:
    writer = csv.DictWriter(f, fieldnames, lineterminator="\n")
    writer.writeheader()
    writer.writerows(rows)