Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .github/workflows/tox.yml
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,8 @@ jobs:
fail-fast: false
steps:
- uses: actions/checkout@v4
with:
submodules: true
- uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
Expand All @@ -49,6 +51,8 @@ jobs:
fail-fast: false
steps:
- uses: actions/checkout@v4
with:
submodules: true
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v5
with:
Expand Down
3 changes: 3 additions & 0 deletions .gitmodules
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
[submodule "multicodec-spec"]
path = multicodec-spec
url = https://github.com/multiformats/multicodec
2 changes: 1 addition & 1 deletion .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
exclude: '.project-template|docs/conf.py|.*pb2\..*'
exclude: '.project-template|docs/conf.py|.*pb2\..*|multicodec-spec/.*'
repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v5.0.0
Expand Down
13 changes: 11 additions & 2 deletions CONTRIBUTING.rst
Original file line number Diff line number Diff line change
Expand Up @@ -60,9 +60,13 @@ Get Started!
Ready to contribute? Here's how to set up `multicodec` for local development.

1. Fork the `py-multicodec` repo on GitHub.
2. Clone your fork locally::
2. Clone your fork locally (include the ``multicodec-spec`` submodule)::

$ git clone git@github.com:your_name_here/py-multicodec.git
$ git clone --recurse-submodules git@github.com:your_name_here/py-multicodec.git

If you already cloned without submodules::

$ git submodule update --init --recursive

3. Install your local copy into a virtualenv. Create and activate a virtual environment::

Expand Down Expand Up @@ -114,6 +118,11 @@ The project provides several ``make`` targets to help with development:
* ``make test`` - Run the test suite with pytest using the default Python version.
For testing across multiple Python versions, use ``tox`` instead.

The spec compliance test (``tests/test_spec.py``) reads ``multicodec-spec/table.csv``.
Initialise the submodule (see step 2) before running the test suite locally.
Source tarballs do not include the submodule; that test is skipped unless you are
in a git checkout with ``multicodec-spec`` initialised.

* ``make pr`` - Run a complete pre-PR check: clean build artifacts, fix formatting,
run linting, type checking, and tests. This is the recommended command to run
before submitting a pull request.
Expand Down
2 changes: 1 addition & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ lint:
pre-commit run --all-files

fix:
python -m ruff check --fix
python -m ruff check --fix --exclude multicodec-spec --exclude docs/conf.py

typecheck:
pre-commit run mypy-local --all-files
Expand Down
21 changes: 18 additions & 3 deletions README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -80,10 +80,25 @@ management functionality:
Updating the lookup table
==========================

Updating the lookup table is done with a script. The source of truth is the
`multicodec default table <https://github.com/multiformats/multicodec/blob/master/table.csv>`_.
Update the table with running:
Updating the lookup table is done with a script. The authoritative table is vendored
in the ``multicodec-spec`` git submodule (``table.csv``). After a fresh clone, run
``git submodule update --init --recursive`` so the submodule and spec compliance test
are available.

To refresh the registry from upstream, bump the submodule pin and regenerate
``multicodec/constants.py``:

.. code-block:: shell

$ git -C multicodec-spec fetch origin
$ git -C multicodec-spec checkout origin/master
$ git add multicodec-spec
$ ./tools/update-table.py < multicodec-spec/table.csv

Alternatively, pipe the latest upstream CSV directly:

.. code-block:: shell

$ curl -X GET https://raw.githubusercontent.com/multiformats/multicodec/master/table.csv | ./tools/update-table.py

Commit the updated ``constants.py`` together with the new submodule commit SHA.
12 changes: 10 additions & 2 deletions docs/installation.rst
Original file line number Diff line number Diff line change
Expand Up @@ -28,11 +28,15 @@ From sources

The sources for py-multicodec can be downloaded from the `Github repo`_.

You can either clone the public repository:
You can either clone the public repository (with the ``multicodec-spec`` submodule):

.. code-block:: console

$ git clone git://github.com/multiformats/py-multicodec
$ git clone --recurse-submodules git://github.com/multiformats/py-multicodec

If you already cloned without submodules::

$ git submodule update --init --recursive

Or download the `tarball`_:

Expand All @@ -52,6 +56,10 @@ For development, install with dev dependencies:

$ pip install -e ".[dev]"

When running the full test suite from a git checkout, initialising ``multicodec-spec``
is required for ``tests/test_spec.py``. Installing from a source tarball without the
submodule skips that compliance test.


.. _Github repo: https://github.com/multiformats/py-multicodec
.. _tarball: https://github.com/multiformats/py-multicodec/tarball/master
1 change: 1 addition & 0 deletions multicodec-spec
Submodule multicodec-spec added at 38e3bf
1 change: 1 addition & 0 deletions newsfragments/41.feature.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Add a spec compliance test validating ``CODECS``, ``CODE_TABLE`` and ``NAME_TABLE`` against the authoritative ``table.csv``, which is vendored as the ``multicodec-spec`` git submodule.
1 change: 1 addition & 0 deletions ruff.toml
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ exclude = [
"dist",
"docs",
"*.egg",
"multicodec-spec",
]

[lint]
Expand Down
97 changes: 97 additions & 0 deletions tests/test_spec.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,97 @@
import csv
import os
from collections.abc import Iterator
from pathlib import Path

import pytest

from multicodec.constants import CODE_TABLE, CODECS, NAME_TABLE

SPEC_TABLE_CSV = Path(__file__).resolve().parent.parent / "multicodec-spec" / "table.csv"


def load_csv(path: Path) -> Iterator[dict[str, str]]:
with path.open(encoding="utf-8") as f:
reader = csv.DictReader(f)
for row in reader:
yield {k.strip(): v.strip() for k, v in row.items()}


def test_spec_table_completeness() -> None:
"""Every entry in table.csv should be in CODECS, and no extra entries should exist."""
if not SPEC_TABLE_CSV.is_file():
message = "multicodec-spec submodule is not initialised; run: git submodule update --init --recursive"
if os.environ.get("CI"):
pytest.fail(message)
pytest.skip(message)

csv_names: set[str] = set()
row_mismatches: list[str] = []

for row in load_csv(SPEC_TABLE_CSV):
name = row["name"]
tag = row["tag"]
status = row.get("status", "")
description = row.get("description", "")
code_str = row["code"]
csv_names.add(name)
code_int = int(code_str, 16)

if name not in NAME_TABLE:
row_mismatches.append(f"Missing codec from spec: {name} ({code_str})")
continue

if NAME_TABLE[name] != code_int:
row_mismatches.append(f"Code mismatch for {name}: expected {code_int}, got {NAME_TABLE[name]}")

if code_int not in CODE_TABLE:
row_mismatches.append(f"Missing code from CODE_TABLE: {code_int} ({name})")
elif CODE_TABLE[code_int] != name:
row_mismatches.append(f"Name mismatch for {code_int}: expected {name}, got {CODE_TABLE[code_int]}")

if name not in CODECS:
row_mismatches.append(f"Missing codec from CODECS: {name}")
continue

codec = CODECS[name]
if codec["prefix"] != code_int:
row_mismatches.append(f"Prefix mismatch for {name} in CODECS: expected {code_int}, got {codec['prefix']}")
if codec["tag"] != tag:
row_mismatches.append(f"Tag mismatch for {name}: expected {tag!r}, got {codec['tag']!r}")
if codec["status"] != status:
row_mismatches.append(f"Status mismatch for {name}: expected {status!r}, got {codec['status']!r}")
if codec["description"] != description:
row_mismatches.append(
f"Description mismatch for {name}: expected {description!r}, got {codec['description']!r}"
)

# Verify no extra entries exist in CODECS that aren't in the CSV
extra_codecs = sorted(name for name in CODECS if name not in csv_names)
if extra_codecs:
row_mismatches.append("Extra codecs in CODECS not in spec: " + ", ".join(extra_codecs))

table_mismatches: list[str] = []
if set(NAME_TABLE) != set(CODECS):
only_name = sorted(set(NAME_TABLE) - set(CODECS))
only_codecs = sorted(set(CODECS) - set(NAME_TABLE))
if only_name:
table_mismatches.append(f"NAME_TABLE keys not in CODECS: {', '.join(only_name)}")
if only_codecs:
table_mismatches.append(f"CODECS keys not in NAME_TABLE: {', '.join(only_codecs)}")

if set(CODE_TABLE.values()) != set(CODECS):
code_table_names = set(CODE_TABLE.values())
only_code_table = sorted(code_table_names - set(CODECS))
only_codecs_from_table = sorted(set(CODECS) - code_table_names)
if only_code_table:
table_mismatches.append(f"CODE_TABLE values not in CODECS: {', '.join(only_code_table)}")
if only_codecs_from_table:
table_mismatches.append(f"CODECS names missing from CODE_TABLE values: {', '.join(only_codecs_from_table)}")

if len(CODE_TABLE) != len(CODECS):
table_mismatches.append(f"CODE_TABLE length ({len(CODE_TABLE)}) != CODECS length ({len(CODECS)})")

all_mismatches = row_mismatches + table_mismatches
if all_mismatches:
report = "\n".join(all_mismatches)
pytest.fail(f"Spec table compliance failures ({len(all_mismatches)}):\n{report}")
Loading