Compliance suite has not gained a case since 2024, while the join engine was rewritten under it

[amc-corey-cox]

Regenerating docs/specification/compliance.md today produces the same content as the version stamped Time_executed: 2024-07-03 — all 66 headings match one for one, differing only by the {#anchor} suffixes pandoc adds. The doc isn't out of date because nobody rebuilt it. It's identical because the suite has not gained a case in two years.

Ten commits have touched tests/test_compliance/ since that date, and every one is mechanical: isort, Run ruff format, Automatic ruff lint fixes, Move "unit" dev group into the main group, a linkml version bump.

What the 55 combos actually cover:

test_simple_unit_conversion    13        test_map_enum           4
test_expr                       9        test_inheritance        4
test_map_types                  8        test_isomorphic         2
test_stringify                  7        test_map_collections    2
test_complex_unit_conversion    4        test_join               2

Seventeen of the 55 are unit conversion. Joins get two — against an engine that has since been replaced:

510b0a5  Set-based DuckDB join engine (default, with per-row fallback)
4f44ec3  Synthesize implicit joins from expression references
a19eb09  Suppress nested object on sparse join miss
66cbcc3  Fix dotted populated_from crash and synthesize its join
d5abfd0  Implicit cross-table join resolution for nested class derivations

Enum handling also changed recently (7cffbf0, routing enum expression evaluation through the gated eval path), and --unrestricted-eval gating is a behavioural boundary a compliance suite ought to pin down.

This is the part that matters more than how the doc gets built: we publish this under Specification in the nav, so it reads as a statement of what LinkML-Map does. Right now it's a statement about mid-2024, and the areas we've changed most are the ones it covers least.

Intended outcome: the compliance suite exercises what the engine actually does now — set-based joins and sparse-join misses in particular — so the published doc is a description of current behaviour rather than a historical one.

Worth doing alongside #318, which covers why the doc can't currently be regenerated reproducibly. Neither blocks the other, but both need to land before a CI drift check could keep this honest on its own.

[github-actions]

Summary

Verified against current main: this issue's numbers hold up. test_join in tests/test_compliance/test_compliance_suite.py (lines ~914–1010) is parametrized over inlined: [True, False] against a single (source_object, target_object) pair — 2 combos total, exercising only the expr-based s1.id / s2.id denormalization path via populated_from. Nothing in the compliance suite invokes the DuckDB engine selection path.

One nuance worth adding

The join engine itself is not actually undertested — it's just untested in the artifact that gets published as the spec. tests/test_transformer/test_join_engine.py is 681 lines with 18+ cases, including test_sparse_miss_suppresses_object (the a19eb09 behavior called out in this issue), test_simple_subject_join_is_engine_capable / test_fk_chain_is_not_engine_capable (the can_use_join_engine gating in src/linkml_map/transformer/join_engine.py:108), test_to_many_join_does_not_explode_rows, and test_empty_joined_file_degrades_to_null. There's also test_cross_class_lookup.py (254 lines) and test_join_synthesis_completeness.py.

So the real gap this issue identifies is precise: it's not "the join rewrite shipped without tests," it's "the join rewrite shipped without compliance-suite tests," and since docs/specification/compliance.md is what's linked under Specification in the nav, readers of that doc have no way to know the set-based engine, its per-row fallback, or sparse-miss suppression exist. That actually makes the fix cheaper than starting from scratch — the scenarios already exist in test_join_engine.py and just need porting into the compliance suite's map_object/scaffold style (per tests/README.md's pattern) rather than being invented fresh. Same logic likely applies to the enum/--unrestricted-eval gating point — object_transformer.py:592 and cli.py:81 confirm that gate is real and current, so it's a matter of surfacing existing behavior in a compliance case, not building new coverage.

Related issues (same audit thread, all from @amc-corey-cox)

• The compliance doc can't be regenerated reproducibly, and the pandoc step is redundant #318 — explicitly linked here. Covers the build side (non-reproducible Time_executed/Package: stamps, redundant pandoc step) of the same doc. Confirmed still open; this issue and The compliance doc can't be regenerated reproducibly, and the pandoc step is redundant #318 are correctly scoped as independent (neither blocks the other) but both need to land before a CI drift check is meaningful.
• Makefile is half broken cookiecutter boilerplate; make deploy always fails #299 — the make specification target this doc depends on was broken until Retire the cookiecutter scaffolding in the Makefile and agent docs #317 fixed a path typo; worth knowing the doc-generation pipeline itself is only recently reachable at all.
• Cut misfiled and mock-based tests (~1,600 lines) #298 — a test-suite complexity audit that explicitly flags test_compliance_suite.py as "the sole coverage for TransformationSpecificationInverter" and says it "stays" — reinforcing that this file is treated as load-bearing spec material, not disposable test cruft, which raises the stakes on it being current.
• Alternative Implementation: Rust Port (linkml-map-rs) #262 — a third-party Rust port (linkml-map-rs) claims "passes the ported upstream compliance suite (55/55 tests) and matches reference Python outputs byte-for-byte." Worth flagging to that author once this suite gains cases: their parity claim is against the 2024-frozen 55, so it says nothing about join-engine or gated-eval behavior yet.

No external papers or web resources apply here — this is an internal test-coverage/doc-fidelity issue.