Skip to content

docs: one table maps each governance control to the failure class it prevents and the test that proves it #5569

Description

@chernistry

Problem

Controls are documented one at a time — grant chains and the sweep (src/bernstein/core/identity/grant_sweep.py, bernstein audit verify), artifact contracts, the typed activity boundary, the audit chain, compliance presets, consensus scoring (docs/quality/consensus-scoring.md), calibration (docs/operations/calibration.md). An operator evaluating the layer for a workload has to reconstruct which failure each control prevents, and nothing states which failure classes have no control at all.

What to build

docs/governance/coverage.md, generated from docs/governance/coverage.yaml:

  • Rows: the failure classes agentic workloads exhibit — goal hijack through injected content; tool misuse and over-broad permissions; identity and privilege abuse across delegation; compromised skills, tools or dependencies; unexpected code execution; memory or context poisoning; cascading failures across agents; exploitation of human trust in agent output; insufficient traceability of decisions; unbounded or runaway behaviour.
  • Columns: control · where it is enforced (module) · the test that proves it (tests/...::test_name) · residual risk · status (covered / partial / none).
  • A unit test that every cited test exists and is collected, and that no row says covered without a test.
  • Every none row becomes an issue in this milestone, linked from the row.

Acceptance criteria

  • test_coverage_table_cites_only_existing_tests
  • test_no_row_is_covered_without_a_test
  • coverage.md regenerates byte-identically from the YAML in CI.
  • Linked from the docs index and from SECURITY.md.

Where the code is

What Path
Grants, sweep, verify src/bernstein/core/identity/, src/bernstein/cli/commands/audit_cmd.py
Quality and calibration src/bernstein/core/quality/, src/bernstein/eval/calibration.py
Existing docs to link docs/quality/consensus-scoring.md, docs/operations/calibration.md, SECURITY.md

One decision left to the implementer

Whether partial needs a one-line justification column or a link to the issue that closes the gap (recommended: the link; prose rots).

Agent brief

Implement this issue in sipyourdrink-ltd/bernstein. Fill the module and test columns from the code, not from memory: for each control, find the test that fails when the control is removed, and cite that one. Where no such test exists, the row is `partial` at best. Generate the markdown from the YAML with a small script under `scripts/` and add the CI regeneration check. Plain language; no product comparisons.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions